Strategy Analyzer consuming memory?

[Skechers]

Hello there,

I've been working with the Strategy Analyzer and noticed when running an optimization it would consume memory, about 25MB per run, but does not release it afterwards.f

I've uploaded an MP4 video here: https://1drv.ms/f/s!AvRv1TeTT2RpgvJ5...-zG8w?e=LAHOqm

It repros using the built in Simple MA Crossover strategy, which is what was run in the video.

I'm having to restart NT after every 3 runs or so.

Thanks, Caesar.

[NinjaTrader_BrandonH]

Hello Caesar,

Thanks for your post.

Unfortunately, I am unable to view the video you shared because when I open the video it says "Sorry, there was a problem loading this video".

That said, make sure you are running the most recent version of NinjaTrader 8 (8.0.28.0 or 8.1.1.6). To find what version of NinjaTrader you are using, go to Control Center > Help > About. If you are not using the current version, please update your version of NinjaTrader using the link below.

https://ninjatrader.com/support/requ...desktop-users/

Computer memory is important for long resource-intensive tests that generate a large number of trades without your machine stalling. Computer processor speed is important to get optimization iterations done quickly and the number of CPU cores allows for more iterations to run simultaneously.

My colleague Jim has created a video demonstrating how the SampleMACrossover strategy, a simple strategy that utilizes IsInStantiatedOnEachOptimizationIteration=false for efficiency, can still quickly utilize memory resources.

Demo — https://drive.google.com/file/d/15pz...w?usp=drivesdk

As we can see there are a number of factors that are involved and memory utilization can climb very quickly depending on a few of these factors. Once memory gets maxed out, we can experience short freezes where memory is decommitted, stored to disk, and then new resources are committed before the backtest is resumed.

We can easily control the number of iterations involved and we can also consider writing our strategies to use IsInstantiatedOnEachOptimizationIteration = false; (Which requires that we reset class level variables in State.DataLoaded.)

See the help guide documentation below for more information about running optimizations.
IsInstantiatedOnEachOptimizationIteration — https://ninjatrader.com/support/help...niteration.htm
Optimization Tips — https://ninjatrader.com/support/help...ionPerformance
Walk Forward Optimization — https://ninjatrader.com/support/help...ss_metrics.htm
Genetic Optimization — https://ninjatrader.com/support/help..._algorithm.htm

Let us know if we may assist further.

[Skechers]

I have 8.1.1.6 64-bit the latest one.

To reproduce, simply optimize the Simple MA Strategy, that's how I created the video.

[Skechers]

I've converted the video to a GIF file. Hopefully that is accessible.

https://1drv.ms/f/s!AvRv1TeTT2RpgvJ5PyfT5TmZL-zG8w?e=79apcH

[QuantKey_Bruce]

Skechers I would offer the observation that Strategy Analyzer is not unique in this regard - .NET in general cleans up memory ("garbage collection") only when it must or when there is idle CPU time to do so. It does not clean up memory (dispose of objects no longer referenced such as old indicator instances, data series, etc.) just because an iteration is finished. I have a machine here with 128GB of RAM and I can run a Strategy Analyzer and use up all of it or almost all of it, and then after the run finishes, it will GC down to say 1GB of usage. The reason for this is NT does not force GC to run - the framework decides when to GC. That's not a bug per se - that's the design of .NET.

Now, that having been said, if you run SA and it NEVER collects the garbage and your memory usage is permanently higher, you would be onto something, but what I would do measure usage, run SA and then let it sit idle for say 5 minutes, then measure usage again. The difference should be the usage of the logs and UI components to show you the trade lists and results etc.

I chased this particular issue early on in the 8.0 beta period and while there were some memory leaks that NT fixed at that time, most of what remained was simply that GC is collected on its own schedule.

You can force GC to run by doing GC.Collect(); but I do not recommend it.

To clarify, are you saying that NT never releases the memory, ever? Like, if you shut down the SA window and leave the computer sitting there for 5+ minutes, it never returns to its pre-SA usage?

[NinjaTrader_BrandonH]

Hello Caesar,

Thanks for your notes.

Once memory gets maxed out, we can experience short freezes where memory is decommitted, stored to disk, and then new resources are committed before the backtest is resumed.

Optimizations in the Strategy Analyzer will use as many computing resources as possible to run the backtests. Strategy optimizations are expected to consume a good deal of CPU resources, simply due to the nature of the iterative data processing they perform.

Are you seeing that once your memory is maxed out that memory is not being decommited?

Does NinjaTrader never release memory as QuantKey_Bruce asked? Try shutting down the Strategy Analyzer window and wait for a little while. Do you see the memory usage go down?

[Skechers]

I'm in the middle of strategy testing at the moment, but will do the test as you both mention.

[Skechers]

So, I restarted NT amd my machine. The ,=memory conssuption is now less (~50-60%), but doesn't release (i.e. static).

I'm fine with this, it means I can run more optimizations before having to restart NT.

[QuantKey_Bruce]

Just for giggles, you could modify your strategy such that in State.Terminate it runs GC.Collect(); so you understand... I don't recommend doing this other than as an experiment... what you would be doing there is forcing the framework to GC all the time (every iteration on every thread) and your memory usage should accumulate only more slowly then, but the optimization would also run more slowly as well because it is running garbage collection too many times and way before it is necessary. The reason memory usage would still increase is SA is keeping track of the results, and the more iterations you run, the more results there are to keep track of.

The .NET philosophy is to run GC only when it is necessary or when the CPU is otherwise idle. This means memory usage of even a simple program can approach 100% of your available memory, if it is continuously busy doing something. That is not a a bug - that is how .NET is intended to function in order to maximize performance. It's only a bug if memory is "lost" and never released after it is done completely with its tasks (in NT, this would mean closing all the SA windows at a minimum and even then, cache data might still be loaded.)

[Skechers]

I tested GC.Collect() in State.Terminated. Optimization ran well at about 49% memory consumption. I had to close all apps, including Chrome which took up quite a bit of memory.

With GC the run is about 50% slower than normal, but that is fine as I don't have to repeatedly restart NT.

Many Thanks, Caesar.

[Skechers]

Bruce & Brandon, yes once the memory is consumed in an optimization, it is never released.

[Skechers]

Also, I use a Series variable:

State.DataLoaded:
TradingDay = new Series<bool>(this, MaximumBarsLookBack.Infinite);
TradingDay[0] = false;

Later on I use
OnBarUpdate:
TradingDay[0] = true; or whatever the value maybe

I test the TradingDay[0] and TradingDay[1] variable at every loop of OnBarUpdate

The trading range is 1 year (365 days).

Could this be the cause?

[QuantKey_Bruce]

Skechers

Well,

(1) I would take out the ", MaximumBarsLookBack.Infinite" - if you're only indexing back 0 and 1, you don't need that. Do the same for any other series unless you have a truly good reason. Better yet, don't use a series at all and just save the previous bar's value in a regular variable. Series have a lot of overhead, and infinite lookback is even worse.

(2) Second, I would investigate converting your strategy to https://ninjatrader.com/support/help...niteration.htm = false so you're initializing all of your variables in state.configure or state.dataloaded and nothing at instantiation. That will be a boost in speed and memory re-use by re-using the instances.

(3) Third, I would do the GC.Collect every N iterations. Make a static int for the iteration count, and a static object mutex = new object(); then lock(mutex) { iteration++; } if (iteration % 100 == 0) GC.Collect(); // GC.Collect every 100 iterations - experiment to find the best balance between memory usage and speed

(4) Fourth, if you don't need the trade history, turn it off: https://ninjatrader.com/support/help...inbacktest.htm

With these optimizations, you should be on a path to improvement. If you are using any nested indicators, you can consider inlining them.

[NinjaTrader_BrandonH]

Hello Caesar,

Thanks for your notes.

You could see the least amount of memory usage NinjaTrader uses on your PC by closing all open web browsers and applications and opening NinjaTrader Desktop with no open workspaces, charts, or other windows (only the Control Center).

This will give you an idea of what the least amount of memory NinjaTrader will use on your PC. Then, open a Strategy Analyzer window and note the memory usage. Run an optimization in the Strategy Analyzer and note the number of resources used.

Keep in mind that optimizations in the Strategy Analyzer will use as many computing resources as possible to run the backtests. Strategy optimizations are expected to consume a good deal of CPU resources, simply due to the nature of the iterative data processing they perform. Once memory gets maxed out, you can experience short freezes where memory is decommitted, and then the optimization will resume.

Next, close the Strategy Analyzer and let the computer sit for a little while. Note the memory resources should reduce over time as the computer sits.

To reduce the amount of resources used when backtesting, you should make sure to reduce the number of iterations involved (such as reducing the number of days being tested) in the optimization and also consider writing your strategies to use IsInstantiatedOnEachOptimizationIteration = false; (Which requires that we reset class level variables in State.DataLoaded.).

See the help guide documentation below for more information about running optimization performance tips and IsInstantiatedOnEachOptimizationIteration.
IsInstantiatedOnEachOptimizationIteration — https://ninjatrader.com/support/help...niteration.htm
Optimization Tips — https://ninjatrader.com/support/help...ionPerformance

[Leeroy_Jenkins]

Skechers how much gigabytes of RAM your PC have?