Strategy Analyser eating memory

[MarketAlly]

I find when I am running 4 strategies (of my custom indicator) at once - mine will cycle between 1Gig and 6Gig but it will reset itself back to 1Gig during the process many times. To execute all 4, they will run for about 12 hours in my case (which is typical for my indicator to process months of data)

Garbage collection in .NET is delayed - so it may hold resources longer than you might expect or differently in NT7 because of the differences in .NET 3.5 and .4.5

Here isa video of it in progress - the garbage collection is delayed. I could put a profiler on it to show you where it releases my objects but I am not sure there is any issue.

http://screencast.com/t/u99OzRRdmi2

[starcd]

That may be all true, but at the moment the hole strat analyser on my system is a memory monster compared to NT7. Especially when it comes to smaler timeframes and baskets of instruments.

It uses Factor 10 Times more memory
It crashes
If not NT8 is getting slower and slower after a couple of runs
It creates tons of XML log files after evry run. I just deletet 15000 of those after 2 Hours of backtesting
It's missing features for basket backtesting

[NinjaTrader_PatrickH]

Hello starcd,

Thank you for your patience.

MarketAlly is correct on this matter for .NET, to release the memory restart NinjaTrader 8.

Are you seeing those same performance issues after restarting NinjaTrader and only running one backtest for strategy?

[starcd]

Memory is released after restart (since beta3) and is still increasing from backtest to backtest. Sometimes it decreases after two or three backtests but not to the mem val it was before running the first backtest. So it eats and eats until it crashes or it is useless slow. Maybe it's only my system. Did you try the config I send you (see pics)?

[NinjaTrader_PatrickH]

Hello starcd,

I have not tested a config as you suggested.

Please confirm you do not see this issue on the inital backtest after restarting.
In addition, are you changing code in-between each backtest?

[starcd]

There is no memory release after the 1st backtest.
I'm not changing code.

[happypappy]

I have noticed this as well, running a year of data and leaving the strategy analysis window open resulted in over 6GB of memory being used by the application. It runs terrible until the window is closed.

There is no memory release on closing the window.

Attempting to rerun results with the existing results left open resulted in seconds of replay data at max speed.

May I suggest adding a 'Refresh' button to this window and only have it populate data when clicked rather then attempt to read data on the fly?

[NinjaTrader_PatrickH]

There is no memory release after the 1st backtest.
I'm not changing code.

Thank you for confirming that the memory is not released. We are aware of this and it is due to .NET and how it releases memory.

More specifically, does the system lock up /freeze after or during the first backtest after restarting? If it doesn't freeze or lock up, does the memory leak occur? Does the memory continue to climb in the Task Manager during or after the initial backtest?

In addition, can you send over your Log and Trace files to platformsupport[at]ninjatrader[dot]com with this thread in reference?

• http://www.ninjatrader.com/support/f...ad.php?t=75932

You can do this in the platform by going to Help > Email Support.

[NinjaTrader_PatrickH]

I have noticed this as well, running a year of data and leaving the strategy analysis window open resulted in over 6GB of memory being used by the application. It runs terrible until the window is closed.

There is no memory release on closing the window.

Attempting to rerun results with the existing results left open resulted in seconds of replay data at max speed.

May I suggest adding a 'Refresh' button to this window and only have it populate data when clicked rather then attempt to read data on the fly?

Hello happypappy,

Thank you for your note.

• Can you answer these questions as well?
• Does the system lock up /freeze after or during the first backtest after restarting?
• If it doesn't freeze or lock up, does the memory leak occur?
• Does the memory continue to climb in the Task Manager during or after the initial backtest?

In addition, can you send over your Log and Trace files to platformsupport[at]ninjatrader[dot]com with this thread in reference?
You can do this in the platform by going to Help > Email Support.

The Reload All Historical Data option does just that which means the NinjaScript objects need to re-calculate over the data.
Reload NinjaScript results in removing and adding back the objects, this will mean the objects must run over the data again.

With those in mind, for your suggestion in the Strategy Analyzer you want the data to load but not have the NinjaScript objects run over the data until you select the button?

[happypappy]

@NinjaTrader_PatrickH

Mate Ill experiment with it more tonight / this weekend.

1) Ill run a year of data with no strategy and leave the form open and see if the form itself is causing whats occuring.
*This did not consume a massive amount of memory* Infact I run an entire year with 2 charts NT only getting to 240MB of used memory.

2) Ill run a year of data with a strategy that just enters once and holds with the form open.
This did not cause the issue.

3) Ill run a year of data with a testing strategy, but have the window closed, then generate the data at the end, see if it uses a lot of memory.
This did not cause the issue.

4) Ill run a year of data with a testing strategy, but have the window open. To determine if it's heaps of entries causing the problem.
This did not cause the issue.

Ill let you know the results. To me it appears that if the Window is open it is consistently interigating for results, even if the user has not clicked the Generate button. That would mean the more entires etc while it's open the more memory it uses.

My testing didnt really help. I have seen it before, if it happens again Ill screenshot it. But using the SMA strategy that comes with the application I was unable to reproduce it.......

[happypappy]

I was able to reproduce the memory hog, if the strategy has an output and I leave the output window open the memory's gets smashed.

[sledge]

I was able to reproduce the memory hog, if the strategy has an output and I leave the output window open the memory's gets smashed.

Is it outputting that much text to eat up memory? What I mean is the amount of memory that's getting smashed the same amount in the output window? or is it totally not proportional? Like 200k of text is consuming 3 gb of ram?

[happypappy]

Nah it's not a vast amount of text. Memory on the other tests sat around 200-250MB, this had about 1000 lines of text and was 3+ GB. Ill do some more test tonight.

[Icehouse]

I'm seeing this too, while backtesting a simple strategy using managed orders. I'm on the most recent build of NT8 (8.0.0.6), 64-bit.

What's also interesting is this: watching memory usage through Task Manager, once an Optimization has finished running and the Strategy Analyzer (with results) is sitting idle, memory usage by NT is at 1.7GB (for example - I've seen it higher with other optimizations). Then, when I launch another optimization the memory usage immediately drops to ~500MB, then steps back up as the optimization runs.

Hopefully that provides a clue. When NT is sitting idle after optimization, with that 1.7GB memory footprint, it's really sluggish (and I don't have any other NT charts or windows open)... as in, slow to respond to clicks or moving windows, etc.

I hope this helps track the issue down, so it can be fixed.

[NinjaTrader_PatrickH]

Hello Icehouse,

Thank you for your patience.

Do you see the same behavior with the SampleMaCrossOver strategy alone?