Memory Leak or poor garbage collection?

[adrian]

After an hour and a half of running backtests, the VM size of NinjaTrader has grown from about 47 Mb to 643 Mb. I shut down NinjaTrader and ran a few backtests using data from 1 Jan 2008 - 1 Apr 2008, noting the memory usage:

startup: 50 Mb
Open a strategy into the editor: 84 Mb (!!!!)
Compile: no change
Run backtest (12 stocks, 3 mo of 5min data): 157 Mb
Comment a single line and recompile: 160 Mb
Re-run backtest with same data: 214 Mb (peaked at 240 Mb)
Uncomment the same line, recompile and re-run: 230 Mb (peaked at 340 Mb)

After an horu or two of use, the memory has gobbled up more than half a Gb and is killing my machine so I have to stop & restart. I watched the VM Size and the Mem Usage to see if the memory would be free()'d or reclaimed but I never noticed it. During backtesting, I would see fluctuations of over 100 Mb but this would get freed pretty quickly, but it seems like some gets left over.

I'm running NinjaTrader 6.5

[NinjaTrader_Dierk]

- please make sure you are on latest 6.5.1000.1
- please make sure you understand the concept of how .NET runtime manages garbage collected memory. Please consult the Microsoft documentation for details: actual mem usage indicated on the task manager is no indication of the app not freeing memory
- please try isolating any potential coding issue in your custom strategy by runnings test like these with NT standard strategies like SampleMACrossOver only

[adrian]

- please make sure you are on latest 6.5.1000.1

Check.

- please make sure you understand the concept of how .NET runtime manages garbage collected memory. Please consult the Microsoft documentation for details: actual mem usage indicated on the task manager is no indication of the app not freeing memory

Why, will this reduce the memory NinjaTrader is using? I understand that using a managed runtime means you're deferring some garbage collection issues to the .NET runtime, but I'm regularly having NinjaTrader exceed 500Mb and 600Mb (I've never let it run long enough to see it grow larger). Researching this sounds like a good task for someone with access to the source code, not an end-user.

(If you would like to give me access to the source code, I won't object )

- please try isolating any potential coding issue in your custom strategy by runnings test like these with NT standard strategies like SampleMACrossOver only

Sure.

With SampleMACrossover:
At NinjaTrader startup: 48 Mb
After run from 1 Jan - 1 Apr: 79 Mb
Open SampleMACrossover editor: 102 Mb
change 'Orange' to 'Red', recompile and re-run: 120 Mb (high of 150 Mb)
change 'Green' to 'Yellow', recompile and re-run: 130 Mb (spike to 180 Mb)

I've no idea what I did to get it up >600 Mb, but I'm sure if I bothered to write down every action during a day, it would happen. Even using the same data-set, memory is growing.

(If it helps, the stocks in my backtest are: AAPL, AMZN, BIDU, EOG, FSLR, GOOG, ICE, ISRG, JASO, MA, MON, MOS, MTL, PCLN, RIG, RIMM)

Hope it helps.

[NinjaTrader_Dierk]

>> I'm regularly having NinjaTrader exceed 500Mb and 600Mb
This is not unusual and could be even higher dependent on actual setup.

a) what's the physical RAM of your machine?
b) how much is still available as you see 600 MB consumed by the .NET runtime?

[adrian]

>> I'm regularly having NinjaTrader exceed 500Mb and 600Mb
This is not unusual and could be even higher dependent on actual setup.

a) what's the physical RAM of your machine?
b) how much is still available as you see 600 MB consumed by the .NET runtime?

Yoink! Then it looks like you already know about the problem and it's not just me.

a) 1Gb RAM
b) I don't know, I didn't check the 'available'. I went into Task Manager and added Mem Usage and VM Size (a better representation), and both were >600 Mb.

From what I understand, .NET can report more memory than is being used in the 'Mem Usage' column, and it will free this to other applications on demand, but VM Size is what's actually being held and won't be released and that's what I'm quoting. My machine was down to a crawl so if there was any memory to be freed, it would have done so.

[NinjaTrader_Dierk]

We are not aware of any problem related to mem consumption on 6.5.1000.1.

I suggest optimizing your setup as per here: http://www.ninjatrader-support.com/H...anceTips1.html

... and then trying to reproduce using SampleMACrossOver only.

[adrian]

We are not aware of any problem related to mem consumption on 6.5.1000.1.

I suggest optimizing your setup as per here: http://www.ninjatrader-support.com/H...anceTips1.html

... and then trying to reproduce using SampleMACrossOver only.

I don't know anyone else running NT so I can't ask them and I don't know how others use it. I can just tell you that when I run with 6.5.1000.1 the memory usage can spiral out of control and seriously degrade my computer's performance. I need to restart NT every couple hours when I'm running a lot of backtests.

I looked over the Performance tips and they are either not applicable or already done. The test results using SampleMACrossOver were run with NinjaTrader tuned as you suggest:


Charting - not applicable. I don't have any charts open and left the strategy analyzer on 'Summary'

Market Analyzer - not applicable.

Misc - already done, but since I didn't connect, it isn't applicable.

Time and Sales - not applicable

Level II - not applicable

Diagnostics - already done.


BTW: thanks for helping diagnose this with me.

[Elliott Wave]

In the context of my own ignorance on the subject, I believe many performance issues are related to the .NET platform, rather than any specific shortcomings in the programming abilities of the NT dev team.

A program like Advanced GET for example (written in the 80-90's by Tom Joseph, peace be upon him.) can load many charts with every indicator in milliseconds, whereas NinjaTrader despite its advantages feels very slow and sluggish in comparison.

As NT7 comes to fruition and more efficient multiprocessor utilization becomes more than a post it note on Ray's monitor (kidding of course) performance should be less of a concern...

[KBJ]

I just upgraded from 2GB to 3GB about a week ago and saw an enormous performance gain. I was having NinjaTrader hangs before this (and I really heavily load my system) and none since.

Spend the $20 that it takes to add 1GB of memory to your computer. It's just not expensive anymore and only takes a couple of minutes to plug in once you have it.