RAM build up causes corrupted charts

[Harry]

Relates to NT 6.5.0.6 not connected to any data feed
Runs on Xeon 5140 / 2 GB RAM / Windows XP
Workspace contains Market Analyzer with 30 instruments, DOM and 3 linked charts
Monitoring Memory and Swap File used by NinjaTrader with the Windows Task Manager

Ninja starts. Memory use by NT 150,000 k. When I scroll through the charts, memory required by NT builds up to over 550,000 k and stays at that level. The swap file also increases from 600,000 K to 1,100,000 K.

When NT required memory exceeds 500,000 k, some charts show false volume and price spikes. Cannot be removed by reloading data or chart templates.

Only thing to do: close NinjaTrader and restart. All the charts are fine again. As memory utilisation gradually builds up with each chart viewed, corrupted charts will reappear.

The whole problem looks like a garbage collection issue. As this is done automatically by .NET environment, some hidden Ninjas are possilby processing the chart data, even when the chart is no longer viewed, thus preventing C#/.NET do to the clean up. This morning I had a similar problem, when NT loaded daily data, although my workspace only contained intraday charts (but a closed workspace simewhere contained daily charts, encouraging NT also to upload daily data).

By the way, I do not think that the problem goes away, if I upgrade to 4 GB RAM, as I still have about 800 MB RAM spare capacity, when NT memory requirements run high.

Does anybody have any suggestions?

[Harry]

RAM build up is a specific BETA problem

I just had the same workspace with the same charts and the same data tested on Ninja Trader 6.0.1000.8.

RAM utilisation by NT stayed under 200,000 K and there were no problems scrolling through the charts.

[Pete S]

Not commenting on the specific issue, but if you want to quote memory usage numbers, you'd be better off using Process Explorer, which can replace task manager and has a bunch of features related to .NET, including heap tracking and garbage collection:

http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx

[Gains]

I just had the same workspace with the same charts and the same data tested on Ninja Trader 6.0.1000.8.

RAM utilisation by NT stayed under 200,000 K and there were no problems scrolling through the charts.

Thanks for doing this test Harry.
I reported a similar problem on Vista but there seemed to be a reluctance on the part of development to believe that there is something wrong with Ninja.

http://www.ninjatrader-support.com/vb/showthread.php?t=4909

I was told in effect...
.NET handles garbage collection therefore any issues I was having was a .NET problem, contact Microsoft, case closed. Not what I expected to hear!

I'd say that your testing on Ninja 6.0 proves that obviously something has changed in Ninja 6.5, perhaps the much needed workspaces are causing a side effect? Perhaps each time a workspace is viewed it is granted new resources while the old are not released? Perhaps .NET is not taking away the garbage because it doesn't not see the "garbage"? Just trying to tigger some thinking outside the box here! ;-)

Hopefully support will now address the issue especially since your symptoms were observed on XP.

Thanks again Harry, lets keep after them. :-)
Gains

[zoltran]

Harry . What does your Market Analyser look like? Do you have any indicators columns? IF so, what time frames.. tic, minute etc?

thks

[Harry]

Zoltran, the market analyzer is pretty simple, just Bid Price, Ask Price, Last Price, Net Change and a Full Stochastics on 10 Min. data. I only added the indicator to test alert and color conditions.

[Harry]

Pete, thanks for your recommendation. I installed process explorer, and it allows to compare the performance graphs for NT Beta 6.5.06 and NT 6.0.1000.8:

For exactly the same workspace with the same templates and the same data stored

- NT 6.0.1000.8 will use up to 190,000 k, levelling out at 160,000 k
- NT BETA 6.5.06 will use up to 620,000 k levelling out 420,000 k

On the graph attached you can see, how NT BETA gradually feeds its hunger for memory, while I am scrolling through my charts! In the end it has grown too fat to display the charts properly!

[NinjaTrader_JoshP]

Hi Harry,

Can you do this with only one chart open? No Market Analyzer or DOM. Also, which indicators do you have on your chart? What is the lookback period and time frame of your chart?

Please try this with no indicators. Default clean chart. Thanks.

[Harry]

Hi Josh,

Thanks for replying. My workspace contains three charts - 1 min, 10 min, 60 min, lookback period 75 days, DOM, T&S, Control Center.

Everything is fine, if I take away the market analyzer, so this makes it clear that the market analyzer is the problem. So I deleted the market analyzer, and created a new one. The new market analyzer would not cause problems, unless I used my standard market analyzer template.

The template was automatically imported from NT6.0.1000.8 when installing the new BETA. This template works fine on NT6.0.1000.8!

Is there any incompatibility between NT6.5.0.6. and the templates of the current NT version?

[NinjaTrader_Dierk]

Let's see what we've got: you're saying that as you compare NT 6.1000.0.8 and 6.5.0.6 with only CC open and one market analyzer (where you applied your custom template), then NT6.5 would exhibit excessive memory consumption as per below.

Also: You checked that you did run your comparison on the same data sets, meaning NT6 and NT6.5 hold the same data set.

If that all is true then could you please mail me to "dierk AT ninjatrader DOT com":
- your zipped NT6 DB
- your NT6 MA template
- a list of instruments you threw on the MA

I suppose you observed this behavior even as your are not connected, correct?

Thanks in advance

[Harry]

Hi Dierk,

maybe I have not been all clear:

My default workspace contains CC, T&S, DOM, 3 charts and the market analyzer. The market analyzer template is imported from NT6.0.1000.8. Now lets look at the different cases

CC, T&C, DOM do not at all interfere with the problem
3 charts alone: no problem
market analyzer alone: no problem

A problem will be there if I add the market analyzer (template) to one of the charts. For example if start with my default configuration, close DOM, T&S and 2 charts, thus only keeping the 60 min chart and the market analyzer, this will be enough to build up the memory.

If I use MA and only 1 chart, memory will build up to 400,000 k, with 3 charts up to 600,000 k.

So there is some sort of unwanted interaction between the chart and the market analyzer (template imported), interaction that does not take place with the current version of NT.

All this was obsxerved when not connected. The screenshots from processs explorer also were taken, when not connected.

I will send you NT DB, MA template and instrument list via Mail.

And, thanks Dierk for your attention and your work. I know that NT is quite an advanced concept, so if I am writing here at the BETA forum, it is just because I want to participate in making NT better, not at all to complain!

[NinjaTrader_JoshP]

We appreciate your efforts in this matter Harry and will see what we can do about the issues at hand.