Historical data import - leaking memory

[tradejockey]

Just one quarter of data is leaking as much as 200 megs of memory... Need to make this a bit more robust so that large files can be loaded.. export doesnt seem to have this problem. Just the import process. Also, please ensure that large files (worth several years) can be loaded as well.. this might come in handy when I actually buy tick data.

Easy to reproduce as well. Just export out a file and try to import it back while watching the memory.

[NinjaTrader_Dierk]

Not sure what you mean by "leaking". Obviously NT needs to allocate memory as you import data. That's exactly what you see.

[tradejockey]

Here are the steps to replicate:
1. Goto Instrument manager and select one of the highly liquid index futures. Set the expiry to ##-## and add it to your list on the left.
2. Export out last four quarters of 2009.
3. Import each one after renaming the file's expiry to ##-##. For example ES 1209.Last.txt to ES ##-##.Last.txt and then import. Check the two boxes on the import tab.
4. Watch the task manager for memory (both in processes and performance tabs). It will not be released. This might make it difficult to import large files.

[NinjaTrader_Dierk]

.NET runtime is a garbage collection system -> monitoring RAM in task manager is no indication of memory leaking

[tradejockey]

Please execute the steps listed below on xp and draw your own conclusions. App shuts down after importing few quarters worth of data. Generally using products such as Boundschecker or Purify would pin point the problem.

[NinjaTrader_Dierk]

You are correct in your findings: NT does consume RAM as you would import tick data and have "generate minute bars" or "generate daily bars" checked. This is a bug which will be fixed with next update.

Thanks for pointing that out.