NT7 memory consumption

[astra]

Unfortunately there only are a couple of weeks in that file. Sorry guys, but I can't denote any more time on dealing with file handling issues: Whitmark, please let me know as you have a file readily available. Thanks

Dierk,

Check the latest email with the link to yousendit.com. You can download it from there for the next 7 days. I send it to support... att: Dierk. Cheers,

astra

Edit: When unzipped you should get a 215.224 KB file: ES: June 2nd 2008- June 5th 2009.

[NinjaTrader_Dierk]

This is the file I downloaded from the yousendit.com link (215 MB). Here are the last few lines of that file as I opened it in notepad:
...
20080912 203110;1252.5;20
20080912 203110;1252.5;4
20080912 203110;1252.75;1
20080912 203110;1252.75;3
20080912 203110;1252.5;1
20080912 203111;1252.75;15

Sorry, due to bandwidth limitations I can't denote any more time on file handling issues...

[whitmark]

Sorry, Dierk, thought this would be more straightforward. I'll take care of it and get it to you.

[makra081]

re

1. open new chart

2. put only bars on it

3. select cross mouse tool

4. move your mouse constantly in cycle.

5. after some seconds the mouse will stop and hook & move & hook & move & hook.

nice memory leak hey ?

Its getting worse if you add some indicator

[astra]

Just FYI (ie, if someone else tries to export large historical data) the process with NT6.5 Historical Data -> Export is not not so straightforward!

What I have discovered (the hard way...) is when memory peaks to the threashold of 1.52GB aprx, NT6.5 will prompt you that the export was successful and sends a file, but this file does NOT represent the whole period you requested, but only the period it manages to export before the limit of 1.52GB memory is reached. That's why my 215MB file I sent represented only the first 15 weeks of data. Eventually, one year's ES tick data was 1.3GB total!

So, be cautious when you export data to free space in your database to deal with the other NT limitation of 2GB total database limit, to export small files up to 200MB and to always check the file if has the whole period you requested (recently I half my DB from 2GB to 1GB doing that with over 50 stocks of 1 year tick data and I after today I dont have the courage to check what I have done...)

Anyway, 8 zip files with 1.3GB total data are now with Dierk and is up to him to judge how to proceed. From my side, and I believe (from our communication) whitmark's as well, I can't think of another way to send the data in one file with NT6.5. I hope NT7 addresses this problem as well.

astra

[NinjaTrader_Dierk]

Thanks for your contribution. I added the results to my original post.

[whitmark]

Setup4, export 1 year of ES continuous contract tick data
NT6.5: User reported was not possible and only could be done in chunks of 1-3 months
NT7: RAM usage < 500MB, took ~5mins

Setup5, backtest of SampleMaCrossOver on 50 ticks ES series, 2008-06-01 through 2009-06-09:
NT6.5: User reported was not possible due to OutOfMemory exception
NT7: RAM usage < 140MB, took < 3mins

More impressive results! Many thanks, Astra, for your persistence in getting the data out of NT6.5 and onto Dierk. The reward for your hard work is an additional test to prove NT7 is up to the task for exporting. (Unclear if an import of the ~1.3 GB file exported by the NT7 export test would yield similar results as large imports in NT6.5 are known to fail)

The results of Setup5 seem to indicate there is plenty of headroom for multiple years of data. If you can share your files, Astra, and if Dierk is willing to test it, I may be able to create multiple previous year test files by modifying the year field on each record.

Whitmark

[ctrlbrk]

More impressive results!

The results of Setup5 seem to indicate there is plenty of headroom for multiple years of data. If you can share your files, Astra, and if Dierk is willing to test it, I may be able to create multiple previous year test files by modifying the year field on each record.

Whitmark

I agree! I love this thread. It makes me happy.

Also, with NT7's largeaddress patch built-in, it automatically doubles the headroom from 6.5 anyway. So you take that and also take the improvements in memory management, and we have ourselves a winner.

I am also particularly impressed with the performance aspect. Dierk, would you mind telling/showing us what your TaskMgr looks like during these backtests -- ie cpu usage?

Mike

[NinjaTrader_Dierk]

Thanks for your support guys

@whitmark: Let's leave that setup for now. It would not yield new information beyond what we already know. Any further research would dependent on the user actual setup anyway. I'll leave that to the beta testers.

[NinjaTrader_Dierk]

CPU usage: the details of what's going on under the hood needed to be understood in order to make a proper judgement. I'm willing to expose more info as we actually rolled first beta, but would not want to start something now (it's just premature).

[xTrader1]

Thanks for your input. You probably are aware of that thread: http://www.ninjatrader-support2.com/...ad.php?t=17699

Please go to my initial post: if you would be able to provide all the info requested by that post, then we would be more than happy to test how your setup would perform on NT7. Thanks in advance

Dierk,

As you referred me to this thread, I'm not sure I qualify for NT7 setup check, because NT does not blow up yet , and I have proprietary indicators, programmed in NinjaScript I'm not interested to post.

I think that an issue I raised in http://www.ninjatrader-support2.com/...444#post102444 is not a bug, but a software architecture issue. In any case, nothing can be done, but to wait for NT7 beta and then to complain, if needed.
Thanks,

[NinjaTrader_Dierk]

Sorry, if my post was not clear. As you stated you feel NT's memory consumption should be much less. We are happy to test if NT7 actually would be more efficient on your setup if you could provide the information required.

[eDanny]

Maybe running the benchmark strategy, http://www.ninjatrader-support2.com/...ight=benchmark, on the same machine with NT6 and NT7 might yield some interesting results for us.

[NinjaTrader_Dierk]

I don't think so. The benchmark rather is a CPU/machine benchmark than a "NinjaTrader" benchmark. Please contact ctrlbrk (the benchmark author) for details.

[eDanny]

That is correct. Isnt NT7 supposed to be a little more CPU friendly so to speak?