Optimizer and system resources

[vides2012]

Hi!

I've downloaded and installed the beta in hope of better optimizing performance. I set up an optimization just to test it, and started it. The first thing was that the application GUI went unrespondive , so the main thread was involved in the calculation process, in which case the app isn't quite well multi-threaded. The other thing was the memory consumption. It consumed all available from my 4G! It actually slow down my whole system that my OS was swapping all the time. The worst was that Ninja did not use all available CPU resources (probably beacause of waiting for the HDD) and thus the optimization was quite slow. I had to abort it from my process explorer ,because the GUI was still unrespondive.

I've attached a screenshot of the application's resource consumption.

edit: I am using Windows 7 64bit. AFAIK in this version this OS is supported.

http://www.imageserve.info/img_store/2010/04/28/3/3cb9191c7e9a0fbc4661af044052abfe.jpg

[NinjaTrader_Jason]

Hello vides2012,

NinjaTrader 7 comes with optimized multi core/multi-processor support. Please see the link below for more information.

http://www.ninjatrader.com/webnew/NT7/NinjaTrader7.html

General Application Performance
This is an area that we spent a lot of time on during the development of NinjaTrader 7. We re-wrote core sections of the application to be more efficient in terms of memory usage and CPU utilization.

However, due to the nature of the Window OS, NinjaTrader 7 would not be able to fully use all available multi-cores/multi-processors at any given time.

[vides2012]

Hello Jason,

I've read the document. I wouldn't say consuming all available memory is efficient since it causes heavy swapping (thus reducing system performance largely, including the optimization performance). I suggest a lower memory consumption.

About the multicore usage: The CPU utilization kept below 20% after a couple of minutes (when the memory usage reached its max, and the swapping began). I wouldn't say it's about the OS. I suggest a configuration setting available to the optimizer algorithm, so I can maximize the memory usage, and determine the no. of threads.

Please revise the optimizer, that the main thread should not participate in the calculations!

[NinjaTrader_JoshP]

vides2012,

Thank you for the post. Can you please provide us with the optimization settings you used, dates tested, time frame used, and strategy? Thanks.

[vides2012]

Dear Josh,

Unfortunately I may not give you the strategy I was using. It uses some drawing methods (besides the trading logic) can it be a problem? It wasn't with NinjaTrader 6.5. I was using minute bars (15 value). The date was from this year January till today, on a custom ticker. I was using the default optimizer, exit on close : false, keeping 10 best results, optimizing on net. profit.

[NinjaTrader_JoshP]

vides2012,

In that case can you please try with SampleMACrossOver? Thank you.

[vides2012]

I've tested the SampleMACrossOver and it was working. So I started to figure out what the source of the problem was. I wrote some snippets to a copy of the SampleMACrossOver, and finally I have found the cause of my problem!

It was a debug Print() call! So I have removed it, and now my strategy works.

Please inform people that the Print method should not be used when optimizing a strategy, or please make it compatible with optimizing.

Thanks!

[NinjaTrader_JoshP]

vides2012,

There should be no issue with having Print() for optimizations as a concept. If you have the Print() in a location that would print many many lines of output then running an optimization would create a very very large requirement of memory and CPU cycles to print all of them at once.