Optimizer - Save key results into file instead of memory

[KirkHammett]

Hi!

I am optimizing on a lot of data and constantly running out of memory which causes NT to crash. I followed already all optimization tips on the help guide and i am have instantiate on each iteration set to false. It is simply not enough with my 32 GB of RAM. The genetic setting is no option for me.

Currently i split the optimizations and export the optimization results into an excel and then compile them into one file and analyze them which is very burdensome.

I basically want to run a lot of iterations on my strategies on a lot of data which would take many hours or even days however saving the results to a file in order to not run out of RAM and therefore causing NT to crash. I don't need all the trading data (like all trades) saved. I just want to save certain key stats from each iteration (eg. net profit, max drawdown) into a file with the StreamWriter, which I can then analyze further outside of NT8. I don't even need the optimization function (eg. optimize on max. net profit) because when I have all key results saved to a file i can sort them afterwards.

Is there any (unsupported) way I can do this? I would very much appreciate any help you may provide.

Thanks in advance.

[NinjaTrader_Jesse]

Hello KirkHammett,

There is no way to save the optimization data to file during the test, taht all runs using ram. If you are running out of ram you would need to isolate what part of your script is taking a lot of memory, that could include items like drawing objects. Alternatively you would need to use less changes to the parameters to create a smaller result pool or decrease the amount of time being tested over.

NInjaTrader will not crash when it uses all the ram, that enters what is called paging in windows which will make everything much slower in both NinjaTrader and the PC. It will appear that the platform is frozen and you wont be able to click it without getting the close program dialog, that is a windows feature and is not related to NinjaTrader. To avoid that you would want to decrease the test size, reduce the parameters being optimized or change the code so it is not using so much memory. You can comment out your code to find which part is using the most memory and then correct that code so it is more efficient to avoid that.

[giogio1]

Thank you Jesse,
this is what we can do on our side. What Ninjatrader can do on their side is to reutilize or deallocate the population objects as they die so memory doesn't need to increase at infinitum with population and generations.

I also noticed that after an optimization Ninjatrader does not release the memory so tu run another one I need to close the platform and restart it.
Is this done intentionally? It seems to me now because when I look at the different populations I still need to run a backtest to see its chart. Am I correct or is just bad?
Best
G

[giogio1]

I'd like to add one note. Respectfully. I switched to Ninjatrader8 expecting a good genetic optimizer and I was quite surprised to see where it is at.
It is, in my opinion, and I am not very knowledgeable, very well written, very well structured. The whole work being done in Ninjatrader is excellent and in some places work of geniuses. And yet I think Ninjatrader is taking a chance by underestimating the importance of a good optimizer. A swarm optimizer should be included too.
Obviously I don't know much about the myriads of desiderata that people are submitting but It seems to me this is not matter that should be left a vote based on enhancement request submissions.
Ninjatrader may need to be more proactive on this. I cannot believe that I've been looking for other platforms despite all the investment of time I have in Ninjatrader code. It is still the bet in so many areas thanks to its flexibility and Object Oriented code. I am really hoping Ninjtrader is working already in this direction.
Best
G