Default Optimizer can use unlimited memory and crash

[jmondanaro]

I have a strategy with several variables and I let it run with a wide range under the genetic optimizer. CPU utilization seems fine and memory doesn't really grow much. I assume the memory utilization would be proportional to the number of results to save. If I select 10 results, the total memory required should be enough to store the current best 10 runs to save plus N more runs being executed. I think N winds up being the number of cores on your machine. Confirm?

Anyway, I took a scenario that runs fine on the genetic optimizer and switched to default. I knew this would run for hours to go through all the permutations, but I didn't expect any problems. I have tried this repeatedly under 8.0.0.9 and 8.0.0.10 and the memory utilization is enormous and can get to the point where the machine runs out and starts to page, effectively killing things.

Why does the default optimizer use so much more memory? The only theory I have is that the Log and Output are somehow to blame. With the genetic optimizer, even if I have 10 generations and 100 as a generation size that's only 1000 total runs that are generating log and output entries. But the default optimizer does all the parameter permutations. So in the cases where the system dies, I'm talking about tens of millions of combinations.

The alternative is that the default optimizer has a memory leak in it or something. But I'd like to know if you've tried running massive combinations through the default optimizer and seen a similar result.

I'm running NT8 64bit on a machine with 32GB of Ram.

-Jason

[jmondanaro]

Found issue/solution?

So looking at the code for the optimizer, I see it is recursive with no throttling. So since I have a HUGE number of combinations it just runs out of memory launching all the iterations as fast as possible. It looks like you have a way to deal with this with the WaitForIterationsCompleted() method. By inserting this command into a copy of the Optimizer all the memory issues went away. Basically I added a parameter to tell how many in progress iterations can be queued up at a time before calling the WaitForIterationsCompleted(). Since this method isn't documented yet I'm not sure what exactly it does but this change makes it so the memory utilization of the Optimizer is now flat or proportional to the Results to Keep parameter.

if (index == Strategies[0].OptimizationParameters.Count - 1) // Is this the Last parameter -> run the iteration
{
if (currentlyRunningIterations >= MaxParallel)
{
Print(String.Format("{0} : Waiting, On Iteration {1} of {2}", DateTime.Now.ToString("yyyy-MM-dd hh:mm:ss"), IterationCount, NumberOfIterations));
WaitForIterationsCompleted(); //Assume this is a blocking wait
currentlyRunningIterations = 0;
}
IterationCount++;
currentlyRunningIterations++;
RunIteration();
}
else
{
// Iterate next parameter in line
Iterate(index + 1);
}

[NinjaTrader_Jesse]

Hello,

I just wanted to reply and let you know this question is being reviewed. Once I have further information I will reply back.

I look forward to being of further assistance.

[NinjaTrader_Jesse]

Hello,

Thank you for providing the information you already have on this subject.

Using the information here I was unable to piece together the amount of combinations needed to directly show this.

I wanted to check if you are able to provide a specific example that demonstrates the difference between the two optimizations, mainly just a sample strategy that I could provide to development for review.

With a sample that demonstrates the combination set needs for this event, we should be able to provide that to development for review on the differences here.

I look forward to being of further assistance.