System.OutOfMemory Exception

[Stephan123]

Hi,

I tried a walk forward optimization with the genetic algorithm.
During the procedure I got several out of memory exeptions and had to close the program.
How can I avoid this?

I use 3GB RAM with 2,5GB free memory, work with daily data from 1983 till now
and used the default settings for the GA optimizer.
I tried to optimize 2 parameters, the first with a range from 0,1 to 1 and an increment of 0,1. The second one with a range from 1,1 to 10 and the same increment as the first one.

Is this normally done by NT without memory problems?
If yes, which script/ which parts in the script I might check for
extraordinary memory consumption (indicators, strategies, Initialize, OnBarUpdate...)
Are only those scripts using memory which are directly called within the strategy beeing optimized?

Thanks, Stephan

[NinjaTrader_RyanM1]

Hi Stephan,

The limit you're running into is a Windows memory limit per application. This memory limit is increased when using 64 bit compared to 32 bit.

NT handles most of the resource clean up but if your script uses custom resources, you may need to handle these through the strategies OnTermination() handler.

http://www.ninjatrader.com/support/helpGuides/nt7/ontermination.htm

Only the strategy you're currently running will be loaded in memory. OutofMemory could be caused by too many tests and/or improper custom resource management.

There are performance tips you can use to reduce NinjaTrader's resource requests. Have a look at these if you are doing other things in NinjaTrader at the same time as optimization.

http://www.ninjatrader.com/support/helpGuides/nt7/performance_tips2.htm

[Stephan123]

I'm running Windows XP Home Edition, I think 32Bit.
How many GB is the per application memory limit?
Can this be somehow shifted to higher values ?

During the optimization I'm doing nothing else.
What do you mean with "OutofMemory could be caused by too many tests " ?
I'm doing the optim on daily data from 1983 till now
using 250 days for optimization period and 80 days as test period.

It's still not exactly clear to me why this hang up actually occures.
When I do ordinary backtests on that strategy everything works fine.
Since during optimization the very same strategy is run many times, there shouldn't actually be any differences in memory consumption.
Are there any differencies in memory usage between backtests and
walk forward optimization I'm not aware of?

I'm using custom indicators within that strategy that open txt-files, and read data from it into the memory . Are those data removed from the memory by NT when the strategy is terminated or do I have to script this individually?
Are there any examples of what to script within OnTermination() ?

In general, using the above mentioned settings, can this out of memory exception
be avoided at all?

Thanks, Stephan

[NinjaTrader_RyanM1]

Memory limits are detailed here:

http://msdn.microsoft.com/en-us/library/aa366778(v=vs.85).aspx

The only way to increase is to use a 64 bit system.

Too many tests refers to the total number of tests run, based on the exact range and # of inputs you're optimizing.

If you are writing to text files, this is done separately for each test, and may be the source of your resource issues. If you're using stream writer, you can see this reference sample for technique on cleaning up its resources. See the OnTermination() section.

http://www.ninjatrader.com/support/forum/showthread.php?t=3475

[Stephan123]

Is there a rule of thumb for the number of tests that can be run without memory issues?
Again, where does the memory consumption come from when running a high number of tests? When memory is cleaned up after every test there shouldn't be any problems
with high test numbers.

I read from txt-files using
string[] fileContent = File.ReadAllLines(pfad);
Afterwards I read fileContent into an 2*2 array.
Both is done within OnStartup() of the indicator by a userdefined method.

The 2*2 array is used within OnBarUpdate() of the indicator.
Is there any handle/variable/memory space that has to be cleaned up manually
at termination in that case?
Thanks, Stephan

[NinjaTrader_Bertrand]

Stephan123, unfortunately there's no rule of thumb involved here as the performance on various setups used would differ, if you're using the Stream objects - are you properly cleaning up custom resources in the OnTermination() as Ryan mentioned?

Thanks,

[Stephan123]

I was just editing my last post.

I read from txt-files using
string[] fileContent = File.ReadAllLines(pfad);
Afterwards I read fileContent into an 2*2 array.
Both is done within OnStartup() of the indicator by a userdefined method.

The 2*2 array is used within OnBarUpdate() of the indicator.
Is there any handle/variable/memory space that has to be cleaned up manually
at termination in that case?

[NinjaTrader_Bertrand]

Stephan, that should not be needed when using FileIO classes, but those are not as high performing as the Stream objects, so you might need to look into changing over to using those.

[Stephan123]

Are dataseries objects automatically cleaned up
at termination?
Those are used within the strategy I try to walk forward optimize.
Is memory used by indicators automatically cleaned up
at termination?

If yes in all cases, It seems I cant do anything to make the optimization work.

[NinjaTrader_Bertrand]

Stephan123, yes those cleanups are handling by the .NET garbage collector as NT releases those resources, unfortunately there's nothing to 'force' here. Are you using the DataSeries on 256 lookback or Infinite?

Best approach would be to simplify your code until you achieve the desired performance and then rework / optimize the other areas that contribute to your slow execution time seen, switching to the higher performing Stream objects combined with a lower DataSeries lookback range could help here in lowering the memory footprint seen.

Good luck,

[Stephan123]

Stephan123, yes those cleanups are handling by the .NET garbage collector as NT releases those resources, unfortunately there's nothing to 'force' here. Are you using the DataSeries on 256 lookback or Infinite?

Best approach would be to simplify your code until you achieve the desired performance and then rework / optimize the other areas that contribute to your slow execution time seen, switching to the higher performing Stream objects combined with a lower DataSeries lookback range could help here in lowering the memory footprint seen.

Good luck,

All datasseries are run without explicit look back period. That should mean default with 256.
The problem is caused by the text file being read, put into the memory within OnStartUp() as a 2D string array and being avaluated within OnBarUpdate().
It consists of 7000 lines and 5 columns.
I would expect the array not to be big enough for RAM freeze.
But obviously it somehow does. Any suggestions on how to use those txt-data
without memory problems?

[NinjaTrader_RyanM1]

You may want to look into streamreader here for better efficiency reading text files.

http://www.ninjatrader.com/support/forum/showthread.php?t=3476

[Stephan123]

What is to be prefered regarding the out of memory problem and efficiency:

a) reading the whole file into an array within on Startup() using streamreader and using this array within OnBarUpdate()
b) scanning the whole file for a specific line with streamreader and load values of that line each time OnBarUpdate is activated

The text-file has 304 KB. I'm surprised that it causes that kind of problems.
Estimated Calculation time for the Optim procedcure is 6 minutes.
After 4min the system freezes. Why is the memory filling up during optimization progress? Shouldn't there be a constand memory usage during the optim process?

[NinjaTrader_JoshP]

Stephan123,

Memory is managed through .NET's garbage collector which we do not control.

How your code uses computer resources we could not comment on without actually seeing the code, but for your question about which is preferred, A. would probably be more efficient at least CPU wise.

Please also be sure you are actually disposing your objects when you are done with them.

[Stephan123]

Stephan123,

Please also be sure you are actually disposing your objects when you are done with them.

Disposing objects manually has to be done only for streamreader type in my case, correct?
Indicators, Bars Objects, dataseries are disposed by NT automatically, correct?

I will try a streamreader version for the file scan. If it doesn't help I will post the code
snippets.