Backtest producing out of memory exception

[vanderhe68]

Hello,

I've been using Ninja trader now for some time and understand how having more charts, strategies, etc open, requires more system memory to handle. However, recently I have been getting the following error when backtesting a strategy using 1 minute and the corresponding 1 second chart for the ES-0310 from 1/1/2010-3/14/2010.

4/5/2010 10:26DefaultFailed to call 'Add' method for period type 'Tick': Exception of type 'System.OutOfMemoryException' was thrown.4/5/2010 6:09DefaultFailed to call 'Add' method for period type 'Tick': Exception of type 'System.OutOfMemoryException' was thrown.

I am running 6.5.1000.15 and my system has 4gig of RAM and it appears that I don't have any other charts or strategies running. When I look at the results, it seems like it loads data fine for all dates except 2/6/2010 through 3/1/2010. There is no data for those dates in the backtest. I have even re-initialized my db as it was around 700MB. However, when I look at the trace file, it appears to be loading 1 minute charts for a bunch of instruments I had open in the past...and there is some line in there about MarketDatabuf that points to files in a \db\data subdirectory.

Can you please help point me in the right direction?

Thanks in advance for your help, Randy

[NinjaTrader_JoshP]

Randy,

When you add a tick series you can quickly use up a lot of RAM, especially on a 3 month period of something as heavily traded as the ES. Each bar requires memory and when you add millions of bars you can see how that quickly adds up. What I suggest is for you to significantly decrease your backtest length or increase the tick series size you are using to reduce the number of bars generated. NT7 will come with lots of memory improvements to the bars to allow you to handle more bars at once, but exceptionally large backtest requests will still result in out of memory messages.

[vanderhe68]

Thanks Josh,

I sort of figured that would be the case. So I know ticks don't exactly equate to time....but are you saying that using something like 233T or 377T would be less intensive than say the 1 second chart?

It does seem however like other charts are loading and causing me to use more memory as well. Can you please address why it appears in the trace files (below) that all these other instruments are being loaded when I don't have charts up for them? I have used charts on them in the past...but shouldn't have anything loaded for them now.

Thanks, Randy

Data.MarketDataBuf.FileName. Directory created: 'C:\Users\Randy\Documents\NinjaTrader 6.5\\db\data\20100405'
2010-04-05 06:09:43:520 ERROR: Failed to call 'Add' method for period type 'Tick': Exception of type 'System.OutOfMemoryException' was thrown.
2010-04-05 06:09:52:704 (Amp Live) Data.Bars.GetBarsNow: instrument='ES 03-10' from='2010-03-11' to='2010-03-31' sessionBegin='00:00:00' sessionEnd='00:00:00' excludeWeekend=False period=1 Tick splitAdjusted=False dividendAdjusted=False bars=0
2010-04-05 06:10:10:066 (Amp Live) Hosted.Unmanaged.Callbacks.Alert: connection=1 alertType=6 code=7 msg='The subscription to this instrument has been aborted. Dependent network services and/or data were not available.'
2010-04-05 06:10:10:081 (Amp Live) Data.Bars.GetBarsNow: instrument='6E 06-10' from='2010-04-05' to='2010-04-05' sessionBegin='00:00:00' sessionEnd='00:00:00' excludeWeekend=False period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2010-04-05 06:10:10:756 (Amp Live) Data.Bars.GetBarsNow: instrument='TF 06-10' from='2010-04-05' to='2010-04-05' sessionBegin='00:00:00' sessionEnd='00:00:00' excludeWeekend=False period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2010-04-05 06:10:11:563 (Amp Live) Data.Bars.GetBarsNow: instrument='SI 05-10' from='2010-04-05' to='2010-04-05' sessionBegin='00:00:00' sessionEnd='00:00:00' excludeWeekend=False period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2010-04-05 06:10:12:432 (Amp Live) Data.Bars.GetBarsNow: instrument='6J 06-10' from='2010-04-05' to='2010-04-05' sessionBegin='00:00:00' sessionEnd='00:00:00' excludeWeekend=False period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2010-04-05 06:10:13:036 (Amp Live) Data.Bars.GetBarsNow: instrument='DX 06-10' from='2010-04-05' to='2010-04-05' sessionBegin='00:00:00' sessionEnd='00:00:00' excludeWeekend=False period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2010-04-05 06:10:13:791 (Amp Live) Data.Bars.GetBarsNow: instrument='6B 06-10' from='2010-04-05' to='2010-04-05' sessionBegin='00:00:00' sessionEnd='00:00:00' excludeWeekend=False period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2010-04-05 06:10:14:491 (Amp Live) Data.Bars.GetBarsNow: instrument='CL 05-10' from='2010-04-05' to='2010-04-05' sessionBegin='00:00:00' sessionEnd='00:00:00' excludeWeekend=False period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2010-04-05 06:10:15:119 (Amp Live) Data.Bars.GetBarsNow: instrument='NG 05-10' from='2010-04-05' to='2010-04-05' sessionBegin='00:00:00' sessionEnd='00:00:00' excludeWeekend=False period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2010-04-05 10:25:25:989 (Amp Live) Data.Bars.GetBarsNow: instrument='ES 03-10' from='2010-01-29' to='2010-01-30' sessionBegin='00:00:00' sessionEnd='00:00:00' excludeWeekend=False period=1 Tick splitAdjusted=False dividendAdjusted=False bars=0
2010-04-05 10:26:45:913 ERROR: Failed to call 'Add' method for period type 'Tick': Exception of type 'System.OutOfMemoryException' was thrown.

[NinjaTrader_JoshP]

233T charts would be less taxing than a 1 second chart if it has less bars. In most cases this would be the case unless you could generate more than 233 ticks within a second to have 2 bars instead of 1.

You may have charts or Market Analyzers open in workspaces in the background. Please be sure you have all background workspaces closed.

[BigMike]

Similar Problem

NinjaTraders,
I have a series of indicators thatb work fine on a chart but when I include them in a Strategy they hangup the computer or crash the ninja software. The Strategy Analyzer was the latest victim.I looked at the trace file and saw multiple system memory errors. Here is a quick look look at one of my indicators.

#region Variables
privateint period = 10;
privatedouble newvalue = 0;
privatedouble dperiod = 0;
#endregion

protectedoverridevoid Initialize()
{
Add(new Plot(Color.FromKnownColor(KnownColor.Teal), PlotStyle.Bar, "Adapfish"));
Add(new Line(Color.FromKnownColor(KnownColor.Maroon),0,"ZeroLine"));
CalculateOnBarClose = true;
Overlay = false;
PriceTypeSupported = false;
}

protectedoverridevoid OnBarUpdate()
{
if (CurrentBar < 20)
return;
{
dperiod= HomodyneDiscriminator(Close)[0];
dperiod= Math.Round(dperiod);
period = (int)dperiod;
newvalue = FisherTransform(Close,period)[0];
Adapfish.Set(newvalue);
Can You see anything here that would be a problem for a strategy to call or for the Analyzer?
Again it works just fine directly on the chart but Hangs up a Strategy.

Thanks For your help.
BigMike

[BigMike]

Similar Continued

My fix is to retype the code for the indicatior directly into the strategy, but that has become a problem in my latest strategy, I can't code around the fix, Coding isn't my strength. I obviously need to fix the problem at the indicator level

[NinjaTrader_JoshP]

Hi Mike,

How long of a backtest are you running? I would suggest you try this on NT7 beta if you have not yet since the memory enhancements would definitely be a plus in all situations.

Also in NT7, there is also a new concept of limiting DataSeries objects to only the last 256 bars. This helps in memory consumption.

As far as the code you are using though, it looks like you are making many many instances of the FisherTransform() indicator with the different possible period values. This can be taxing on the system if you end up with hundreds of different period values and that may be the issue. To address this you could try getting the FisherTransform logic into your strategy as a custom method and call that method instead of calling an indicator which loads up historically for every single bar. This would only work if the FisherTransform logic does not require every single bar and can actually be calculated internally though. I have not evaluated if this indicator's logic will be OK with such an operation. Hopefully that helps you get some ideas though.

[BigMike]

As Far as I Can Tell

Josh,
Thanks for looking into this.
As far as I can tell it recalculates the period at the close of each bar. I run my charts at 300 to 400 tick and have 5 to 7 days worth which is not that many on the YM and NQ that I trade. Where the problem begins is when I make a call to the indicator from the Strategy.

example if iI use the following in a strategy I get the problem mentioned,

if (AdaptiveFish(Close,period)[0]>AdaptiveFish(Close,period)[1])
do something

This is where I think it goes awry, and why it works if I retype the entire indicator into the Strategy. The Period is critical. The indicator wants to calculate the period and I am passing the period to it. What do you think ?,is there a way to use the above command without causing a conflict?
I haven't tried coding the Homodyne discriminator portion in the strategy and take it out of the indicator and only pass the period.
Anyway any input from you will be appreciated.
Mike

[NinjaTrader_JoshP]

Mike,

Passing in the period would definitely be a point to investigate if you are passing in many different periods throughout the life of your strategy. Each time you use a new period you are instantiating a new instance of the indicator and all of that needs to be held in RAM till the end of the strategy. Even if you only used the value once, the complete series is held in RAM. If you call many of these you can quickly see how it adds up as "dead space" in RAM. In such a scenario the only way around is to move the calculations to be held natively in the strategy.

[BigMike]

Makes Sense

Josh,
Thanks for the Help.
I Took the Discriminator out of the Indicator and Put it in the strategy and passed only the new period it calculated. that seems to have clerared up the memory problem as the Analzer is working well with no errors generated.

Mike