OutOfMemoryException in strategies using Add(PeriodType.Minute, 1) in Initialize()

[ChiTrader2000]

Hello,

Whenever I backtest intraday strategies that have an extra 1 minute bar time series added in the "Initialize()" method with the following line of code: "Add(PeriodType.Minute, 1);" I get the following fatal exception before I can finish testing all of my contracts (over 1000 contracts in about 10 different Instrument Lists):

"Error on running backtest: Exception of type 'System.OutOfMemoryException' was thrown."

I was able to even reproduce this error with a copy of the "SampleMACrossOver" strategy that comes pre-packaged with Ninja Trader by adding the line ""Add(PeriodType.Minute, 1);" in its "Initialize()" method.

In my tests of my copy of "SampleMACrossOver", the problem only happens when the line of code is there, and it doesn't happen on the same Instrument List or contract each time.

I am able to use parameters that produce 100,000's of trades when the line of code is not there and everything works great, but I can't use parameters that even products 10,000's of trades when the line of code is there without Ninja Trader crashing.

Ninja Trader doesn't crash until 90% or so of my Instrument Lists have been tested. When I start from the bottom of my Instrument Lists (on the lefthand side of the Strategy Analyzer) and work up, right clicking and backtesting each entire Instrument List at once, NT crashes when I am almost at the top, and when I backtest starting from from the top down, NT crashes when I am almost to the bottom.

My strategies need to have a hidden time series of 1 minute bars to be useful, and I need to test 1000's of contracts at once.

How can I do this in Ninja Trader without it crashing after testing 90% of my contracts?

I have access to the strategy code, and am willing to write ANY memory management code that I need to to get this working.

Perhaps I can use some form of "Remove()" to remove the 1 minute bar time series for contracts that have ALREADY been tested (In initialize of 1 contract, perhaps I could wipe out 1 minute bars that were stored in memory from the backtest with some sort of "Remove()" command)?

Thank you very much in advance!
ChiTrader2000

[NinjaTrader_JoshP]

ChiTrader2000,

Memory is managed by .NET. If it says you ran out then you need to decrease your load on the system. Highly suggest you decrease your backtesting time frames.

[ChiTrader2000]

NinjaTrader_Josh or anybody who can help me,

This OutOfMemoryException problem is killing me. And I am only doing the backtests on 1 year of data. I need to be able to do it on more like 10 years of data.

I will do ANYTHING in code, or with money (buy a 64-bit Windows machine to perhaps handle memory problems?) to solve this problem.

I need one of two things: 1) Someone on this forum to tell me how I can accomplish the task I described earlier in this thread without Ninja Trader crashing with an OutOfMemoryException, or 2) I will need to switch to RightEdge.

I seriously do NOT want to switch to RightEdge, as I like EVERYTHING ELSE ABOUT NINJATRADER.

Here is how RightEdge handles this problem:

==================================
A quote from:
http://www.rightedgesystems.com/forums/Topic6858-7-1.aspx

Large data set support
Most systems only need a certain amount of bar history for their calculations, and there is no need to keep bar data available past that amount. When working with large amounts of high frequency data, it may not be possible to store all of the data in memory at once. To allow systems to be run with large amounts of data, RightEdge now includes a MaxLookBack property in the RList class. If its value is greater than zero, RightEdge will discard any items past the maximum lookback. If you try to access any of the discarded items, an exception will be thrown. You can set the value in your symbol script startup method, like this:
Bars.MaxLookBack =1000;

==================================

I can tell from the nature of the scientific experiment described earlier in this thread that the "Add(PeriodType.Minute, 1);" on the copy of SampleMACrossOver keeps all the 1 minute bars in memory for all products that have ALREADY been tested in a multi-product backtest, even though the NinjaTrader backtesting engine has already moved on and is backtesting the next product.

Couldn't Ninja Trader just either purge these 1 minute bars from memory at the end of each product's backtest or at the start of the next product's backtest?

Does Ninja Trader have a similar ability to purge the 1 minute bars out of memory like RightEdge can with "BarsMaxLookBack=1000"?

If not, is there ANYTHING I can do to accomplish my objective with the Ninja Trader software?

Thank you very much in advance,

ChiTrader2000

[NinjaTrader_JoshP]

ChiTrader2000,

In NT7 we will be introducing a max bar lookback concept as well. This should help you considerably in your scenario.

[ctrlbrk]

ChiTrader2000,

No, you are not alone. I've posted many times about this.

In short, you can't do anything except test small time frames and wait for NT7.

A 64-bit machine does nothing. I have x64 and 12GB of memory, makes no difference.

Mike

[ChiTrader2000]

NinjaTrader_Josh,

I am EXTREMELY happy to read your post that "In NT7 we will be introducing a max bar lookback concept as well. "!!!

When will I be able to download NT7 with the "max bar lookback concept"?

I would be perfectly happy to use a Beta version and help you test this feature as soon as possible!

A little background... My superiors ($1 Billion+ managed futures fund) loved my NT demo and gave me the go-ahead to polish off a final version of my Ninja Trader modification for all the traders, but they have not yet been informed of the OutOfMemoryException problem yet. I don't want them to think that they have wasted 3 months of my salary to pay me to work on an application that we will have to throw out due to our large number of products and long backtesting time range (about 10 years).

Thank you very much!
ChiTrader2000

[NinjaTrader_JoshP]

Beta is slotted for end of Q2.

[ChiTrader2000]

NinjaTrader_Josh,

Thank you for the quick reply. I will download the Beta the first day it becomes available.

Until then, is there anything I can do besides decreasing my time range for testing?

Note: I have already tried all the tips located at: http://www.ninjatrader-support.com/H...rformanceTips1

Also, if there is any way I can get a "pre-beta" quicker, even if it is untested, please let me know.

Thanks,
ChiTrader2000

[NinjaTrader_JoshP]

All I can suggest is to decrease the backtest sample size. Unfortunately I do not have information on any "pre-beta" release.

[ChiTrader2000]

NinjaTrader_Josh,

Thanks for the quick reply. I will wait for the Beta, or, if there ends up being a Pre-Beta, I would love to get my hands on it as soon as it is available.

Thanks again,
ChiTrader2000

[NinjaTrader_JoshP]

We will let you know. Thanks for the patience.

[ChiTrader2000]

NinjaTrader_Josh,

Thank you very much. I am very much looking forward to it.

ChiTrader2000

[lukaszkn]

Does that happening when optimizing strategy or just backtesting?

Luke

[NinjaTrader_JoshP]

Luke,

Backtests generally will not cause you to run out of memory. Usually if you run a very extensive optimization will you have the potential of running out. NT7 will provide improvements for this across the board.

[NinjaTrader_JoshP]

Also, for clarification ChiTrader2000, the max bar lookback that will be available in NT7 will be used to improve the performance on indicators and strategies that have internal DataSeries objects. This is not to say that it starts dropping off bar data out of memory. NT7 will provide considerable reduction in the memory footprint of bar data though so you will still see significant improvements with larger bar series sets.