[Optimizer] Approx. how many iterations can NT handle before hitting out of memory?

[Nyboe]

Does anyone at NJ trader have any benchmarks -- or does anyone with lots of experience have a sense about how many iterations/permutations the optimizer can handle before running into the dreaded "Out of memory exception"?

My dataset is the ES from 1/1/2009 - 9/25/2009 ... so about 9 months of data. My strategy had 3 variables and I varied them like this:
VAR1: 2 => 4
VAR2: 4 => 16
VAR3: 4 => 32

This turns out to be 1,131 iterations, and NT's optimizer handled this no problem--it took a few hours to complete, but it work and I was thrilled.


I then thought of a couple additional factors, which led me to 5 variables, and I expanded by VAR2 and VAR3 variables slightly. So I ended up with this:
VAR1: 2 => 4
VAR2: 2 => 32
VAR3: 2 => 32
VAR4: 2 => 32
VAR5: -32 => 32

Those variables above equate to 5,809,245 combinations!!! And NT ran into the dreaded "Out of Memory exception" [memory peaked around 1.2 GB].


So my question is this: does anyone have a good understanding about the optimizer and how many permutations/iterations it can handle before it's "too many"? I don't mind logically breaking up my optimizations...I just want to setup an optimization, let it run for 12+ hours, and have confidence that it completes successfully and that it didn't run out of memory or whatever.

Thanks for any thoughts about this.

-Aaron

[NetTecture]

Hello,

The limitation depends on your computers memory, so there is no way we can perdict it for you. I would clear out some of your memory and/or upgrade computers, or optimize it in sections.

That answer is one thing: NOT TRUE. It absolutely does NOT depend on the memory in the computer as:

* If it needs more than the phsyical memory, it will go to virtual memory. THis will get slow, but run.
* But most important, as NT is a 32 bit process, there is a VERY LOW high ceiliing of how much memory the process an use.

Basically... it wont use more than 2gb memory (3gb with some dll manipulations). This mostly translates in a 1gb - 1.6gb limit on .NET, depending how things are programmed.

Even on a 64 bit machine.

Given that 4gb is a pretty low phsyical limit these days (i.e. serious machines will have that or more)...

...this means that the answer most likely will NOT be dependant on the computer memory at all, but only on the limitations of the 32 bit address space

WIthout manipulation of the NT executable the programmatic limit is below what is a normal hardware configuration these days.

Sorry, Ben, this was NOT good support. It was WRONG support.

[NinjaTrader_Dierk]

I wasn't asking how big of a machine I need...I have 4 GB of RAM, running Vista 64-bit, etc.

I'm trying to find out approximately how many iterations it is reasonable to expect NT 6.5's optimizer to handle. Throwing more memory on my machine isn't going to help anything...I'm running into that 1.2GB-1.3GB max memory utilization before NT throws the dreaded 'Out of Memory' exception.

So given the fact that I have enough memory...any idea how many iterations NT 6.5 should be able to run the optimizer over given 9 months of ES data? [10 tick bars if that is important too]

I know for a fact that 1,131 iterations works fine ... but over 5 million iterations causes the out of memory exception. I'm just trying to nail down if the threshold is around 1 million; 3 million; 500,000; 150,000; etc.

And I know there is no EXACT answer, but I'm hoping the NT enigineers/testers can give me a rough estimate OR the general users: how many iterations have you run optimizer over in the past?

Thanks.

Unfortunately we are unable to provide the info you are looking for. For the most part this is dependent on your actual strategy (how many trades would it take, how many indicators would it use, what data series etc.).
I suggest trying by yourself to see where your actual setup would hit the wall.

Also: I doubt you would be able to load a 9 month 10 tick series at all. However, you should not experience that issue on a minute based series.

[Nyboe]

Ok...so basically you're telling me to just try it myself. which is fine, I was going to do that anyway, and YES I'll post my results on here, i.e. I'll let the forum know how many iterations the optimizer went through and if it completed for me.

I was just hoping somehow could give me a rough estimate, b/c I can set my 5 variables up to have 1,000,000 iterations or 200,000.


Anyway, your last comment was interesting. I was able to use the Merge function...so I have a combined "ES 12-09" for Jan 2009 - present. It is tick data I believe. I run my backtest/optimizer with 10 ticks/bar. Backtests work great for me over the whole year--it takes a handful of minutes but it works. And like I said, I was able to run an optimizer on the whole year that had 1,131 iterations (this was with 3 variables).

So I think my tick data, optimizing over the current year is working well. I'm curious why you think it wouldn't work?

[NinjaTrader_Dierk]

Great it's working for you. Usually people experience trouble on managing tick based series for more than 3-4 months.

[Nyboe]

I hope I did my merge right!

That's actually a question I was wondering about...is there anyway I can interrogate an instrument and confirm what date range of data is inside it? better yet, can I get a count of ticks per day...just to make sure all the data was merged correctly?

If that data were in a SQL table--one row per tick--I basically want to run this query [assuming mySQL syntax]:
SELECT MONTH(dateColumn), DAY(dateColumn), COUNT(*)
FROM TblName
GROUP BY 1, 2

I then want to browse the result set and make sure I'm not missing any ticks on days...or better put, make sure there isn't a day with a noticeable lower # of ticks.

I just want to spot check my merged instrument ... is there any way I can do this?

Thanks.

[NinjaTrader_Dierk]

As of NT6.5 you could open a chart to see if the data "looks right". NT7 will come with a data manager which would allow you to view the data in a table.

[Nyboe]

ah ok ... yet another reason to be salivating for NT7 - can't wait!!!

So what does it take to become a "development partner"?

[NinjaTrader_JoshP]

If you send a note to sales[at] ninjatrader [dot] com they will be able to go through the process with you for becoming a partner.