Out of sample optimization

[tradetree]

Is there any way to optimize on one set of data, but then re-optimize on a second set of data? One of the big problems with optimizing is curve-fitting. You set your parameters based on the curve of the in-sample data. Then you run the system on new future data. But in out-of-sample testing, you set aside a set of samples to optimize against, and a different set to backtest against. This is called "out of sample optimization".

NT can optimize on any date range you pick. That works fine. The problem is that I want to find the best set of parameters that work on the out-of-sample data. This means you automatically optimize against the first data set, then find the best run against the out-of-sample data. Is this possible?

The only way I've thought of doing this is to save the parameter sets found in the first round, then backtest against those sets for the out-of-sample period.

[NinjaTrader_Matthew]

Hello,

Can you clarify what you mean by data set? Are you wanting to optimize on another date range? Or are you looking to change the interval? I.e., optimize on a 1 minute data series, and then again on a 5 minute?

[tradetree]

The timeframe is set at 2 min bars. I looked at your Walk Forward optimization but even the video doesn't tell how the time periods work. In my case I want to use from 1/1/1008 to 10/1/1012. One part of that should be the optimization data, another part the test data. If I set the Walk Forward optimiation period to 400 days, then the test period to 600 days, it will optimize from 1/1/2008 for 400 days. Then test from 2/1/2009 to 10/1/2010 which is about 600 days. Is that correct?

That means I can't optimize from 2/1/2009 to 10/1/2010 and then test from 1/1/2008 for the 400 days?

[NinjaTrader_Matthew]

Thanks for the clarification.

In this case, you would need to save your input parameters and re-run the test for the second date range and compare the results.