Backtesting best results not consistent when changing time frame

[Gwaihir]

I ran an optimization run and got the list of 10 top parameter combinations.

Then I shortened the time frame and got different optimal parameter combinations. Surprisingly most of them were loss making - and the top combination from the first run was not in the list. So I manually entered this combination and surprise it did make money, not the best one but far better than most of the results presented by the SA. These runs stop after some seconds though the time estimate in above 3 Minutes.

Playing around with this it and looking at the results it seems the optimizer stops the run after testing the parameter combinations the minimal value for the second parameter and ignores the rest of the parameter sphere...

Switching back to the longer time frame gives a long parameter run (with decent results - presumably). Suggestions?

[NinjaTrader_Bertrand]

Gwaihir, I'm not sure I follow you're doing / expecting - of course different timeframes will produce different optimal parameter sets as the underlying data is not identical.

[Gwaihir]

Of course results should be different but if I get (example)

for a time horizon 1995 - 2005 with
Parameter combination a result of
3 / 8 / 15 +1000 (and constant positive returns during that period)

then I should not get a optimizer result list for the time horizon 2000-2005 of
1 / 6 / 12 +400
2 / 6 / 11 +100
4 / 6 / 13 -400

Manually entering 3/8/15 yields +200 for that period. Why does that combination does not show up in the second result list (at second place)? (6 btw is min value for parameter 2)

[NinjaTrader_Bertrand]

On which performance criterion where you optimizing to generate the results?

[Gwaihir]

Oh, you're correct. I was optimizing inadvertently for max profit factor and looking at max result.

Unfortunately the missing strategy has a higher profit factor than the best one found... (When optimizing for net profit the parameter combination still does not show up)

[NinjaTrader_Bertrand]

Were you keeping all other related parameters in the optimization run the same? Is the same happening with the SampleMACrossOver strategy?

[Gwaihir]

Yes. No change, only change in time frame.

No, that would be much of a conincidence, wouldn't it.

The obvious indication that something is wrong is that first the estimate says like 2 to 3 minutes and after 3 to 4 second the results are presented (prematurely). I have run quite a couple of optimization runs and so far have never experienced the estimate to be that out of sync with the actual result. Also all presented "best" results had the minimal value at parameter #2. I played around a bit and even asking for the top 100 does not lead to much as only results with the parameter #2 set to min are returned (regardless of where I set the parameter, and it has been always lower than max).

[NinjaTrader_Bertrand]

The times given are just the 'best' estimates it can arrive at, there's no way to 'predict' when a run is fininshed.

If you see this reproducibly with the SampleMACrossOver please post the steps to arrive at those results so I can check into here.

Thanks

[sandeshrai]

While running back test, I noticed that 5th Oct 2011 is completely blanked out for no reason. This appears to be defect in back testing.

Every other trading day around it shows all 24 hours, but on the 5th Oct 2011 my trading window (6:30 AM - 9:00 AM PST) is not displayed.

I was backtesting based on the 5 Range chart using the 24/7 template.

Please let me know what you need from me to help you verify.

[NinjaTrader_Bertrand]

Can you please check into the Historical Data Manager for this specific day in question? Is the tick data present there for this date?

Thanks,

[sandeshrai]

Can you provide me the instructions or link to the instructions on how to check the Historical Data?

[NinjaTrader_Matthew]

Hello sandeshrai,

You can check the historical data contained in your database by going to Tools--> Historical Data Manager--> Edit tab.

Click here for more information on using the Historical Data Manager

[sandeshrai]

Chart is data is missing

Can you please check into the Historical Data Manager for this specific day in question? Is the tick data present there for this date?

Thanks,

Yes, tick data is available. In fact, on the chart it displays the whole chart. But in the back testing result chart the information is missing and the chart does not display the information. For your information, I am using a 5 Range chart

[NinjaTrader_Matthew]

Do you see this on multiple symbols are just one in particular? If just one, what symbol is this?

If you see this for multiple symbols, does it always occur at the same time?

To confirm, you do not see this gap with one of the default system strategies such as the SampleMACrossOvers?

[sandeshrai]

Tried it on the on one of the default strategies 'SampleMACrossover'. When I run it on a 1 min chart and 5 min chart, it shows the chart until 11:30 AM and nothing after that. But when I run it on a 5 range chart it shows about 8 bars until 22:32 and nothing else. This does not match the chart for the day as it transpired by any means. When I look at the chart on the date now however, it shows the partial data there too.