Curve fitting - Concrete example

[cunparis]

This isn't specific to Ninjatrader so I'm hoping someone here can share their opinion and/or their experience.

I just found a good example of what happens with curve fitting.

Before my vacation I ran optimizer for my strategy up until 12/4/2008 and I got values for the indicator parameters. I wrote it all down in excel (I always do this so I can get the values later, I also copy the results like %, R/R, P/L, etc.)

So today I reran the optimizer through through jan 2 and I got different values. Very different. I couldn't see how this was possible, so I reran optimizer to 12/4 and got the old values again. This confirmed that I hadn't copied it wrong when I put it in excel.

Then I ran the backtest for 12/4 - 1/2/2009 using the values obtained 12/4 and guess what? There was an $800 loss during this time! So when I ran the optimizer through jan 2, it choose other parameters to eliminate this bad trade!!

I'm kind of stuck here because I don't know what to do. If I use the new values, it's possible to have a bad trade come up that wouldn't have been choosen had I kept the original values. If I use the original, the reverse is possible.

My thinking is to use the new values. I think we can't know either way which set are the best and only forward testing will tell. So my idea is to always use the values obtained from the most comprehensive data (longest timespan).

But if one runs the optimizer every month then won't the system be more and more curve fitted? Would that be a bad thing?

Just for discussion, imagine a system that we can backtest for the past 10 years. It would seem that after 10 years we have the "optimal" values (let's assume that the market conditions don't change!). So it would seem that optimizing every month during the 10 years would move the system more and more towards the optimal values.

However the market conditions change and what worked before may stop working.

This is all very confusing to me. I'm afraid that my strategy won't work as well in the future as it did in the past. That is to be expected I think. If it will work similar then i'll be happy. If it doesn't work at all I'll be very disappointed.

Final question: How many trades should one have in the backtest in order to have confidence that it isn't just curve fitted? Is 100 enough?

[ctrlbrk]

Personally, I feel that less than 100-150 trades, at minimum, need to be evaluated over as long of a period as possible, before I feel the results are not curve fitted.

It is a very difficult thing to accomplish. I usually start by backtesting 1 year, and then once I get a core idea to work, I increase to 2 and 3 years. Usually about this time is when things fall apart.

In some cases what I can do is simply have the system wind up taking few to no trades during those longer periods, instead of making bad trades. A filter for instance, for less volatile times.

Another thing recently I am trying is only backtesting thru September of 2008. I feel after Sep 08 things are incredibly psycho in the market and likely they have settled down some now, and will continue settling a small bit each passing week and month. So, my goal is to get positive backtest results up to Sep 08, then forward test them to make sure they don't completely blow up, and go from there.

One last piece, for me at least, is to focus on making a LONG strategy work during this past year. Getting a short strategy to work is easy when the market is down 40% for the year.

I have seen strategies backtest for 5, 7 and 10 years with high probability of success. They are possible to accomplish.

[cunparis]

Hi Ctrlbrk, thanks for sharing your ideas. I'd like to give a few responses.

It is a very difficult thing to accomplish. I usually start by backtesting 1 year, and then once I get a core idea to work, I increase to 2 and 3 years. Usually about this time is when things fall apart.

What I've found is certain systems work during a bull market and certain systems work during a bear, but it's rare to find a system that works in both. I first found this out when testing strategies others posted in other forums. I'd test them over the past couple years and find out they didn't work. And I couldn't understand how someone said it was working when it clearly didn't. Then I found the idea of testing differently for bull & bear.

The problem is that our current bear market started around beginning 2008, so it's hard to get 100 trades.

In some cases what I can do is simply have the system wind up taking few to no trades during those longer periods, instead of making bad trades. A filter for instance, for less volatile times.

Do you have ideas for a filter for less volatile times? It's a good idea that I haven't explored yet. I just learned about the VIX during the crash in october, maybe one could use that?

Another thing recently I am trying is only backtesting thru September of 2008. I feel after Sep 08 things are incredibly psycho in the market and likely they have settled down some now, and will continue settling a small bit each passing week and month. So, my goal is to get positive backtest results up to Sep 08, then forward test them to make sure they don't completely blow up, and go from there.

This is what I did at first but I showed losses going forward. So what I thought was it's better to include oct-dec in the backtest results. What if it January we top off and then head back down? If we break the current lows there would be a ton of panic.

One last piece, for me at least, is to focus on making a LONG strategy work during this past year. Getting a short strategy to work is easy when the market is down 40% for the year.

Yeah I thought I found a short strategy that was 98% last year. I couldn't believe it. Then I found out it was just going short and waiting through any bear rally until it went back down. Kind of funny.

I have seen strategies backtest for 5, 7 and 10 years with high probability of success. They are possible to accomplish.

This is where I'm limited by IQFeed's data. Their eminis go back to 2005 which essentially gives me 1 bull and 1 bear market. I am seriously considering subscribing to another data provider that has long term data. I would like futures back to before 2000 so that I can have 2 bull and 2 bear market phases. Who are you using for data?

Great comments, thanks again.

[ctrlbrk]

My strategies are generally written to go after about 2 points on the ES, so they should work in any market, more or less, and are generally in the market for a matter of minutes.

If your strategies go for bigger swings or overnight carries then it would clearly benefit from trading with the trend. This would also make it a lot harder to get those 100 trades I personally target to know I am not curve fitting.

It's not that I think swings are bad trades, I've just not yet written a strategy I found to work well enough in a multi-day or overnight swing on the ES.

I have tried to come up with some that trade individual stocks within the S&P 500, but backtesting is so incredibly time consuming that I just gave up. I don't think Ninja can really do this properly. I have another program called Blocks (StockFinder) that is incredible for this purpose, but I've yet to learn their language to write a strategy so I can backtest multi-day swings there.

Regarding volatility, I have one strategy that looks at $TICK to determine volatility. I think VIX might work but it is too long term for me.

My data provider is IQfeed. Generally I only backtest around 3 years. I am not sure which data provider goes back 10 years as I haven't yet written a strategy that even works across 3 years. I have witnessed strategies that do, as much as 10 years, but they were not my own.

All of my strategies are still based on minute data. I use multiple time frames for entry/exit. My goal is to use tick level data, so I can use Range instead of Minute, since I prefer shorter timeframe trades. But, I cannot find any backtesting software that was written efficiently enough to make use of tick data over a long time frame.

[styr_trader]

good topic to discuss. In a lot of places it is recomended to use most recent optimization result, unless there is hope of building totally "universal" system that works in any market conditions (which is unrealistic).

I think human intervention is required to adjust certain parameters of the system every day based on the prediction of choppines/volatility of the market, say period 2 min or 5 min, stops 5 or 20 points, smaller stops for shorts and bigger for longs etc.