Optimizing Curve Fitting

[webart]

I have some strategies that backtest well over the last 6-9 months using 1 minute candles.

I need to re optimize some variable in the strategy to suit different instruments, ie YM and ES (very similiar markets) won't work with the same settings. Does this indicate that I am badly curve fitting ?

Also what period of data is best to back test on. The market is always changing, so I can't see the point of backtesting too far back, on the other hand don't want to badly curve fit.

[ctrlbrk]

Arguments can be made every which way on this. Backtest further back because the market is always changing. Or not, because the market is always changing. etc.

I believe the overall consensus is if your strategy is profitable over longer time frames (3 years+) then you've got a good strategy. However, that has been quite elusive for me at least, so I personally work on about 1 year at a time and just keep tweaking things.

If I could make a strategy work over 3 years though, I would consider it far superior.

As for a strategy working on one market and not another (YM/ES), I believe the behavior of each market is unique. The general principal should be the same, but you might need to fine tune the variables. I think this is not curve fitting, but instead accepting the fact each market is different. Again, there are different opinions out there that say a sound strategy must work on all markets.

If I could make a strategy work on all markets, I would consider it far superior.

Hope this helps!

[NinjaTrader_Ray]

Thanks for the comments ctrlbrk.

[Elliott Wave]

Perhaps when you backtest, reserve some recent data and walk forward.

The market next year is likely to be totally different than the market this year in many ways. Just imagine someone who designed, backtested and tweaked their system to perfection during 2007 and went live in 2008. The more the system was curve-fitted the more likely it is to fail.

I spent months at the start curve-fitting systems like crazy before I really knew what curve-fitting even was.

I've come to believe now that the the simpler the system the better. I recently read the book 'Trading Systems: Secrets of the Masters' and there were some common themes and curve-fitting definitely wasn't one of them.

In my opinion, proper position sizing and risk control seems to be much more important than any specific rules based on technical analysis. Another very important factor is that the underlying math of a system be statistically sound and adapt to the market. Cynthia Kase's book is very good in this regard and gives plenty of ideas on how to make systems more adaptible without added complexity.

A good way to evaluate optimization is with a 3D optimization chart and looking for stability. Hopefully this is something we'll see in a future version of NT.

The attached chart shows the optimization results of a system with only two optimization parameters. The goal is not to choose the most profitable parameters, but the most robust. Also if your system is all over the map it likely means its a weak system and over-optimized.

[ctrlbrk]

Elliott,

Very good info.

I absolutely agree, the simpler the system the better. The more variables in there to the more it is curve fitted to the data.

But I do believe in some weeding out sideways markets or markets that are not as volatile if that is what your script needs. Using ADX, ATR, or StdDev to do this works well for me.

Also a lot of time people spend all their time on the entry, and pay no attention to the exit. It is possible to curve fit exits as well, but I think in general you should be able to customize your exit to try and get the most MFE possible (max favorable excursion) out of each trade. That is the goal at least.

Last, we should not forget about time frames. A strategy may work well on 1 minute data and fail miserably on 2 minute data. This probably suggests a problem, but it is all dependent on your strategy. Scalping strategies need as small of time frame as possible. Whereas, if your average trade is two hours long, then an extra minute or two on the entry shouldn't make much difference.

Personally, I like to find the entry signal, then program my script to wait for a retracement or wait for MACD divergence to near 0 indicating a pullback, etc, before entry. Sure you miss a few calls but in all I get better entries using this method.

[webart]

Great info, thx Guys. I agree if you could create a system that backrests well for 3 years and works on all markets that would be great. I believe this was done by a Neural system in the book The Encyclopedia of Trading Strategies, however such a system is probably beyond my ability to even attempt, atleast at this point.

On the other hand I have read about systems that have been performing well for a few years and have failed recently, and resulted in massive draw downs.

Also agree the need to keep the number of factors to a minimum, but I have found in my case it is better to add a couple of noise filters to keep out of the choppy chop.

When I first started back testing I thought I had hit the grail when I saw a system that produced 100% results lol. Now I get suspicious if I see above about 65% success rate.

The further I optimize back, the less returns the system makes. If I optimize back 6 months, I can get a system that makes as much or more than a system optimized for 9 months. That is curve fitting I am sure, but all optimization is curve fitting. I guess the more curves you can fit (longer history) the more curve you are likely to fit in the future.

On exits, I'm still trying to work this out. The market is so volatile right now that small stops can really knock you out of allot of good trades, and there are allot of big moves most days, so a relatively large stop with a larger fixed target seems to back test best, but I think I will set a B/E just about 3/4 of the way to the target for those trades that don't go far enough. If the strategies work well live, then maybe I could run a couple of contracts with different money management on each.

Attached are a couple of reports on the YM. one for Longs, one for Shorts.

[ctrlbrk]

The equity curve looks good.

One thing I try and consider is the number of trades. If you are trading hundreds of times, with high probability, I think you should not consider your results curve fitted. But, if you are trading a dozen times with huge returns on each, then sure -- curve fitted it is.

That is something I struggle with on my strategies. Eliminating too much noise and leaving myself with 1/3 or 1/2 of the trades I started with, definitely curve fitting.

I also like to always sort the results and eliminate the top 5%. In other words if I have 100 trades then lets take the 5 (5%) most profitable and just chop them off, so I am left with more realistic numbers, imo. This way if you are optimizing a a certain figure suddenly gives you an extra 2k return, you can eliminate it if it all came from a single trade.

[webart]

Yeah its curve fitting, but I don't think it is bad curve fitting. Surely its better to have a system that produces less false triggers from noise.

The other problem with eliminating noise is that the more noise you eliminate the later you get into a good trade ... there are trade offs for everything.

There are over 100 trades in that graph. The profit is the same for each trade as the target and stop are fixed. The tricky bit is trying to work out the best money managment without being able to backtest it.

[ctrlbrk]

You may wish to implement a dynamic stop and target. I believe using the ATR is a good method to determine this, or there is an indicator in the file share forum ATRtrailing for trailing stops.

I think it is always best to be dynamic with your exits. It is not very easy to implement, but the reward can be very good.


EnterLongLimit(DefaultQuantity, GetCurrentAsk(), "long 1 es");

SetStopLoss("long 1 es", CalculationMode.Price, Close[0] - (ATR(BarsArray[2], 4)[0] * Long1smaatr1), false);

SetProfitTarget("long 1 es", CalculationMode.Price, Close[0] + (ATR(BarsArray[2], 4)[0]) * Long1smaatr2);


Where BarsArray[2] is a longer time frame, like 60 minutes. My average trade lasts approximately 90 minutes on this particular strategy. The variables Long1smatr(x) are to allow me to adjust the multiplier for profit/loss, like 1:1 or 1.6:0.8, etc (1.6:0.8 is what I am currently using for target/stop).

[webart]

Thanks, appreciate the code. Currently I am planning to just call the Ninja ATM strategy and use the break even / trailing stops from there.