Backtesting and live (paper) trades are different

[lookOutBelow]

I'm having a weird issue and I'm not quite sure what to make of it.

I'm seeing differences between live trades and backtests. The discrepancies are much more than slippage. Both are using 150 tick charts and the same session, same strategy, and same instrument. However, the entries/exits times and prices are quite a bit different.

This happened on all 7 of my strategies that ran throughout the day today on 7 different instruments. Only 1 of the strats was in the same ballpark with trade times and entry and exit prices.

I've attached an example image of what the backtest said and what the account performance said from the live trading after the day was over.

Thanks

[dave1992]

What type of orders do you use? Have you compared against a market replay, which should be closer to live than backtest. Live and replay have bid ask volume, which can make a difference - check out the Ninja page on the difference between live and test.

Don't forget that if your live fails on one trade because there's no volume at that point, then your setup is no longer valid then you might take a trade 10 mins later that your test didn't because you weren't flat in test. A whole set of trades can differ because of that.

[NinjaTrader_Jason]

Hello lookOutBelow,

Please see the link below for more information why you can expect differences between backtests and running strategies live.

http://www.ninjatrader.com/support/helpGuides/nt7/discrepancies_real_time_vs_bac.htm

[lookOutBelow]

No, there is something else going on. The trades are triggering at completely different times under completely different circumstances. Like I said, it isn't a slippage thing. And not a volume thing either. I understand the differences between live and backtesting and how fills are made and bars are built. This seems to be something else.

I'm using market orders, so it should be filling immediately, even if the price is way off. So, that isn't the case either when you see 1 hour difference in an extremely liquid (ES for example) instrument.


I'm going to let it run again today and more closely scrutinize the differences. I've had problems in the past with backtesting giving me different results after no changes to parameters. Something is still screwy to that effect and may be related.

[dave1992]

If trades enter at wildly different times isn't it a fairly simple task to have a print statement outputting all the criteria and just compare them?

[lookOutBelow]

If trades enter at wildly different times isn't it a fairly simple task to have a print statement outputting all the criteria and just compare them?

yeah. If after today, I see the same weird problems, I'll have to do that. There has got to be a reason for this. Maybe, printing out every single value at the time of the trade would help. I'm just overly frustrated. This isn't the first time I've had issues with backtesting giving me strange results from one minute to the next.

[dave1992]

Yes, happens to me too. Actually I couldn't develop properly without that print statement, and all my strats have one. It's a bit of a hassle, but invaluable for going from backtest to replay to live. There are plenty of diffs between replay and live too! On the plus side, when you sort out these diffs you occasionally spot something important in your logic that you'd missed.

[lookOutBelow]

After testing against 10/19's live/paper trading and backtesting the same date, I can see that the trades are off quite a bit again. I also downloaded Market Replay Data for 10/19 and had a third set of different results!

The link provided by Jason gives some examples of why tick data can be different from backtests and live and even how two backtests can be different. My question is, is it more likely to be closer to accurate if I use minute bars instead of ticks? Or do we have the same issues either way?

[dave1992]

I would have thought it's more likely to be accurate. There is less data and less issues about timestamps.

[lookOutBelow]

Could someone from NT answer this?


Is it more likely that backtests and live trading will be closer to the same if I use minute bars instead of tick bars? Or do we have the same issues either way? Right now the differences with tick bars are so huge, backtesting has very little relevance to what actually happens during the day.

[NinjaTrader_Jason]

It would depend on what tick interval you used before and what minute interval you are going to use. If you were using a 10 tick interval and subsequently backtest using a 15 minute interval with the same set of historical data, results will be closer since less bars are used.

[lookOutBelow]

It would depend on what tick interval you used before and what minute interval you are going to use. If you were using a 10 tick interval and subsequently backtest using a 15 minute interval with the same set of historical data, results will be closer since less bars are used.

I was using a 150 tick interval for this. And would switch to a 1min if it would help.



I don't understand how people are working around this. It seems like such a deal killer. Rendering backtesting practically useless (my live trading and backtesting of the same date are off by hundreds $$ on one instrument a day and sometimes in ONE TRADE. Yesterday, I had a loss in backtesting of $500 on one instrument. But in live/sim it was an over $1000 gain. That $1500 differences equated to a separation of hours on the entry exit times and a huge tick differential.

It is at the point now where I'm questioning the results of any backtesting I've done. Does it really apply to future trading at all? I realize slippage and conditions may change. But these trades are drastically different at times. I have a hard time trusting anything backtesting has told me.

[dave1992]

I don't understand why you're not putting some debugging in. Aren't you intrigued to find out exactly what the difference is? Once you know, you can probably sort it out. If my backtest and live trades were occurring at wildly different times I'd want to know why, not simply - the data must be wrong.

[ThatManFromTexas]

I was using a 150 tick interval for this. And would switch to a 1min if it would help.



I don't understand how people are working around this. It seems like such a deal killer. Rendering backtesting practically useless (my live trading and backtesting of the same date are off by hundreds $$ on one instrument a day and sometimes in ONE TRADE. Yesterday, I had a loss in backtesting of $500 on one instrument. But in live/sim it was an over $1000 gain. That $1500 differences equated to a separation of hours on the entry exit times and a huge tick differential.

It is at the point now where I'm questioning the results of any backtesting I've done. Does it really apply to future trading at all? I realize slippage and conditions may change. But these trades are drastically different at times. I have a hard time trusting anything backtesting has told me.

The following is just my "2 cents worth" based upon my personal experience , as always , your mileage may vary.

In the past 10 years , first with Trade Station then with Ninja Trader, I have NEVER found a correlation between back testing and actual sim/live trading in Tick charts, Minute charts or Range Bar charts.

[lookOutBelow]

I don't understand why you're not putting some debugging in. Aren't you intrigued to find out exactly what the difference is? Once you know, you can probably sort it out. If my backtest and live trades were occurring at wildly different times I'd want to know why, not simply - the data must be wrong.

I've done that Dave. I'll post more later when I have time.

But, even if I confirm things are weird, it doesn't mean I can simply "sort it out". If it is an issue inherent with how NT handles backtesting (as seems to be suggested on other forums were people are having their own backtesting problems).