Backtesting and live (paper) trades are different

[lookOutBelow]

Could be loads of reasons - different data feed, bad values in ticks etc etc. Fix that and you're done.

LOL. Not sure those things are fixable.

Anyhow, thanks. I'll do some fiddling after market hours.

[NinjaTrader_Ray]

Btw, I never said or meant that NT was "flawed".[/SIZE][/COLOR]

That reference came from a post you made on a different forum. If you did not mean it litterally, I apologize for misunderstanding your meaning. I hope you are able to gain a better understanding of what is happening. My team is here to help.

[lookOutBelow]

Thanks for providing your take. I'll chew on that as I try to figure out what is going on.

Maybe these closed bars in backtest would have triggered logic that generated signals based on an open position but in live, these conditions were missed
o The result is that this trade has now completed changed the outcome of your strategy for the rest of the day

This makes sense, but it is not uncommon for the very first trade of the day to be completely off or non existent in one or the other. Which makes things even more strange.


Btw, I never said or meant that NT was "flawed". But perhaps just the way things are handled- whether it is an issue with how data comes from the data provider or not--, it causes these issues to come up in certain situations using a particular methodology (whether that is a combination of session templates, tick charts, data provider, whatever).

Thanks

[dave1992]

18 out of 20 would be a blessing (I'd be lucky to get 8 out of 20). Unfortunately I'm not getting anywhere near that. So, when I have the data in front of me, what exactly do you suggest that I look for and compare between backtest and live? I can't exactly compare values at trade time since trade times are no where near each other in most instances.

Here's what I would do.

Put a print in which prints your trade setup conditions. Print that for every bar, plus a timestamp of the bar. Run in both backtest and live (or, if you like, on the live one only print the line when all trade setup conditions are true - this will vastly reduce your output in live)

Then take your trade from live (or backtest). Look at the value of each condition. Look at the same time bar in backtest. Look at the values there. If you've done everything properly, at least one of the values will be different. Guaranteed.

Find out why that value differs. Could be loads of reasons - different data feed, bad values in ticks etc etc. Fix that and you're done. It may be that the two runs are starting from different time points, for example - that would certainly explain what you see.

[NinjaTrader_Ray]

I assume you have read the link Jason posted regarding the differences between live and backtest. Here is my take which includes some comments already stated by other users.
- You could only expect a backtest and live to be equal if all possible variables (data set, order fills etc..) are eliminated which is not realistic
- A live run can be vastly different than a backtest if you are to measure against a small time sample (one day for example)
- Let me give you an example
o You are running a backtest on 150 tick bars
o In backtest each bar will be evaluated and the state of your order and position is already known on the next bar after the signal is generated
o In live, same signal is generated, order is submitted to the live simulator (or market, makes no difference), during the time the order is filled several bars have closed
o Maybe these closed bars in backtest would have triggered logic that generated signals based on an open position but in live, these conditions were missed
o The result is that this trade has now completed changed the outcome of your strategy for the rest of the day
o The PnL change could very well be drastically different
- Time based bars in theory should reduce the discrepancy however, backtesting and comparing results at a daily level is far too granular in my opinion.
- 1 tick difference in real time data vs historical can completely alter the bars and result in vastly different results (just another example)
- Analyzing the difference at a granular level to understand why the results are different is a comprehensive task but if you take the time to do this, you will be provided an logical answer to the discrepancy and I assure you that it will not be due to some “flaw” in NinjaTrader

[lookOutBelow]

If I run a 1-tick strat in backtest and live, I usually get 18/20 trades agree between the two, and I usually know why the other two didn't agree. I can live with that.

18 out of 20 would be a blessing (I'd be lucky to get 8 out of 20). Unfortunately I'm not getting anywhere near that. So, when I have the data in front of me, what exactly do you suggest that I look for and compare between backtest and live? I can't exactly compare values at trade time since trade times are no where near each other in most instances.

[dave1992]

I've done that Dave. I'll post more later when I have time.

But, even if I confirm things are weird, it doesn't mean I can simply "sort it out". If it is an issue inherent with how NT handles backtesting (as seems to be suggested on other forums were people are having their own backtesting problems).

Yes, it probably does mean you can sort it out. At the moment, we don't know what's wrong. What is unlikely, is that a slight difference in tick times makes all of your trades execute at different times. If I run a 1-tick strat in backtest and live, I usually get 18/20 trades agree between the two, and I usually know why the other two didn't agree. I can live with that.

Without any debugging data, we have no idea why yours fails. It may be that it's inherent, but there's no evidence presented for that yet, and my experience seems against that.

I honestly think the data will help you sort this out. It always has for me.

[lookOutBelow]

The following is just my "2 cents worth" based upon my personal experience , as always , your mileage may vary.

In the past 10 years , first with Trade Station then with Ninja Trader, I have NEVER found a correlation between back testing and actual sim/live trading in Tick charts, Minute charts or Range Bar charts.

I've never used Tradestation, so I cannot comment. However, just to be straight, I don't expect backtesting and sim/live to line up exactly. But, I do (especially using calculateonclose = true) expect backtesting and sim to be in the same general neighborhood. Otherwise, it kind of voids the usefulness of backtesting if you ask me.

Again, we are talking big differences. As another example to the one above. In one instrument yesterday, I had 10 trades take place in sim/live, but in backtest, there were 2. And only one of those was anywhere close to one of the sim/live trades. If you are/have experienced the same, I'd certainly like to hear it. Talking with some other previous NT users, it appears I'm not the only one having this issue. And perhaps I need to switch platforms like they did. I just like NT and would rather not. I've invested so much time into it.


At the end of day, I'm going to look closely at the results again. I'm not going to open a Strategy Analyzer window until the day is over. Just to rule out that it is somehow tainting data. Seems unrealistic that it would, but at this point, I'd try just about anything.

[lookOutBelow]

I don't understand why you're not putting some debugging in. Aren't you intrigued to find out exactly what the difference is? Once you know, you can probably sort it out. If my backtest and live trades were occurring at wildly different times I'd want to know why, not simply - the data must be wrong.

I've done that Dave. I'll post more later when I have time.

But, even if I confirm things are weird, it doesn't mean I can simply "sort it out". If it is an issue inherent with how NT handles backtesting (as seems to be suggested on other forums were people are having their own backtesting problems).

[ThatManFromTexas]

I was using a 150 tick interval for this. And would switch to a 1min if it would help.



I don't understand how people are working around this. It seems like such a deal killer. Rendering backtesting practically useless (my live trading and backtesting of the same date are off by hundreds $$ on one instrument a day and sometimes in ONE TRADE. Yesterday, I had a loss in backtesting of $500 on one instrument. But in live/sim it was an over $1000 gain. That $1500 differences equated to a separation of hours on the entry exit times and a huge tick differential.

It is at the point now where I'm questioning the results of any backtesting I've done. Does it really apply to future trading at all? I realize slippage and conditions may change. But these trades are drastically different at times. I have a hard time trusting anything backtesting has told me.

The following is just my "2 cents worth" based upon my personal experience , as always , your mileage may vary.

In the past 10 years , first with Trade Station then with Ninja Trader, I have NEVER found a correlation between back testing and actual sim/live trading in Tick charts, Minute charts or Range Bar charts.

[dave1992]

I don't understand why you're not putting some debugging in. Aren't you intrigued to find out exactly what the difference is? Once you know, you can probably sort it out. If my backtest and live trades were occurring at wildly different times I'd want to know why, not simply - the data must be wrong.

[lookOutBelow]

It would depend on what tick interval you used before and what minute interval you are going to use. If you were using a 10 tick interval and subsequently backtest using a 15 minute interval with the same set of historical data, results will be closer since less bars are used.

I was using a 150 tick interval for this. And would switch to a 1min if it would help.



I don't understand how people are working around this. It seems like such a deal killer. Rendering backtesting practically useless (my live trading and backtesting of the same date are off by hundreds $$ on one instrument a day and sometimes in ONE TRADE. Yesterday, I had a loss in backtesting of $500 on one instrument. But in live/sim it was an over $1000 gain. That $1500 differences equated to a separation of hours on the entry exit times and a huge tick differential.

It is at the point now where I'm questioning the results of any backtesting I've done. Does it really apply to future trading at all? I realize slippage and conditions may change. But these trades are drastically different at times. I have a hard time trusting anything backtesting has told me.

[NinjaTrader_Jason]

It would depend on what tick interval you used before and what minute interval you are going to use. If you were using a 10 tick interval and subsequently backtest using a 15 minute interval with the same set of historical data, results will be closer since less bars are used.

[lookOutBelow]

Could someone from NT answer this?


Is it more likely that backtests and live trading will be closer to the same if I use minute bars instead of tick bars? Or do we have the same issues either way? Right now the differences with tick bars are so huge, backtesting has very little relevance to what actually happens during the day.

[dave1992]

I would have thought it's more likely to be accurate. There is less data and less issues about timestamps.