Backtesting and live (paper) trades are different

[lookOutBelow]

The following is just my "2 cents worth" based upon my personal experience , as always , your mileage may vary.

In the past 10 years , first with Trade Station then with Ninja Trader, I have NEVER found a correlation between back testing and actual sim/live trading in Tick charts, Minute charts or Range Bar charts.

I've never used Tradestation, so I cannot comment. However, just to be straight, I don't expect backtesting and sim/live to line up exactly. But, I do (especially using calculateonclose = true) expect backtesting and sim to be in the same general neighborhood. Otherwise, it kind of voids the usefulness of backtesting if you ask me.

Again, we are talking big differences. As another example to the one above. In one instrument yesterday, I had 10 trades take place in sim/live, but in backtest, there were 2. And only one of those was anywhere close to one of the sim/live trades. If you are/have experienced the same, I'd certainly like to hear it. Talking with some other previous NT users, it appears I'm not the only one having this issue. And perhaps I need to switch platforms like they did. I just like NT and would rather not. I've invested so much time into it.


At the end of day, I'm going to look closely at the results again. I'm not going to open a Strategy Analyzer window until the day is over. Just to rule out that it is somehow tainting data. Seems unrealistic that it would, but at this point, I'd try just about anything.

[dave1992]

I've done that Dave. I'll post more later when I have time.

But, even if I confirm things are weird, it doesn't mean I can simply "sort it out". If it is an issue inherent with how NT handles backtesting (as seems to be suggested on other forums were people are having their own backtesting problems).

Yes, it probably does mean you can sort it out. At the moment, we don't know what's wrong. What is unlikely, is that a slight difference in tick times makes all of your trades execute at different times. If I run a 1-tick strat in backtest and live, I usually get 18/20 trades agree between the two, and I usually know why the other two didn't agree. I can live with that.

Without any debugging data, we have no idea why yours fails. It may be that it's inherent, but there's no evidence presented for that yet, and my experience seems against that.

I honestly think the data will help you sort this out. It always has for me.

[lookOutBelow]

If I run a 1-tick strat in backtest and live, I usually get 18/20 trades agree between the two, and I usually know why the other two didn't agree. I can live with that.

18 out of 20 would be a blessing (I'd be lucky to get 8 out of 20). Unfortunately I'm not getting anywhere near that. So, when I have the data in front of me, what exactly do you suggest that I look for and compare between backtest and live? I can't exactly compare values at trade time since trade times are no where near each other in most instances.

[NinjaTrader_Ray]

I assume you have read the link Jason posted regarding the differences between live and backtest. Here is my take which includes some comments already stated by other users.
- You could only expect a backtest and live to be equal if all possible variables (data set, order fills etc..) are eliminated which is not realistic
- A live run can be vastly different than a backtest if you are to measure against a small time sample (one day for example)
- Let me give you an example
o You are running a backtest on 150 tick bars
o In backtest each bar will be evaluated and the state of your order and position is already known on the next bar after the signal is generated
o In live, same signal is generated, order is submitted to the live simulator (or market, makes no difference), during the time the order is filled several bars have closed
o Maybe these closed bars in backtest would have triggered logic that generated signals based on an open position but in live, these conditions were missed
o The result is that this trade has now completed changed the outcome of your strategy for the rest of the day
o The PnL change could very well be drastically different
- Time based bars in theory should reduce the discrepancy however, backtesting and comparing results at a daily level is far too granular in my opinion.
- 1 tick difference in real time data vs historical can completely alter the bars and result in vastly different results (just another example)
- Analyzing the difference at a granular level to understand why the results are different is a comprehensive task but if you take the time to do this, you will be provided an logical answer to the discrepancy and I assure you that it will not be due to some “flaw” in NinjaTrader

[dave1992]

18 out of 20 would be a blessing (I'd be lucky to get 8 out of 20). Unfortunately I'm not getting anywhere near that. So, when I have the data in front of me, what exactly do you suggest that I look for and compare between backtest and live? I can't exactly compare values at trade time since trade times are no where near each other in most instances.

Here's what I would do.

Put a print in which prints your trade setup conditions. Print that for every bar, plus a timestamp of the bar. Run in both backtest and live (or, if you like, on the live one only print the line when all trade setup conditions are true - this will vastly reduce your output in live)

Then take your trade from live (or backtest). Look at the value of each condition. Look at the same time bar in backtest. Look at the values there. If you've done everything properly, at least one of the values will be different. Guaranteed.

Find out why that value differs. Could be loads of reasons - different data feed, bad values in ticks etc etc. Fix that and you're done. It may be that the two runs are starting from different time points, for example - that would certainly explain what you see.

[lookOutBelow]

Thanks for providing your take. I'll chew on that as I try to figure out what is going on.

Maybe these closed bars in backtest would have triggered logic that generated signals based on an open position but in live, these conditions were missed
o The result is that this trade has now completed changed the outcome of your strategy for the rest of the day

This makes sense, but it is not uncommon for the very first trade of the day to be completely off or non existent in one or the other. Which makes things even more strange.


Btw, I never said or meant that NT was "flawed". But perhaps just the way things are handled- whether it is an issue with how data comes from the data provider or not--, it causes these issues to come up in certain situations using a particular methodology (whether that is a combination of session templates, tick charts, data provider, whatever).

Thanks

[NinjaTrader_Ray]

Btw, I never said or meant that NT was "flawed".[/SIZE][/COLOR]

That reference came from a post you made on a different forum. If you did not mean it litterally, I apologize for misunderstanding your meaning. I hope you are able to gain a better understanding of what is happening. My team is here to help.

[lookOutBelow]

Could be loads of reasons - different data feed, bad values in ticks etc etc. Fix that and you're done.

LOL. Not sure those things are fixable.

Anyhow, thanks. I'll do some fiddling after market hours.

[lookOutBelow]

NT freezes when you do certain operations

Okay. So, I realize something with NT that may not be the root cause of this problem stated in the original post. But in the very least, it is a serious issue that affects performance. And it certainly affects running strategies.

Lucky for everyone, it is easily reproduce-able.


Yesterday, I decided to let NT run throughout the day and not touch it. Didn't edit any strategies, didn't compile any code, didn't do anything. Guess what? The results were much closer than previous days where I had wild discrepancies. Live/sim and backtesting almost lined up as you would expect.

With that in mind, this morning, I did a couple of simple tests and I saw that NT actually FREEZES when you do certain operations.

Open up ES in a DOM and a time and sales. Also open a Market Analyzer window and put several instruments in there. You can test this yourself.

Now, open a strategy analyzer and run a backtest over a long period of time, say 6 months. Now, after you get your results, hit your "Chart" tab. The time it takes NT to calculate a chart , it actually freezes either the data coming into NT or the program freezes for whatever reason.

Now, go edit one of your strategy code. Now compile it. Everything freezes again!


Now, if you are back testing and editing code like I've been doing all day long, these freezes are going to screw up entries and exits. This may be the issue!!! Or maybe not. Whatever the case, this is a problem itself.

[NinjaTrader_Jason]

Hello lookOutBelow,

Could you please send us your strategy, so we can test it on our end. I have tried to reproduce it using the SampleMACrossOver strategy, but so far I was unable to do so.

Please send it to support[at]ninjatrader[dot]com and reference this forum thread. You can export strategies as per the instructions at the link below.

http://www.ninjatrader.com/support/helpGuides/nt7/export.htm

In addition, could you perhaps include an example of how you edit your strategy code if possible or provide more information on this? That way we can better replicate this step on our end as well.

[lookOutBelow]

Could you please send us your strategy, so we can test it on our end. I have tried to reproduce it using the SampleMACrossOver strategy, but so far I was unable to do so.

I can reproduce this using SampleMACrossOver. Look at my screenshot to get the settings and try to match it. After running the backtest. Click on the chart. During the time it takes to build/display the charts (took around 10 seconds or more), all quotes, and everything else on NT freezes. Make sure you have a couple of DOMS open and maybe TS to see it clearly.

[lookOutBelow]

In addition, could you perhaps include an example of how you edit your strategy code if possible or provide more information on this? That way we can better replicate this step on our end as well.

Well, I do it the standard way. Tools > edit ninjascript > strategy. Don't even have to change the code. Just hit the compile button. Everything freezes until it is done compiling.

[lookOutBelow]

BTW, another easy way to freeze it is to delete an indicator or strategy.

[NinjaTrader_Ray]

I ran a test with many DOMs, T&S, MA with 40 markets on it, backtested SampleMA with your settings with a longer data set, March 1 through October 1st and for sure there is an application busy state, for me about 1 second not ten (could be difference in hardware) between my set up and yours. This is expected since as you click on the Chart Tab, loading 6+ month of 150 tick bars plus all associated trade markers takes processing time, all this is done on the main thread where all UI is being handled. Although the application UI is no responsive during this time, you will not lose any events in your strategy, all will be processed.

[lookOutBelow]

I ran a test with many DOMs, T&S, MA with 40 markets on it, backtested SampleMA with your settings with a longer data set, March 1 through October 1st and for sure there is an application busy state, for me about 1 second not ten (could be difference in hardware) between my set up and yours. This is expected since as you click on the Chart Tab, loading 6+ month of 150 tick bars plus all associated trade markers takes processing time, all this is done on the main thread where all UI is being handled. Although the application UI is no responsive during this time, you will not lose any events in your strategy, all will be processed.

Hmm. So, even if there is a long delay for charts to build or something to be compiled, and during this period the GUI is freezing, buy and sells will still happen in a strategy at the exact time they would normally happen if the GUI wasn't frozen?

Okay, then I'm at a loss for why this (seems to) significantly affected the accuracy of live trading vs backtest. I guess it is coincidence. But it seems awfully peculiar that the day I decided to not do any compiling or backtesting that this whole discrepancy issue was mostly a non-issue.