Huge discrepancies in backtests

[samish18]

It appears that starting immediately after the strategy is running live, the values diverge

[samish18]

I did another livetest/backtest comparison and found that this time, the data that once again started on the same bar, diverged when the historical data was loaded for the live strategy. The data diverges on the first bars on 9/25, BEFORE THE STRATEGY IS EVEN LIVE. How does one fix this?

[NinjaTrader_Jesse]

Hello samish18,

I won't be able to tell what is happening from the output, are you saying that you got a different result when you ran the strategy on historical vs live?

[samish18]

Yes, that is what this thread is about - very different results between live and backtest over the same period. You asked me to give more information using prints: I have shown 2 instances where this divergence occurs.

[NinjaTrader_Jesse]

Hello samish18,

I would need to see how the script is coded as mentioned however the output is mainly for you to use to find where the problem occurs. I would suggest doing the print in post 12 to make sure the two tests started on the exact same data point. Beyond that if the indicators calculate differently in realtime vs historical that may be due to the settings you had used in the strategy. An additional step would be to debug the indicators to see how the math is calculating on the bars where you see deviations.

[samish18]

We are now going in circles as we have already gone through all of this: I previously uploaded the script. You can see from the prints that it does indeed start on the same bar.

What settings could make the indicators calculate differently in real time vs backtest?

Additionally, as I showed with the second set of prints, the bars themselves were different between live and backtest - when I pulled up the bars on a chart afterwards they still differed from the backtest.

If you are unable to help me resolve these issues, please refer me to someone who can.

[NinjaTrader_Jesse]

Hello samish18,

There has not been any script uploaded with prints in this thread so I wouldn't be able to compare the output you provide against anything.

The scripts Calculate setting could cause differences between realtime and historical.

If the bars in the test are different then you should not expect the tests to be similar. The bars would need to be exactly identical in both tests to see a similar outcome. I would suggest to remove your historical data and redownload it if you see differences between the chart and backtest.

We have a guide that goes over debugging scripts between realtime and historical tests in the following link, that goes over how to use prints and compare them to identify differences.

https://support.ninjatrader.com/s/article/Developer-Guide-Comparing-Strategy-Results?language=en_US

[samish18]

It is the script uploaded before with printing the time on each bar and printing when a crossover occurs - as suggested.

I am always using calculateOnBarClose.

As you can see from both comparisons I sent, both tests started on the exact same bar. They only deviated later

[NinjaTrader_Jesse]

Hello samish18,

Are you referring to a different thread for the script? The attached EMAstrat had no prints which is why I originally suggested that you use prints for debugging.

If you have confirmed that the tests are identical meaning that the bars are exactly identical then the next step would be to debug the indicator being used. If the bars were not identical like you had mentioned then those are not equal tests and should not produce similar results.

[samish18]

The bars started identical (on the same bar), but differed later on - how might I fix this?

[NinjaTrader_Jesse]

Hello samish18,

You would need to reload the historical data if you had a difference between tests. Generally you would need to do the following steps if you are comparing a realtime to historical test.

Reload all historical data for the instrument
Apply the strategy and allow it to work in realtime
Collect results
Reload the strategy
It will now process those same bars as historical
Collect results
Compare two sets of results

[samish18]

I am attaching another test showing the same thing. Please look at the attached files as this will keep us from continuing to go in circles - on each bar, the script prints 1) the time 2) the indicator value 3) the 10 ema value and 4) the 50 ema value.

There are two huge discrepancies between the two for which you have provided no answer:
1. The bars in the livetest proceed 9/27 11:45, 9/27 12:00, 9/27 12:05
in the backtest the bars proceed 9/27 11:45, 9/27 11:50, 9/27 12:00, 9/27 12:05
ANOTHER BAR IS ERRONEOUSLY ADDED

2. Comparing the "closes" column of the 2 .txt files which has the indicator value, the value is consistent between the two until the strategy begins running live - this indicator is simply the ratio between two instruments

[NinjaTrader_Jesse]

Hello samish18,

Unfortunately the output wont help me to see why there is a difference, that is mainly only helpful for you to analyzer in contrast to the specific chart where the strategy is applied.

For the data being different I wouldn't be able to tell why that may have happened without testing the exact same scenario on my end. Did you follow the steps that I listed in that exact order when doing the comparison? To compare realtime bars to historical you would have to run the script in realtime and collect the prints from the realtime bars and then re run the strategy historically which it will now cover the same bars as historical. If you are doing that process the bars can't change, only the script is reloaded and no new data is going to be included. That is done by pressing F5 after collecting the realtime results to reload the strategy.

For the indicator value you would need to debug the indicator if it starts to have a discrepancy when entering live. You would have to add prints into that indicator to see why the math is evaluating that way on realtime bars vs historical bars.

[samish18]

Yes, I did follow the steps above. Even now, when I load the data on a chart from the control center vs looking at a chart in the strategy analyzer it differs.

This is the indicator I'm using. I am just using the "spread" variable which is the ratio of two instruments. Why might the values differ live.

[NinjaTrader_Jesse]

Hello samish18,

If that indicator produces differences between historical and realtime you would have to add prints into its logic to see why that is. You would have to output the values used in the math and then manually calculate the values when the deviation happens to see why the math is different.