Huge discrepancies in backtests

[samish18]

This is absolutely ridiculous that I need to ask again. Fix your backtesting! I am running simple crossover strategies and getting vastly different results between live and running backtests over the same period. I would like a straight answer - I am running on ES 5 minute timeframe, same data.

[NinjaTrader_Jesse]

Hello samish18,

When comparing realtime and historical results there are some reasons that would cause the strategy to calculate differently. While we would be unable to provide an exact answer of why that is with your custom strategy you can debug the strategy to find out the exact answer. Because every strategy is different there is not one single answer as to why that would be different, that heavily depends on the settings you used and how the strategy was coded.

We have a guide linked below that goes over how you can add debugging to the strategy to explore differences between tests. The guide goes over comparing historical and realtime which is what you explained.

This page lists some common items which will cause differences between realtime and historical, you can use this page to see if any of these apply to your use case:

https://ninjatrader.com/support/helpGuides/nt8/discrepancies_real-time_vs_bac.htm?zoom_highlightsub=discre

This guide goes over how to add debugging and then compare results. The short summary is that you add prints into your code and then collect the output after running the strategy in realtime. Then restart the strategy so it processes historically and then collect those prints as well. You can then compare the prints to see how the strategy logically worked in both use cases. Based on your observations you can make any changes needed to have the strategy results become more close to realtime.
https://support.ninjatrader.com/s/ar...language=en_US

[samish18]

That is not a straight answer. When I restart the strategy and process historical data, it is identical to the backtest.

[NinjaTrader_Jesse]

Hello samish18,

Unfortunately there are no specific details from your description that would help to determine why there may have been differences originally. You would need to use prints if you are seeing differences to identify what is causing them. There are many reasons why historical and realtime tests may not match based on a variety of factors.

[samish18]

I have been using prints. The EMA values differ when running live. When loading a strategy, the prints on the loaded data are identical to those in backtests - the loaded data is the same as the backtest.

[NinjaTrader_Jesse]

Hello samish18,

Are you using OnEachTick or OnPriceChange for the Calculate setting? That could cause differences in indicator values when running live vs historical.

[samish18]

No, I made sure it is onBarClose

[NinjaTrader_Jesse]

Hello samish18,

In that case you would have to continue to use prints to see why that is being calculated differently. That may be due to having different starting points for data, each test would need to be exactly identical to get similar results.

[samish18]

When creating a new strategy, the template that pops up includes several specifications in one of the first sections (including calculate = onBarClose). Is it necessary to keep these or would simply selecting onBarClose when setting up the strategy from the strategies tab suffice?

[NinjaTrader_Jesse]

Hello samish18,

The default that is always used is OnBarClose, you can select a different default which will be used every time you apply the strategy from that screen. It is not required to set a value for that since OnBarClose is used by default.

[samish18]

I attached the strategy. Any clues as to why the discrepancy might be occurring?

[NinjaTrader_Jesse]

Hello samish18,

By just looking at the code it would be impossible to say why your unique test had differences, you would have to add prints into this script to know what happened. A starting point would be to output when your entry conditions become true just to see if the conditions are even happening on the same bars in the test. If any of the executions are different that would be a reason for discrepancies.

if (CrossAbove(EMA(10),EMA(50),1) && Close[0] > EMA(10)[0]) {
    Print("CrossAbove condition: " + Time[0]);
    EnterLong();
}​

You could then run the script in realtime and collect the output, then re run the script in historical to see if the prints match up.

I would also suggest printing the starting bar to make sure your tests are using the same data:

if(CurrentBar == 0)
    Print("First bar: " + Time[0]);

[samish18]

I previously added prints and did so. The starting points were identical and the orders executed when the conditions became true - it seems that the data diverges once it is running live

[NinjaTrader_Jesse]

Hello samish18,

If you have a test where you have done that kind of work I would need to see what you printed to find the difference that is happening. For example if the indicators are calculating different in realtime I would need to see how the strategy is coded, where your prints are and what settings are used in the strategy. It would also be helpful to provide a sample of the output you had observed that shows the difference that you see.

[samish18]

Attached are the results from the backtest and liverun over the same period. As you can see, different trades are executed