BackTests Results not equal to Reality

[Capablanca]

I coded a Strategy that uses two instruments to trigger entries in the ES. (the triggers are indicated with yellow arrows)
However when the strategy is run with live data, the results are quite different from the backtests.

The triggers are exactly the same, nevertheless the prices at which the positions enters and exits are different.
I guess it has something to do with the ticks synchronization of the two instruments.

What can I do to guarantee that the backtest respresents the reality?

Thanks!

[NinjaTrader_Jesse]

Hello Capablanca,

Thank you for the post.

In most cases you can expect to see some differences between a realtime test and a historical test due to differences in how the two modes operate. You can find the most common differences listed in the help guide here: https://ninjatrader.com/support/help...ghtsub=discrep

In this case you are likely seeing a difference between the historical fill engine and live fill engine which is related to the incoming data opposed to historical data points and logic.

You can experiment with using intrabar granularity by adding a secondary tick series. This can be used to simulate a more granular strategy in a backtest which can in some situations help more closely reflect what it will do in realtime depending on the overall logic used: https://ninjatrader.com/support/help...ipt_strate.htm If this is simply the fill price that is different and otherwise the rest of the logic is happening as expected between tests, this would not likely assist in providing any greater accuracy.

I look forward to being of further assistance.

[Capablanca]

Thank you Jesse,

The screenshots attached in my post are from a strategy running in a chart: the second one from what resulted with the live data and the first one after I pressed F5 and restarted the strategy.
What makes me think if the data feed synchronization varies between the two instruments when the strategy is live and when it's backtested.
Did I explained myself?

Is there a solution to this?

[NinjaTrader_Jesse]

Hello Capablanca,

After pressing F5, the strategy is going to reprocess the data which was received live as historical data. The live trades that had happened live will now be re processed through historical data so you may see a different result after doing that.

In this case you are comparing historical backtest results to live test results when you press F5 to reload the one script. You would need to leave both scripts running in realtime after having started them at the exact same time to see identical results.

I look forward to being of further assistance.

[Capablanca]

Thanks Jesse,

Does NT7 has the option of High Fill Resolution in the Strategy Analyzer?
Would that improve the similarity of the Historical vs Real Time results?

[NinjaTrader_Jesse]

Hello Capablanca,

I just wanted to confirm, is this post in reference to NT7 or NT8? I see that we are in the NT8 forum, after further inspection it looks like NT7 in the image and now another NT7 question has come up. Should this post be moved to the NT7 forums or is this post comparing NT7 against NT8 features? I previously replied with an NT8 answer due to the forum that we are in.

In any case, the following can help explain NT7 and NT8 in this respect:

NT7 and NT8 are different in this area, nt7 did not have a fill resolution property. NT7 has what is called a Fill Type which can be default or liberal.

https://ninjatrader.com/support/helpGuides/nt7/fill_types.htm?zoom_highlightsub=filltype

In NT8 FIllType is no longer a NinjaScript type and was moved to being internal logic with a set specification.

https://ninjatrader.com/support/helpGuides/nt8/en-us/understanding_historical_fill_.htm

In NT8 this works differently than NT7, NT8 can use the default setting and will use the existing bar type and interval that you are running the backtest on to fill your orders or can use a secondary bar series to be used as the price data to fill your orders. In NT7 the FillType only controls the fill action but does not add any additional data. If you wanted to replicate the fill resolution from NT8, that would require both using a secondary series in your script for added granularity and also to create a custom fill type which is similar to the help guide description of NT8's fill process. NT8's fill resolution is unique to NT8.

I look forward to being of further assistance.

[Capablanca]

Thanks Jesse,

My mistake, I didn't see it was a forum of NT8. Indeed I'm using NT7.
Thanks for your reply.

[Capablanca]

Dear Jesse,

sorry to bother with the same issue.

But I've been doing some strategy tests and I get the following results:

A) The strategy behaves the same using the replay and live data
B) The strategy behaves the same using the strategy analyzer and the F5 in a chart.
C) However, the results vary significantly from points A) and B)

The whole purpose of the backtesting is that the results are the quite similar using live data.
Nevertheless I would expect the backtest to be IDENTICAL to a replay, since historical data is being used.

The images I'm attaching are using a Replay (point A) and then Pressing F5 (point B).

[NinjaTrader_Jesse]

Hello Capablanca,

C) However, the results vary significantly from points A) and B)

Correct, you are comparing two different tests.

In A you are using live data in both tests.
In B you are using historical data in both tests, when you press F5 your script re processes all the historical data as a "backtest", the same as it does in the analyzer.

A and B should not be identical as you are comparing a historical test versus a realtime test, there are expected differences between these types of tests which you can find some information on here: https://ninjatrader.com/support/help...hlightsub=disc

The whole purpose of the backtesting is that the results are the quite similar using live data.

Backtesting can give you insight on how the script will perform in realtime, however you are using less granular data and also simulating fills so there will be differences because this is not the same as running in realtime. Using the market replay connection would be more similar to how the script will work in realtime as that is realtime data.

Nevertheless I would expect the backtest to be IDENTICAL to a replay, since historical data is being used.

This is not correct, replay is not using historical data to play back. You do see some historical data to populate the range you selected however it is using Playback data which is recorded live data. Historical data is much less granular, you only have 4 points of data versus building a bar with all data points received.

I look forward to being of further assistance.

[Capablanca]

I understand now.
Is there a way to add more granularity to a backtest?
Maybe adding two additional instruments with smaller tick periods (since I am using a multi time series strategy).

[NinjaTrader_Jesse]

Hello Capablanca,

Yes,

You can experiment with using intrabar granularity by adding a secondary tick series. This can be used to simulate a more granular strategy in a backtest which can in some situations help more closely reflect what it will do in realtime depending on the overall logic used: https://ninjatrader.com/support/helpGuides/nt7/backtesting_ninjascript_strate.htm If this is simply the fill price that is different and otherwise the rest of the logic is happening as expected between tests, this would not likely assist in providing any greater accuracy.

You can experiment adding multiple series, however this changes how your logic is executed so you will need to also make sure you understand how multi series scripts work before adding more series. I would suggest to review the following page in addition to the above sample: https://ninjatrader.com/support/help...lightsub=multi

You would mainly need to add a 1 tick series if you wanted the most granular calls to OnBarUpdate. As noted this really only changes the way your script if called, if you see that you are only getting differences in fills this is not likely going to aid in more accuracy. You can modify your logic to submit orders to the 1 tick series to more closely recreate a realtime test as well.

I look forward to being of further assistance.

[ricardo88]

Hi NinjaTrader Customer Service,

I've created a mock (simulator) of filled orders and this gave me same results as expected in live data. I would like to know how can I help to Ninjatrader to improve backtesting because this is very important to Strategy Tests.

[NinjaTrader_Jesse]

Hello ricardo88,

If you created something which works for you and your strategy I would suggest to just use that if it works for your goals. We can additionally submit bug reports in case you see some kind of a problem in the existing backtest tool based on how it is designed to work.


Please let me know if I may be of further assistance.