Market Replay vs Back Test results

[Malletto]

Ok. But what else is the issue if there calculating on the same thing- bar close?

[NinjaTrader_BrandonH]

Hello Malletto,

Thanks for your notes.

You must enable Tick Replay when backtesting the strategy so that indicator values in the script can update tick-by-tick historically as they would in real-time.

See this help guide page about Tick Replay linked on post # 27 for details about how Tick Replay works.

[Malletto]

But what if the strategy is a tick strategy?

[NinjaTrader_BrandonH]

Hello Malletto,

Thanks for your notes.

I have addressed your questions in the support ticket you have open with the NinjaTrader Support deparment.

Please refer to the email I sent replying to your question and direct any further inquiries to that email instead of posting on this forum thread.

[Ymcapital]

I believe there's an issue with the timestamps in the NT backtest with Kagi bars. The new Kagi bar doesn't print until there is a reversal and in the backtest NT fills you retroactively. Generally speaking, using a secondary time series such as tick data can solve issues seen with backtesting Ninzarenko bars or HA candles but because of what I see with the backtests, tick series data doesn't solve this issue.

Brandon, can you confirm if that's accurate?

[Malletto]

I was told that backtesting does open, high, low, close. That’s the issue.

[NinjaTrader_BrandonH]

Hello Ymcapital and Malletto,

Thanks for your notes.

The issue here is really the use of kagi bars. The issue of backtests being different from real time would be more easily seen with exotic bars types like Point and Figure and Kagi.

See this help guide page: http://www.ninjatrader.com/support/h.../bar_types.htm

To backtest those more accurately we normally suggest either running in Playback using Market Replay or submitting orders to an added 1 tick series in backtesting, so a higher fidelity fill can be worked with - however, the 1 tick series would simply provide more accurate fill prices.

You'd get the most accurate to real time results using Playback.

[Malletto]

No. Isn’t the the backtest using OHLC? My live only trades based of close. That’s a big difference

[NinjaTrader_BrandonH]

Hello Malletto,

Thanks for your notes.

When in historical data, only the Open, High, Low, and Close will be available and there will be no intra-bar data. This means actions cannot happen intra-bar, fills cannot happen intra-bar. All prices and actions come from and occur when the bar closes as this is all the information that is known.

Please review the help guide document on the differences on real-time vs backtest (historical).

https://ninjatrader.com/support/helpGuides/nt8/discrepancies_real-time_vs_bac.htm

As stated in post # 52, the issue of backtests being different from real time would be more easily seen with exotic bars types like Point and Figure and Kagi. This is noted on the help guide page linked above.

To backtest those bar types more accurately we would suggest either running in Playback using Market Replay.

Or, adding a 1-tick series to the strategy, submitting orders to that 1-tick series in backtesting with Tick Replay enabled, so a higher fidelity fill can be worked with. The 1 tick series would simply provide more accurate fill prices.

The most accurate testing results to real-time would come from testing the strategy using the Playback connection.

[Ymcapital]

Hello Ymcapital and Malletto,

Thanks for your notes.

The issue here is really the use of kagi bars. The issue of backtests being different from real time would be more easily seen with exotic bars types like Point and Figure and Kagi.

See this help guide page: http://www.ninjatrader.com/support/h.../bar_types.htm

To backtest those more accurately we normally suggest either running in Playback using Market Replay or submitting orders to an added 1 tick series in backtesting, so a higher fidelity fill can be worked with - however, the 1 tick series would simply provide more accurate fill prices.

You'd get the most accurate to real time results using Playback.

Can you confirm that there's no issue with how the Kagi bars are calculated in the Strategy Analyzer?

I have a strategy that uses underlying tick data for this exact purpose so that regardless of the timeseries the strategy is being run on, there is underlying tick data to assure the accuracy of the backtest. However, even with that all being built-in and verifying that the underlying tick data allows my strategy to work the same in live markets, the strategy analyzer backtests, and the market replay, when I run it on Kagi bars the results are different.

I wouldn't be opposed to using Market Replay but why does NinjaTrader make it such a pain in the butt to use? To begin with it's super slow to load things and it's cumbersome to only be able to run it when you're not connected to live trading. That limits when you can do that as an algorithmic trader. Why do users have to painfully, individually select each day they want to download to then test on market replay? You don't have to do any of that when using Strategy Analyzer. The platform downloads all the data automatically. Why has this not been updated since NT8 was released back in 2016? The fact it's so slow and temperamental makes it hard to use. The NT8 architecture doesn't support GPU usage to help handle the load so even though I have a 6-core Ryzen CPU with 16 GB of DDR5 ram, SSD HD, and an unlimited gigabit connection up and down on my server near the exchange, I still experience general slowness sometimes and with the Market Replay feature it really boggs down the system.

I know that's a lot to unpack and I'm not blaming you for the shortcomings but considering all this, I try to stick to Strategy Analyzer. The biggest question I have is if there is an issue with the way NT is timestamping the bars which causes strategies to enter retroactively on Kagi bars.

UPDATE: I downloaded two months of market playback data to run a test using market replay and I've waited over 20 minutes for the playback box to load. So here's another issue with running Market Replay, you can't even test for long multi-month periods, even if you're using a computer that has modern, more than capable hardware.

[NinjaTrader_BrandonH]

Hello Ymcapital,

Thanks for your notes.

I am not aware of any specific issues when it comes to backtesting on Kagi bars.

When it comes to testing on Kagi bars, the best option would be to use the Playback connection with Market Replay data. You'd get the most accurate to real-time results using Playback.

For backtesting in the Strategy Analyzer on Kagi bars, the best option would be to add a 1-tick secondary series and submit orders to that secondary series.

I have tested the SampleMACrossover that comes default with NinjaTrader and see similar results. when comparing the Strategy Analyzer Chart display to a Chart window set to the same dates being tested over.

We are tracking interest in the ability to download more than on day of market replay at a time in an existing feature request. I have added your vote to this feature request. This request is being tracked under the number SFT-2943.

As with all feature requests, interest is tracked before implementation is considered, so we cannot offer an ETA or promise of fulfillment. If implemented, it will be noted on the Release Notes page of the Help Guide.

Release Notes — https://ninjatrader.com/support/help...ease_notes.htm

If you are having an issue with using Playback with Market Replay, please write in to support[at]ninjatrader[dot]com and one of our support technicians will be happy to investigate this matter further.

[Malletto]

Then why does the backtest show different results/entries then what actually occurred real time?

[NinjaTrader_BrandonH]

Hello Malletto,

Thanks for your notes.

In your case, you would need to add intrabar granularity to your strategy and enable Tick Replay when backtesting the script to improve backtest results as we discussed over email.

Or, test the strategy using the Playback connection with Market Replay which would provide you with the most accurate results to realtime.

This is noted on post # 52 of this forum thread also.

Further, below is forum thread detailing information about comparing historical results to realtime resutls to playback results. You could use the tips on the forum thread to find what exactly are the differences between historical, realtime, and playback results.
https://forum.ninjatrader.com/forum/...nce#post100192

[Malletto]

That’s not answering my question. What makes the backtest execute differently compared to the real-time market. Please let me know

[NinjaTrader_BrandonH]

Hello Malletto,

Thanks for your notes.

The Strategy Analyzer does not have any intrabar information available to calculate logic. Whereas real-time data does have intrabar information available for calculations in the script.

Review the help guide document on the differences on real-time vs backtest (historical). This help guide page details the differences between real-time results vs backtest (historical) results.
https://ninjatrader.com/support/help...ime_vs_bac.htm

Further, when backtesting the order fills are determined by the historical fill algorithm. Real-time order fills are determined by market dynamics.

Historical Fill Processing and Understanding Order Fill Resolution: https://ninjatrader.com/support/help...ill+algorith+m

Adding intrabar granularity and enabling Tick Replay will allow your strategy backtest results to be more accurate compared to real-time. Tick Replay would allow for historical data to process tick-by-tick as it would real-time and submitting orders to an added secondary 1-tick series will simply provide more accurate fill prices.

Testing may be done with the Playback Connection to fully simulate standard Kagi bars with accurate reversals and order fills.