Market Replay vs Back Test results

[Ymcapital]

Hello Ymcapital,

Thanks for your notes.

The timestamps of Kagi bars in the Strategy Analyzer Chart display do not match the timestamps on a regular chart window.

This has been reported to the development team and they will be investigating this matter further. Once the development team is finished investigating this matter, I will update this forum thread with their findings.

That's good to know. I spent a disgusting amount of time really trying to dig through this. I thought it would be a quick project looking at this and finding what you were saying to be validated and the more I dug, the more I found the timestamps not matching up and the discrepancies piling up. I'd appreciate knowing when this is resolved. Thanks.

[NinjaTrader_BrandonH]

Hello Malletto,

Thanks for your notes.

We do not have an update regarding the behavior reported to the development team at this time.

Further, we do not have an ETA regarding when they will be done investigating this matter.

Once the development team has finished looking into this behavior I will update this forum thread with their findings.

[Malletto]

It’s been months brandon?

[NinjaTrader_BrandonH]

Hello Ymcapital,

Thanks for your notes.

The timestamps of Kagi bars in the Strategy Analyzer Chart display do not match the timestamps on a regular chart window.

This has been reported to the development team and they will be investigating this matter further. Once the development team is finished investigating this matter, I will update this forum thread with their findings.

[Ymcapital]

Hey Brandon, I wanted to follow up on here to see what the outcome was with Malletto. I spent several weeks doing a deep dive into this and it appears there's a GROSS mismatch of timestamps with NinjaTrader's Kagi chart. Based on all my data, it's not the strategies fault. No matter how much intra-bar granularity you have, it will not improve the results. I have a script that has tick series built in and what it shows is that the time stamping of Kagi bars on NinjaTrader's end is what makes the results impossible to replicate in a live environment.

Is that the same conclusion that was reached with Malletto? If so, is there anything being done on the backend side to improve the mismatch in timestamps? Thanks for your help.

[NinjaTrader_BrandonH]

Hello Malletto,

Thanks for your notes.

When testing the strategy on minute bars there could still be discrepancies when comparing backtest results to realtime results unless you add intrabar granularity to your strategy.

Add intrabar granularity to your strategy if you want the backtest results to be more accurate.

See more about intra-bar granularity here.
https://ninjatrader.com/support/forum/forum/ninjatrader-8/strategy-development/94098-isfirsttickofbar-vs-onbarclose-for-backtest-live?p=773377#post773377

If you cannot program the script to include intrabar granularity yourself, you could contact a professional NinjaScript consultant who would be happy to assist you.

You can search our extensive library of NinjaScript consultants through the link below. Simply enter a consultant name or search by using our filter categories. Once you have identified your consultants of choice, please visit each consultant's site for more information or contact them directly to learn more!

https://ninjatraderecosystem.com/sea...mming-services

Educators - https://ninjatraderecosystem.com/sea...ures=education

You can locate the contact information for the consultants on their direct websites for any additional questions you may have. Since these consultants are third-party services for NinjaTrader, all pricing and support information will need to be obtained through the consultant.

This NinjaTrader Ecosystem website is for educational and informational purposes only and should not be considered a solicitation to buy or sell a futures contract or make any other type of investment decision. The companies and services listed on this website are not to be considered a recommendation and it is the reader's responsibility to evaluate any product, service, or company. NinjaTrader Ecosystem LLC is not responsible for the accuracy or content of any product, service or company linked to on this website.

[Malletto]

Ok. But is the normal minute timeframe the most accurate out of all charts

[NinjaTrader_BrandonH]

Hello Malletto,

Thanks for your notes.

We would not be able to say exactly how much of a discrepancy there would be. This would be dependent on multiple factors such as the logic of the script, the data you are testing on, the bar type being used, and more.

This is something you would need to debug and test on your own by following the notes mentioned in the forum thread linked on post # 100 which is also linked below.

https://forum.ninjatrader.com/forum/...nce#post100192

To improve the backtest intrabar granularity must be added to the script. With intrabar granularity coded into the script, the accuracy of the backtest would improve to within 1-tick.

See more about intra-bar granularity here.
https://ninjatrader.com/support/forum/forum/ninjatrader-8/strategy-development/94098-isfirsttickofbar-vs-onbarclose-for-backtest-live?p=773377#post773377

[Malletto]

Please let me know.

[Malletto]

Well how much of a discrepancy? Are my backtest results real on the 1 minute?

[NinjaTrader_BrandonH]

Hello Malletto,

Thanks for your notes.

Correct, the timestamps of the Minute bars on the Chart Display of the Strategy Analyzer match the timestamps of the Minute bars on the Chart window.

As discussed, the timestamps of the Kagi bars on the Chart Display of the Strategy Analyzer did not match the timestamps of the Kagi bars on a Chart window. The Development team is still looking into this and I will provide an update as soon as they have finished investigating.

That said, a certain level of discrepancy between realtime and backtest results would be expected for any bar types as noted in the help guide documentation linked below.

Please review the help guide document on the differences on real-time vs backtest (historical).
https://ninjatrader.com/support/help...bac.htm

To improve the accuracy of backtests, you would need to add intrabar granularity to the script. This is noted on the forum thread linked in post # 100.

See more about intra-bar granularity here.
https://ninjatrader.com/support/forum/forum/ninjatrader-8/strategy-development/94098-isfirsttickofbar-vs-onbarclose-for-backtest-live?p=773377#post773377

[Malletto]

Ok. So the normal minute timeframe should have no discrepancies? Like the issue with Kagi bars?

[NinjaTrader_BrandonH]

Hello Malletto,

Thanks for your notes.

As stated in the email thread we had opened, the reported behavior does not occur for time-based bars such as a 1-minute bar.

The Development team is still looking into the reported behavior regarding using Kagi bars and we do not have an ETA as to when they will be finished investigating this.

If you are seeing discrepancies when comparing backtest results to real-time results using minute bars, read through the forum thread linked below detailing comparing historcal, realtime, and playback results, add prints to the script that prints out ALL of the logic being used for calculations in the logic of the script, and test your strategy to see where differences might be. The forum thread below details how to accomplish this is contain a sample script you could view.

https://forum.ninjatrader.com/forum/...nce#post100192

Review this forum thread below demonstrating how to use prints to understand the behavior of a script's logic.
https://ninjatrader.com/support/foru...121#post791121

[Malletto]

Ok. Is this an issue for the regular/normal time frame? Or just a Kagi issue

[Malletto]

I just sent an email. Let me know if you received