Market Replay vs Back Test results

[NinjaTrader_BrandonH]

Hello Malletto,

Thanks for your notes.

The Strategy Analyzer can only test historical data, so this is what you would be comparing the backtest to on the live chart.

I let the strategy run for a little while on the live chart window. Then, I ran a backtest on the strategy on the Strategy Analyzer and see the dots drawn on the chart were the same as historically as they were on the live chart.

Further, after running another backtest the Trend Magic plot of the Live chart at 11:54 in the Mini Data Box window matches the Trend Magic plot of the Chart display of the Strategy Analyzer at 11:54. The bar values also match.

Demo video: https://brandonh-ninjatrader.tinytak...NV8yMjQ5NDI4Nw

Ultimately, if you see your logic is calculating differently then it would be up to you to debug your strategy's logic to see what the differences are in your custom logic of the strategy. This is detailed in the forum thread below which also contains a reference sample.

https://forum.ninjatrader.com/forum/...nce#post100192

Below is a link to a forum post that demonstrates how to use prints to understand behavior.
https://ninjatrader.com/support/foru...121#post791121

[Malletto]

Thank you for all of this. Did you see my email? I sent you the difference

[NinjaTrader_BrandonH]

Hello Malletto,

Thanks for your notes.

I have not received an email from you that I can see.

If you are seeing differences then you must debug your script by adding prints to determine what exactly is evaluating differently. This would be up to you to do.

Note we do not offer debugging services in our support.

Read through the forum thread about comparing historcal, realtime, and playback results, add prints to the script that prints out ALL of the logic being used for calculations in the logic of the script, and test your strategy to see where differences might be. The forum thread below details how to accomplish this is contain a sample script you could view.

https://forum.ninjatrader.com/forum/...nce#post100192

Review this forum thread below demonstrating how to use prints to understand the behavior of a script's logic.
https://ninjatrader.com/support/foru...121#post791121

[Malletto]

How can i send it to you then. I want to show you what im seeing

[NinjaTrader_BrandonH]

Hello Malletto,

Thanks for your notes.

You could share the image or demonstration video on this forum thread.

To clarify, have you tested the strategy script I shared in post # 86 using the same steps seen in the demonstration video I shared?

Do you see a difference when testing that test script I shared using the steps I provided?

If the test strategy I shared is behaving the same way seen in the demonstration video I shared, this means it is likely the logic of your script causing discrepancies. Without debugging the logic of your script to find the line(s) of code causing the differences, there is nothing specific I could advise on.

It would be up to you to debug the logic of your script to understand exactly how it is evaluating your custom logic. It is against our policy to debug a script for you on your behalf in the Support department at NinjaTrader.

If you need someone debug the logic of your strategy on your behalf, you can also contact a professional NinjaScript Consultant who would be eager to assist you with your script.

You can search our extensive library of NinjaScript consultants through the link below. Simply enter a consultant name or search by using our filter categories. Once you have identified your consultants of choice, please visit each consultant's site for more information or contact them directly to learn more!

https://ninjatraderecosystem.com/sea...mming-services

Educators - https://ninjatraderecosystem.com/sea...ures=education

You can locate the contact information for the consultants on their direct websites for any additional questions you may have. Since these consultants are third-party services for NinjaTrader, all pricing and support information will need to be obtained through the consultant.

This NinjaTrader Ecosystem website is for educational and informational purposes only and should not be considered a solicitation to buy or sell a futures contract or make any other type of investment decision. The companies and services listed on this website are not to be considered a recommendation and it is the reader's responsibility to evaluate any product, service, or company. NinjaTrader Ecosystem LLC is not responsible for the accuracy or content of any product, service or company linked to on this website.

[Malletto]

It won’t work. Can we email so i can send it

[NinjaTrader_BrandonH]

Hello Maletto,

Thanks for your notes.

You could reply to the email ticket that we had opened to send the image to me.

Further, please test the strategy I shared using the same steps and settings seen in the demonstration video and confirm if the strategy is behaving the same way seen in the demonstration video.

[Malletto]

I just sent an email. Let me know if you received

[Malletto]

Ok. Is this an issue for the regular/normal time frame? Or just a Kagi issue

[NinjaTrader_BrandonH]

Hello Malletto,

Thanks for your notes.

As stated in the email thread we had opened, the reported behavior does not occur for time-based bars such as a 1-minute bar.

The Development team is still looking into the reported behavior regarding using Kagi bars and we do not have an ETA as to when they will be finished investigating this.

If you are seeing discrepancies when comparing backtest results to real-time results using minute bars, read through the forum thread linked below detailing comparing historcal, realtime, and playback results, add prints to the script that prints out ALL of the logic being used for calculations in the logic of the script, and test your strategy to see where differences might be. The forum thread below details how to accomplish this is contain a sample script you could view.

https://forum.ninjatrader.com/forum/...nce#post100192

Review this forum thread below demonstrating how to use prints to understand the behavior of a script's logic.
https://ninjatrader.com/support/foru...121#post791121

[Malletto]

Ok. So the normal minute timeframe should have no discrepancies? Like the issue with Kagi bars?

[NinjaTrader_BrandonH]

Hello Malletto,

Thanks for your notes.

Correct, the timestamps of the Minute bars on the Chart Display of the Strategy Analyzer match the timestamps of the Minute bars on the Chart window.

As discussed, the timestamps of the Kagi bars on the Chart Display of the Strategy Analyzer did not match the timestamps of the Kagi bars on a Chart window. The Development team is still looking into this and I will provide an update as soon as they have finished investigating.

That said, a certain level of discrepancy between realtime and backtest results would be expected for any bar types as noted in the help guide documentation linked below.

Please review the help guide document on the differences on real-time vs backtest (historical).
https://ninjatrader.com/support/help...bac.htm

To improve the accuracy of backtests, you would need to add intrabar granularity to the script. This is noted on the forum thread linked in post # 100.

See more about intra-bar granularity here.
https://ninjatrader.com/support/forum/forum/ninjatrader-8/strategy-development/94098-isfirsttickofbar-vs-onbarclose-for-backtest-live?p=773377#post773377

[Malletto]

Well how much of a discrepancy? Are my backtest results real on the 1 minute?

[Malletto]

Please let me know.

[NinjaTrader_BrandonH]

Hello Malletto,

Thanks for your notes.

We would not be able to say exactly how much of a discrepancy there would be. This would be dependent on multiple factors such as the logic of the script, the data you are testing on, the bar type being used, and more.

This is something you would need to debug and test on your own by following the notes mentioned in the forum thread linked on post # 100 which is also linked below.

https://forum.ninjatrader.com/forum/...nce#post100192

To improve the backtest intrabar granularity must be added to the script. With intrabar granularity coded into the script, the accuracy of the backtest would improve to within 1-tick.

See more about intra-bar granularity here.
https://ninjatrader.com/support/forum/forum/ninjatrader-8/strategy-development/94098-isfirsttickofbar-vs-onbarclose-for-backtest-live?p=773377#post773377