Market Replay vs Back Test results

[NinjaTrader_BrandonH]

Hello Malletto,

Thanks for your notes.

The Calculate mode is noted in the OnStateChange State.SetDefaults section of the indicator script.

You would also need to review the OnBarUpdate() logic in the script to see if it requires Calculate.OnPriceChange or Calculate.OnEachTick processing. For example, if the script uses IsFirstTickOfBar in OnBarUpdate() for calculations, then the script would need to use OnPriceChange or OnEachTick.

If the indicator is not an assembly (dll) then you could open a New > NinjaScript Editor window, open the Indicators folder, and double-click on the indicator file to view it's code.

Or, you could contact the developer directly to confirm if the script must use Calculate.OnPriceChange or Calculate.OnEachTick processing.

Adding debugging prints to your strategy could be done to confirm how your logic is behaving when running realtime compared to a backtest.

Below is a link to a forum post that demonstrates how to use prints to understand behavior.
https://ninjatrader.com/support/foru...121#post791121

[Malletto]

Ok. So how is the trend magic calculating on the backtest. I want to clarify. Thank you so much!

[NinjaTrader_BrandonH]

Hello Malletto,

Thanks for your notes.

All scripts will be processed with Calculate.OnBarClose when backtesting since this is the only Calculate mode the Strategy Analyzer can use.

Calculate: https://ninjatrader.com/support/help.../calculate.htm

[Malletto]

Ok thx. Is there anything else that could make the indicator behave differently when comparing the backtest and real time?

[NinjaTrader_BrandonH]

Hello Maletto,

Thanks for your notes.

No, this would be dependent on the indicator's logic and settings.

If the indicator uses Calculate.OnPriceChange or Calculate.OnEachTick but is forced to use OnBarClose processing when backtesting in the Strategy Analyzer, this would be a difference in settings which would cause different results.

You may try enabling Tick Replay when running a backtest. Tick Replay would be used to have the logic process OnEachTick or OnPriceChange with historical data.

Review this help guide page for information about Tick Replay: TickReplay — https://ninjatrader.com/support/help...ick_replay.htm

[Malletto]

But I’m watching it now. What makes the indicator act different. If you’re aware of the trend magic, it basically turns red or green. There’s times when the backtest turns a color when the realtime doesn’t. Why is that. Something is different

[NinjaTrader_BrandonH]

Hello Malletto,

Thanks for your notes.

Do you have Tick Replay enabled when running the backtest?

This would be caused by how the script's logic calculates between backtests compared to realtime.

One example would be if the script requires processing OnEachTick or OnPriceChange for certain conditions, such as using IsFirstTickOfBar in a condition, but the Strategy Analyzer is using OnBarClose, there is a difference in how logic is calculated which will cause different results when comparing backtests to real-time.

Using Tick Replay when running a backtest would allow the indicator to calculate logic tick-by-tick historically as it would when running real-time. Tick Replay would be used to have the logic process OnEachTick or OnPriceChange with historical data

Please review the Tick Replay help guide page linked in post # 80.

[Malletto]

No i haven’t. Thank you for this. I’m just trying to reason what the difference is. Because there definitely is a difference in the indicator’s behavior. So I’m wondering what it is. When you mean on bar close or on price change. Is that something within the script, or something you edit in the data series when loading the strategy?

[NinjaTrader_BrandonH]

Hello Malletto,

Thanks for your notes.

The Calculate mode is set within the code of a NinjaScript.

The difference is due to how the script's logic is processed between backtest and real-time.

Enable Tick Replay when backtesting so the indicator calculates tick-by-tick on historical data as it would when running real-time.

[Malletto]

But this calculation is different from when you just turn it on through data series, correct?

[NinjaTrader_BrandonH]

Hello Malletto,

Thanks for your notes.

Are you referring to setting the Calculate mode in the UI of the Indicators window?

Your strategy will use whichever Calculate mode the indicator is set to use in the code of the script itself. You would see what Calculate mode an indicator uses by noting the 'Calculate' property when adding the indicator to a chart.

I have imported the Trend Magic indicator from the Ecosystem User App Share linked below and created a simple test strategy using this indicator. The way the test strategy works is when the Close price crosses above the Trend Magic plot, a blue dot is drawn 2 ticks above the High of the bar.

Trend Magic: https://ninjatraderecosystem.com/use...d/trend-magic/

I have tested this sample strategy by running it on a chart while connected to a live data feed connection. Then, I ran a backtest on the strategy using the exact same settings as the real-time strategy and data series.

When testing this script, I see that the bars on the Strategy Analyzer Chart display match the bars on the live chart. The indicator plots on the Chart display of the Strategy Analyzer matches the live chart's historical data. The dots drawn by the strategy when the condition becomes true also match.

See this demonstration video: https://brandonh-ninjatrader.tinytak...Ml8yMjQ5NDIyOQ

Attached is the simple sample script used to test this. You could try testing the attached script using the same settings and steps seen in the demonstration video.

If you are seeing different behavior occur when running your custom script then it would be up to you to debug your script's logic to see exactly how your custom logic is evaluating between realtime and backtests to see where differences might be.

If your script was programmed by a third-party developer, reach out to that developer directly so they can debug the script they created.

[Malletto]

I appreciate this a lot. Are you comparing with real-time, or is that just loaded on the chart. Because i notice when it’s loaded on the chart it’s identical to the backtest. But when it’s real-time, that’s when you see the difference.

[Malletto]

Could you also run the results on the strategy analyzer and let me know what you get

[NinjaTrader_BrandonH]

Hello Malletto,

Thanks for your notes.

As previously stated, I added the strategy to a live chart window while connected to realtime data. Then, I ran a backtest on the strategy in the Strategy Analyzer using the same strategy settings and data series settings as used on the live chart.

Then I compared the results and see the bars, indicator plots, and dots being drawn by the strategy match. Showing the strategy is working the same historically as is it real-time.

See the demonstration video which shows the strategy run on the Strategy Analyzer.

Note that a backtest can only test on past historical data, not incoming live data. You could take the same steps seen in my demonstration video to test the sample script attached in post # 86 and to test your script.

You must add debugging prints to your script to understand exactly how your logic is behaving historically compared to realtime.

This forum thread talks about comparing historical to realtime to playback results and adding prints to a script to find where differences might be, if any. Review this forum thread and sample script, and add debugging prints to your strategy to understand how logic is evaluating.

https://forum.ninjatrader.com/forum/...nce#post100192

Below is a link to a forum post that demonstrates how to use prints to understand behavior.
https://ninjatrader.com/support/foru...121#post791121

[Malletto]

Yeah i watched the video, for the sake of this experiment, can you let it run live until another change happens. What your showing is only historic movement