Announcement

Collapse
No announcement yet.

Partner 728x90

Collapse

Market Replay vs Back Test results

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    #76
    Hello Malletto,

    Thanks for your notes.

    The Calculate mode is noted in the OnStateChange State.SetDefaults section of the indicator script.

    You would also need to review the OnBarUpdate() logic in the script to see if it requires Calculate.OnPriceChange or Calculate.OnEachTick processing. For example, if the script uses IsFirstTickOfBar in OnBarUpdate() for calculations, then the script would need to use OnPriceChange or OnEachTick.

    If the indicator is not an assembly (dll) then you could open a New > NinjaScript Editor window, open the Indicators folder, and double-click on the indicator file to view it's code.

    Or, you could contact the developer directly to confirm if the script must use Calculate.OnPriceChange or Calculate.OnEachTick processing.

    Adding debugging prints to your strategy could be done to confirm how your logic is behaving when running realtime compared to a backtest.

    Below is a link to a forum post that demonstrates how to use prints to understand behavior.
    https://ninjatrader.com/support/foru...121#post791121
    Brandon H.NinjaTrader Customer Service

    Comment


      #77
      Ok. So how is the trend magic calculating on the backtest. I want to clarify. Thank you so much!

      Comment


        #78
        Hello Malletto,

        Thanks for your notes.

        All scripts will be processed with Calculate.OnBarClose when backtesting since this is the only Calculate mode the Strategy Analyzer can use.

        Calculate: https://ninjatrader.com/support/help.../calculate.htm
        Brandon H.NinjaTrader Customer Service

        Comment


          #79
          Ok thx. Is there anything else that could make the indicator behave differently when comparing the backtest and real time?

          Comment


            #80
            Hello Maletto,

            Thanks for your notes.

            No, this would be dependent on the indicator's logic and settings.

            If the indicator uses Calculate.OnPriceChange or Calculate.OnEachTick but is forced to use OnBarClose processing when backtesting in the Strategy Analyzer, this would be a difference in settings which would cause different results.

            You may try enabling Tick Replay when running a backtest. Tick Replay would be used to have the logic process OnEachTick or OnPriceChange with historical data.

            Review this help guide page for information about Tick Replay: TickReplay — https://ninjatrader.com/support/help...ick_replay.htm
            Last edited by NinjaTrader_BrandonH; 12-04-2023, 10:04 AM.
            Brandon H.NinjaTrader Customer Service

            Comment


              #81
              But I’m watching it now. What makes the indicator act different. If you’re aware of the trend magic, it basically turns red or green. There’s times when the backtest turns a color when the realtime doesn’t. Why is that. Something is different

              Comment


                #82
                Hello Malletto,

                Thanks for your notes.

                Do you have Tick Replay enabled when running the backtest?

                This would be caused by how the script's logic calculates between backtests compared to realtime.

                One example would be if the script requires processing OnEachTick or OnPriceChange for certain conditions, such as using IsFirstTickOfBar in a condition, but the Strategy Analyzer is using OnBarClose, there is a difference in how logic is calculated which will cause different results when comparing backtests to real-time.

                Using Tick Replay when running a backtest would allow the indicator to calculate logic tick-by-tick historically as it would when running real-time. Tick Replay would be used to have the logic process OnEachTick or OnPriceChange with historical data

                Please review the Tick Replay help guide page linked in post # 80.
                Brandon H.NinjaTrader Customer Service

                Comment


                  #83
                  No i haven’t. Thank you for this. I’m just trying to reason what the difference is. Because there definitely is a difference in the indicator’s behavior. So I’m wondering what it is. When you mean on bar close or on price change. Is that something within the script, or something you edit in the data series when loading the strategy?

                  Comment


                    #84
                    Hello Malletto,

                    Thanks for your notes.

                    The Calculate mode is set within the code of a NinjaScript.

                    The difference is due to how the script's logic is processed between backtest and real-time.

                    Enable Tick Replay when backtesting so the indicator calculates tick-by-tick on historical data as it would when running real-time.
                    Brandon H.NinjaTrader Customer Service

                    Comment


                      #85
                      But this calculation is different from when you just turn it on through data series, correct?

                      Comment


                        #86
                        Hello Malletto,

                        Thanks for your notes.

                        Are you referring to setting the Calculate mode in the UI of the Indicators window?

                        Your strategy will use whichever Calculate mode the indicator is set to use in the code of the script itself. You would see what Calculate mode an indicator uses by noting the 'Calculate' property when adding the indicator to a chart.

                        I have imported the Trend Magic indicator from the Ecosystem User App Share linked below and created a simple test strategy using this indicator. The way the test strategy works is when the Close price crosses above the Trend Magic plot, a blue dot is drawn 2 ticks above the High of the bar.

                        Trend Magic: https://ninjatraderecosystem.com/use...d/trend-magic/

                        I have tested this sample strategy by running it on a chart while connected to a live data feed connection. Then, I ran a backtest on the strategy using the exact same settings as the real-time strategy and data series.

                        When testing this script, I see that the bars on the Strategy Analyzer Chart display match the bars on the live chart. The indicator plots on the Chart display of the Strategy Analyzer matches the live chart's historical data. The dots drawn by the strategy when the condition becomes true also match.

                        See this demonstration video: https://brandonh-ninjatrader.tinytak...Ml8yMjQ5NDIyOQ

                        Attached is the simple sample script used to test this. You could try testing the attached script using the same settings and steps seen in the demonstration video.​

                        If you are seeing different behavior occur when running your custom script then it would be up to you to debug your script's logic to see exactly how your custom logic is evaluating between realtime and backtests to see where differences might be.

                        If your script was programmed by a third-party developer, reach out to that developer directly so they can debug the script they created.
                        Attached Files
                        Last edited by NinjaTrader_BrandonH; 12-04-2023, 10:50 AM.
                        Brandon H.NinjaTrader Customer Service

                        Comment


                          #87
                          I appreciate this a lot. Are you comparing with real-time, or is that just loaded on the chart. Because i notice when it’s loaded on the chart it’s identical to the backtest. But when it’s real-time, that’s when you see the difference.

                          Comment


                            #88
                            Could you also run the results on the strategy analyzer and let me know what you get

                            Comment


                              #89
                              Hello Malletto,

                              Thanks for your notes.

                              As previously stated, I added the strategy to a live chart window while connected to realtime data. Then, I ran a backtest on the strategy in the Strategy Analyzer using the same strategy settings and data series settings as used on the live chart.

                              Then I compared the results and see the bars, indicator plots, and dots being drawn by the strategy match. Showing the strategy is working the same historically as is it real-time.

                              See the demonstration video which shows the strategy run on the Strategy Analyzer.

                              Note that a backtest can only test on past historical data, not incoming live data. You could take the same steps seen in my demonstration video to test the sample script attached in post # 86 and to test your script.

                              You must add debugging prints to your script to understand exactly how your logic is behaving historically compared to realtime.

                              This forum thread talks about comparing historical to realtime to playback results and adding prints to a script to find where differences might be, if any. Review this forum thread and sample script, and add debugging prints to your strategy to understand how logic is evaluating.

                              https://forum.ninjatrader.com/forum/...nce#post100192

                              Below is a link to a forum post that demonstrates how to use prints to understand behavior.
                              https://ninjatrader.com/support/foru...121#post791121
                              Last edited by NinjaTrader_BrandonH; 12-04-2023, 10:58 AM.
                              Brandon H.NinjaTrader Customer Service

                              Comment


                                #90
                                Yeah i watched the video, for the sake of this experiment, can you let it run live until another change happens. What your showing is only historic movement

                                Comment

                                Latest Posts

                                Collapse

                                Topics Statistics Last Post
                                Started by Shansen, 08-30-2019, 10:18 PM
                                17 responses
                                756 views
                                0 likes
                                Last Post SpartanGR77  
                                Started by Kraken29, Today, 12:14 AM
                                4 responses
                                22 views
                                0 likes
                                Last Post Kraken29  
                                Started by Creamers, 02-25-2024, 01:42 PM
                                4 responses
                                33 views
                                0 likes
                                Last Post Creamers  
                                Started by plenamar27, 02-21-2022, 08:40 AM
                                11 responses
                                592 views
                                0 likes
                                Last Post Net_Trader  
                                Started by lloydteo, Today, 09:57 AM
                                0 responses
                                3 views
                                0 likes
                                Last Post lloydteo  
                                Working...
                                X