Problem understanding backtesting tool

[meganoxus]

Hello! So as I have read, backtesting tool from Strategy analyzer works calculating "on bar close", so, what changes when we activate "tick replay" in the backtesting tool?

Other thing: I have read a lot of cases where people obtain good results from backtesting and then in real time, the results were bad because their strategy had some conditions that became true or changed while a candle is forming. My thought is: If I have good results in backtesting ("in bar close" calculated), if I change my strategy way of calculate from "on price change" to "on bar close", will the real time results be closer to the backtesting results?

[NinjaTrader_Gaby]

Hello meganoxus,

Thank you for your post.

TickReplay is used for the intra-bar actions. This adds a 1 tick series to the script to be used for triggering OnBarUpdate when Calculate is set to .OnEachTick or .OnPriceChange. This also allows OnMarketData to trigger historically for last market events only (the bid and ask will not trigger).

-TickReplay will allow for indicators to update for each tick or price change and for scripts to trigger actions intra-bar.

-TickReplay must be enabled for the Calculate setting in State.SetDefaults to take effect in the Strategy Analyzer.

-TickReplay allows for IsFirstTickOfBar to work correctly in historical data and will be true for the first tick of the bar only.

-TickReplay does not provide fill prices for accurate order fills.


Tick Replay - https://ninjatrader.com/support/help...ick_replay.htm


Changing the Calculate setting will not affect whether the results are closer to realtime results. To get more accurate backtest results, you can use Tick Replay with an added 1 tick data series.

Please see the forum post which discusses the differences in realtime vs backtest, and how to improve the accuracy of a backtest:

https://forum.ninjatrader.com/forum/...377#post773377

If you have any other questions, please let me know.

[meganoxus]

Hello meganoxus,

Thank you for your post.

TickReplay is used for the intra-bar actions. This adds a 1 tick series to the script to be used for triggering OnBarUpdate when Calculate is set to .OnEachTick or .OnPriceChange. This also allows OnMarketData to trigger historically for last market events only (the bid and ask will not trigger).

-TickReplay will allow for indicators to update for each tick or price change and for scripts to trigger actions intra-bar.

-TickReplay must be enabled for the Calculate setting in State.SetDefaults to take effect in the Strategy Analyzer.

-TickReplay allows for IsFirstTickOfBar to work correctly in historical data and will be true for the first tick of the bar only.

-TickReplay does not provide fill prices for accurate order fills.


Tick Replay - https://ninjatrader.com/support/help...ick_replay.htm


Changing the Calculate setting will not affect whether the results are closer to realtime results. To get more accurate backtest results, you can use Tick Replay with an added 1 tick data series.

Please see the forum post which discusses the differences in realtime vs backtest, and how to improve the accuracy of a backtest:

https://forum.ninjatrader.com/forum/...377#post773377

If you have any other questions, please let me know.

Hello Gaby, thanks for your time and your repply.

About this statement: "This adds a 1 tick series to the script". There must be something I am missing. Because... if I use tick replay in the backtesting, for example, I obtain a good positive profit, but if I add a data series of 1 tick for the triggers and use the backtesting in 1 tick chart, that result becomes negative. Only changed that I added the 1 tick data series. So if Tick replay adds a 1 tick series to the script, and when I add a 1 tick series the results have that considerable change, there must be something I am missing here.

For better understanding I am going to explain the "on price change" parameters of my strategy: Entries are when price breaks the high or low of the previous candle (depending on buy or sell direction), then I just add a trailing stop. Those are the only 2 parameters which has to use the "on price change" aspect. So I don't understand why is there so much change in the results, I need robust data to trust.

So, in summary. If I put the strategy in backtest with tick replay, will these paremeters be backtested "on price change" or "on bar close"? I have been trying the strategy with real time data this week in a simmulation account, and results have been negative, but If I put the strategy on the strategy analyzer and backtest it, it says It should have got profits, with a "big" difference from the real results. That is the reason I think that the backtesting with tick replay is missing something.

Thanks in advance and sorry if my english is not very good (not my mother tongue).

[NinjaTrader_Gaby]

Hello,

If calculate is not OnBarClose, then you will need both Tick Replay and an added 1 tick series for intra-bar fills.

TickReplay is used for the intra-bar actions. Intra-bar granularity adds a second data series such as a 1 tick series using AddDataSeries() so that the strategy or indicator has the individual ticks in the historical data in between the High and Low of the primary series. This allows for more accurate trades by supplying the correct price at the correct time for the order to fill with. The orders placed will need to use the BarsInProgress (BarsArray) index of that finer resolution series.


The reason the difference is so great in backtest is because when in historical data, only the Open, High, Low, and Close will be available and there will be no intra-bar data. This means actions cannot happen intra-bar, fills cannot happen intra-bar. All prices and actions come from and occur when the bar closes as this is all the information that is known.


Because of the lack of intrabar data, OnBarUpdate will only update 'On bar close' as it does not have the intra-bar information necessary for 'On price change' or 'On each tick' and the script will not have the intra-bar information to accurately fill an order at the exact price and time.

Adding tick replay allows actions to happen intra bar. Adding a 1-tick data series allows fills to happen intra bar.

I also wanted to amend something from my last post: Changing the Calculate setting can have some affect on results. If you use market orders with Calculate.Onbarclose, the results should be pretty close from backtest to real-time without any intra-bar granularity.

Please let me know if you have any other questions.

[meganoxus]

Hello,

If calculate is not OnBarClose, then you will need both Tick Replay and an added 1 tick series for intra-bar fills.

TickReplay is used for the intra-bar actions. Intra-bar granularity adds a second data series such as a 1 tick series using AddDataSeries() so that the strategy or indicator has the individual ticks in the historical data in between the High and Low of the primary series. This allows for more accurate trades by supplying the correct price at the correct time for the order to fill with. The orders placed will need to use the BarsInProgress (BarsArray) index of that finer resolution series.


The reason the difference is so great in backtest is because when in historical data, only the Open, High, Low, and Close will be available and there will be no intra-bar data. This means actions cannot happen intra-bar, fills cannot happen intra-bar. All prices and actions come from and occur when the bar closes as this is all the information that is known.


Because of the lack of intrabar data, OnBarUpdate will only update 'On bar close' as it does not have the intra-bar information necessary for 'On price change' or 'On each tick' and the script will not have the intra-bar information to accurately fill an order at the exact price and time.

Adding tick replay allows actions to happen intra bar. Adding a 1-tick data series allows fills to happen intra bar.

I also wanted to amend something from my last post: Changing the Calculate setting can have some affect on results. If you use market orders with Calculate.Onbarclose, the results should be pretty close from backtest to real-time without any intra-bar granularity.

Please let me know if you have any other questions.

Hello Gaby, I am finding this conversation interesting.

So, in summary: for an "On bar close" calculated strategy, backtesting with 'tick replay' enabled will be accurate. And for an "On price change" and "On each tick" calculated strategy, backtesting will be accurate if I use 'tick replay' and also add in the code of the strategy a 1-tick data series of the primary chart (I am only using one symbol which is <primary> Range 5 in strategy builder so I just have to add <Primary> 1 tick). Right? In this second case, backtesting has to be done in a 1 tick chart, or can I make it in a, for example, Range 5 chart?

I think it could be interesting to add these statements in the help guide. Because I couldn't find that and started to distrust backtesting results.

Thanks for your time.

[NinjaTrader_Gaby]

Hello,

You could backtest using the 5 range chart, but as mentioned, submitting orders to a 1 tick chart will allow you to have intrabar fills. Using Tick Replay and intrabar granularity are optional tools to increase the accuracy of your backtest.

Please let us know if you have any other questions.