Accurate Backtesting & Tick Replay

[dallen133]

I developed my own strategy, and I want to be able to trust the backtesting results but with all the options it's just not clear how.

I am trading the ES on a tick chart.

Backtesting options:

1. Strategy Analyzer:

If I run the strategy analyzer with Tick Replay should I assume that the backtesting is looking at real data that is moving tick by tick (so I can assume my entries and exits are correct), or is it still just OHLC (and therefore not analogous to real-time data)?

2. Historical Data:

If I download one month of tick data, then open a chart with tick replay on for the same dates I just downloaded, then open my strategy, then look at Strategy Performance, should I assume the entries/exits/results are the same as would occur in real-time data?


The real question here is, how in the heck do I get real results when backtesting?! Is Tick Replay showing me real results that I can trust?

[NinjaTrader_Emily]

Hello dallen123,

Thank you for your post and welcome to the NinjaTrader forums!

I would like to start by clarifying that Tick Replay was not designed for order fill accuracy in backtests; it is intended to be used for market data events where OnMarketData is called after OnBarUpdate so your script can get granular historical information about volume, bid, last, and ask prices. For the Strategy Analyzer, only the standard order fill resolution may be used with Tick Replay enabled. This uses OHLC values for order fills and does not offer tick-by-tick data. When you open a chart and then load a strategy and look at Strategy Performance, this is essentially the same as running a backtest in the Strategy Analyzer to get historical results. That is why the strategy parameters have a "Historical fill processing" section with a dropdown menu for Order Fill Resolution, and again you may only use the standard resolution with Tick Replay enabled.

If you would like more granular data for more accurate results, you could turn off Tick Replay and utilize the High order fill resolution setting to add a more granular data series to base order fills on. The most granular is a 1-tick series though that is more resource-intensive and could result in longer backtests. Another option is to develop your script so that orders are submitted to a 1-tick data series as shown in the following reference sample:

https://ninjatrader.com/support/helpGuides/nt8/backtesting_ninjascript_strate.htm

For more details on Order Fill Resolution:

https://ninjatrader.com/support/helpGuides/nt8/index.html?understanding_historical_fill_.htm#UnderstandingOrderFillResolution

We have a page about discrepancies between real-time and backtest here:

https://ninjatrader.com/support/helpGuides/nt8/discrepancies_real-time_vs_bac.htm

As mentioned in the bottom section of the previous link, if you are backtesting with a bar type such as Renko or Heiken Ashi, it may be challenging to get realistic results in the Strategy Analyzer. Instead, you could try testing your strategy on the Playback connection or use a different custom bar type or an indicator rather than the bar type.

My colleague has written a very detailed post about this subject here:

https://forum.ninjatrader.com/forum/ninjatrader-8/strategy-development/94098-isfirsttickofbar-vs-onbarclose-for-backtest-live#post773377

Please let us know if we may be of further assistance.

[QuantKey_Bruce]

My colleague has written a very detailed post about this subject here:
https://forum.ninjatrader.com/forum/...ive#post773377

dallen133 You should hop over and read every word of what NinjaTrader_ChelseaB has written there as it is very thorough in explaining the nuance of this issue.

But, if you're going to be inspecting fills with that level of detail, you really should add a 1-tick series and place your orders on that using the order overloads that allow you to specify which bars array you're trading on, and only trade on the 1-tick one.

In that way, you won't need to do a tick replay (unless you need it for some other reason) or use High order fill resolution because the 1-tick bars you are putting your orders on already have all of the available information about the intrabar sequence of events.

[dallen133]

Thank you both for your answers, I found some good info! Two questions I have that I didn't find answers to...

1. with tick replay and calculate oneachtick, will I get accurate order fills for limit orders?

2. for setstoploss and setprofittarget, is there a workaround for getting those to submit to the 1-tick data series if I set up the code to use tick replay AND high resolution as described in this video:

[NinjaTrader_Emily]

Hello dallen133,

Thank you for your reply. I have addressed your questions below:

1. Tick Replay is not used for order fill accuracy. If Calculate is set to On Each Tick, that will only apply to how often OnBarUpdate() is called in real-time and does not affect historical processing or order fills. Tick Replay would be used to obtain intrabar data during historical processing.
   • Here is the help guide page about the Calculate property:
     • https://ninjatrader.com/support/help.../calculate.htm
2. When using the Set() method, you should only submit orders to the first bars for the instrument. This means that using SetStopLoss() and SetProfitTarget() for an added 1-tick data series really isn't suggested. It would likely be more beneficial to manually create and monitor your positions with Exit orders for protection instead of Set() method orders for your stops and targets.
   • This is mentioned in the notes on the following three pages:
     • https://ninjatrader.com/support/help...ionInformation
     • https://ninjatrader.com/support/help...etstoploss.htm
     • https://ninjatrader.com/support/help...ofittarget.htm
   • The following reference sample demonstrates how you could use OnOrderUpdate() and OnExecutionUpdate() to submit protective orders:
     • https://ninjatrader.com/support/help...and_onexec.htm

Please let us know if we may be of further assistance.

[imonlysleeping]

so if we set up the strategy to have the 1-tick series as BarsInProgress[1], but we have some logic already in onBarUpdate() in order to check for incoming signals, which is only meant for the primary series, how do we determine which logic needs to be run through the 1-tick series and which doesn't? i am trying for the best efficiency and speed possible here.

currently, if an incoming signal is found, then we immediately calculate the entry, stop, and target price within onBarUpdate() on the same iteration.

the entry orders are conditional after the signal is found, and are also within onBarUpdate(). it sounds like now the entry orders need to be submitted against the 1-tick series in order to get the accurate fill prices (even though they are limit orders, so i still don't understand how that wouldn't be accurate on the primary without the 1-tick series. a limit order is a limit order and is accurate by default right?).

anyway, how do i handle that? do i have to let onBarUpdate() run on both series completely in order for everything to be accurate?

[QuantKey_Bruce]

None of your logic should run on the 1-tick series unless you need it to. Most likely, if your analysis is all on the primary chart series, and if you're entering with limit orders, the only thing you'll want to do in OnBarUpdate when BarsInProgress == 1 would be to repeat your limit orders that you placed earlier during the same bar until they get filled, intrabar relative to the primary chart series - assuming you're using the managed order approach.

[imonlysleeping]

well right now i am only running onbarupdate() on the primary series. the entry limit orders are placed conditionally within the primary logic but they use the overload which contains barsinprogress and i have set them as liveuntilcancelled and placed them against the 1-tick series. so would i still need to do the BarsInProgress == 1 as u described above? seems redundant, unless i’m missing something?

[QuantKey_Bruce]

If you set them liveuntilcanceled you do not need to repeat them - you just need to cancel them when you want them canceled. Repeating them is what you would do if they are not liveuntilcanceled.

[imonlysleeping]

the strategy i had originally programmed shows expected results when i open up a tick replay enabled chart on sim account and load up the strategy. it also shows expected results when using market replay with tick replay on. this original version of the strategy does not use a secondary 1-tick series for the entries and exits. rather it uses conditional limit order entries and utilizes setstoploss() and setprofittarget() for the OCO position protection. all logic and orders happen within onbarupdate().

i understand that market replay with tick replay on will provide accuracy for the intrabar actions (price movement intrabar), but that the order fills will happen "immediately", and therefore market replay does not accurately reflect realtime market conditions in regards to order fills due to this timing discrepancy. i assume it is the same for any historical trades the strategy is showing on a chart with tick replay on. please correct me if i am wrong.

but other than this latency that will be expected under real live market conditions, if i am seeing the expected results for the strategy in the aforementioned cases, should i expect for the strategy to function exactly the same (aside from the internet latency for order fills) on live market data, whether on sim or live account?

[QuantKey_Bruce]

There are other questions besides latency - for instance, in the "real world" you're not always at the top of the queue, so just because a trade happened at your limit price doesn't mean your particular limit order will be executed there. Also, if it's not regular trading hours or you're trading anything that's thinly traded (even a futures contract in the small hours of the morning), your order affects the other participants, because they see your limit order and may change their orders because they see yours. That's not an issue during RTH for something like ES because there's just too much going on. The latency, combined with the question of your place in the queue (especially at turning points or trying to enter during a momentum move), account for a lot of the difference that happens between backtesting and forward testing on a real account. You can estimate these factors by allowing for slippage on exits like stop loss orders, and by understanding that limit orders won't get filled on the entry side every time so you won't get all of those trades (and sometimes, you only get the ones you don't want).

[imonlysleeping]

Thanks very much for your replies.

Sure, that is all understandable. And there will always be limits to backtesting. But are you saying there is a way to program it that would account for these real life unknowns, or does it sound like the results i’ve described are as close as i can get to real life accuracy using ninjatrader to backtest?

[QuantKey_Bruce]

No, it is not possible (within reason, and within NinjaTrader's fill models) to simulate things like your place in the queue and limitations of liquidity within the latency in question. It is possible to program a simulation of this using perhaps level 2 data and some information on your average latency, but that fill model would be extremely slow running and NinjaTrader does not do that. Simulating your effect on the market in situations of low liquidity? No, not really.

[imonlysleeping]

Thanks for your input. Still looking for a clear answer to the question i’m asking.

In regards to my post #10 on this thread, could you give your thoughts specifically in regards to the way i have coded the strategy in combination with the settings and testing options i’m using, for my specific purposes and if this seems an accurate / efficient way of getting results that i can trust?

I have had a lot of confusion as to which is the best programming / testing / settings combo, due to what seem like blatant contradictions in the reference manual as well as what feels like too many available options and combinations of options that are all apparently unsatisfactory for getting trustworthy results. (still trying to figure out why any methods which can produce inaccurate results would even be available at all? is that useful to anyone?).

i’m hoping to confirm that i have chosen a combination of programming and settings that will provide me with results that are trustworthy (aside from the general latency and liquidity issues regarding order fills we already discussed, which would be common to all automated trading strategies, and are therefore redundant in this context) for my specific needs, which i have laid out generally in post #10.

happy to elaborate more if more details are needed in order to provide a satisfactory answer.

if i have not chosen the right combination, please advise on what that is.

very much appreciate all efforts and information you’ve all provided thus far. thank you!!

[QuantKey_Bruce]

All such settings are a compromise in some way. There is no "perfect" backtesting. You have to do your best, based on your understanding of the documentation, advice you have gathered, and the specifics of your strategy, to choose the best set of compromises for your scenario, and then test carefully on a "going forward" basis - first in simulation and then with small size in live markets. It would be inappropriate for anyone else to come along and tell you that you have done it all the best way possible - they can't know this. All they can give you is general guidance based on experience.