Differences between TrailStop in backtest vs. realtime in TickReplay + OnMarketData

[tony_28217]

Essentially in the backtest, it says I should be winning 90% of my trades while using the back test. In real-time... I lose about 80%...

How things are setup:
1. Strategy uses tick replay and Onmarket Data
2. Before implementation of trailstop, convergence was nearly 100%.
3. The strategy trades both long and short on 1 instrument (futures)
4. Tick is not my primary data series (but is included in the ninjascript) and runs perfectly otherwise

So I set the values for trail stop in ticks as recommended. It's done statically (so I do it in State.Configure as recommended). I technically do a SetTrailStop in relation to BOTH directions (long + short) separately but both are set to the same value. I only do this separate assignment to set IsSimulated to True (seems there is no other way)

Questions:
A. My opening orders are limits. But is it correct that the SetTrailStop are computed off the filled price?
B. I don't currently subscribe to marketdepth data. Is this problematic? I presumed there was no NT native marketdepth data for TickReplay but am wondering if somehow the orderbook has something to do with this phenomena?
C I do not see a difference when IsSimulated is True or False... Not sure if that is a problem or not?

All this points to the fact that there is some discrepancy between backtest and real time.. I can't quite trace it back. I have tried to model slippage into this at varying levels and don't seem to have any good convergence... is there anything I've missed with the implemenation of Trailstop?

Thanks!

[Leeroy_Jenkins]

Try to replay day/week/month and compare results with backtest. You're probably doing something intrabar so it doesn't reflect in backtests. If that the case use replay results only.

[tony_28217]

Sure. I mean intrabar trades on other backtests doesn't have other issues on my other tick replays but running Replay does replicate the discrepancy.. I have multiple data series so not sure how what you described is a solution for diagnosis?

One of the things that I see happening is the trail stop submitted as market orders can lead to rejections and ultimately strategy failures. So I am trying to understand the mechanics of the SetTrailStop and had listed multiple questions in the original post.

Thanks for the help but still looking for a bit more.

[NinjaTrader_BrandonH]

Hello tony_28217,

Thanks for your post.

SetTrailStop() is placed a set distance from the average entry price of the entry order.

OnMarketDepth() would only need to be used if you need to get information for every change in level two market data (market depth) for the underlying instrument.

When the IsSimulated property of SetTrailStop() is set to true it will simulate the stop order and submit as market once triggered. See the help guide page below about simulated stops.

https://ninjatrader.com/support/helpGuides/nt8/simulated_stop_orders.htm

Please review the help guide document on the differences on real-time vs backtest (historical).

https://ninjatrader.com/support/helpGuides/nt8/discrepancies_real-time_vs_bac.htm

A strategy running real-time (live brokerage account, live market simulation, Playback connection etc...) will produce different results than the performance results generated during a backtest.

When in historical data, only the Open, High, Low, and Close will be available and there will be no intra-bar data. This means actions cannot happen intra-bar, fills cannot happen intra-bar. All prices and actions come from and occur when the bar closes as this is all the information that is known.

Because of this, OnBarUpdate will only update 'On bar close' as it does not have the intra-bar information necessary for 'On price change' or 'On each tick' and the script will not have the intra-bar information to accurately fill an order at the exact price and time.

Below is a link to the help guide on Calculate.
https://ninjatrader.com/support/help.../calculate.htm

To improve the accuracy of a backtest, you may use Tick Replay along with an added 1-tick series to have logic processed intra-bar and have orders filled intrabar.
Tick Replay would be used to have the logic process OnEachTick or OnPriceChange with historical data, but this does not allow for intra-bar order fills. You would need to add a single tick data series and submit orders to that single tick data series for a strategy that uses Tick Replay.

High Order Fill Resolution allows for intra-bar order fills with historical processing, but is not compatible with Tick Replay.

Please reference the SampleIntrabarBacktest example and the following Help Guide links for more information.

SampleIntrabarBacktest 'Backtesting NinjaScript Strategies with an intrabar granularity' - https://ninjatrader.com/support/helpGuides/nt8/backtesting_ninjascript_strate.htm

TickReplay — https://ninjatrader.com/support/help...ick_replay.htm

Developing for Tick Replay -
https://ninjatrader.com/support/helpGuides/nt8/developing_for__tick_replay.htm?zoom_highlightsub= developing+for+tick+replay

Additional information may be found in this NinjaTrader Forum post —
https://ninjatrader.com/support/forum/forum/ninjatrader-8/strategy-development/100192-comparing-real-time-historical-and-replay-performance?t=102504

[tony_28217]

Hey Brandon, thanks so much for the reply but I think I need to clarify a few things.

1. I have a strategy that I have been running for 2 months every day from Futures open on Sunday to Futures close on Friday evening. It uses OnMarketData and TickReplay (with the requisite 1-tick data series and any other appropriate considerations). I use Calculate.OnEachTick. And OnBarUpdate is Empty within this strategy as I use OnMarketData for all the calculations. While I use a 30 minute time frame as my primary series, all of my trades (open and closes, long and short) all happen on the tick data series so while they do not function 'intrabar' on the 30 minute time frame, they do trace to each tick.

The reason I want to describe this in detail is that, for the last 2+ months, this strategy has had (except for a few points of slippage during highest volatility), nearly 100% convergence between backtest and realtime trading. I frequently rerun backtests over the last day or two to confirm this and have moved that now to weekly as I have such high confidence in the results.

So Backtest is 99.9% accurate to real time.


2. I have taken EXACTLY this strategy and added the SetTrailStops as I described in my initial post. And THAT is where the issues started.

Quite literally, nothing else changed. And yet now there is this discrepancy between backtest and live testing.

And my backtest now resembles 20% accuracy at best to real time.



In the strategy, I submit opening orders that are IsLiveUntilCancelled = True so, while they are submitted within the BarsInProgress data series that relates to the tick they do not close until necessary/expected.


Also I have confirmed with your colleagues in another message that there is no difference between the Live vs. Simiulated account data source as I pay for the monthly data subscription. See here for Zach's response https://forum.ninjatrader.com/forum/...ion-accurately This isn't really part of the conversation but you referenced differences between data sets. I do understand there are SOME, but this is not some. This is a massive difference so I can only infer there is some other dynamic at play here.


So I hope you can see why I am confused that the implementation of a trailing stop, there is not an issue between backtest/realtime until after I add it.


My biggest concern is understanding why this discrepancy is happening and how best to optimize and/or account for some variables that were otherwise doing fine prior to using SetTrailStop. I cannot optimize well in MarketReplay mode since this does not allow the use of the MarketReplay data within StrategyAnalyzer.

Hopefully you can shed light on this discrepancy.
Thanks.

[NinjaTrader_BrandonH]

Hello tony_28217,

Thanks for your notes.

To clarify, are you calling AddDataSeries() to add a 1-Tick series to the script and are submitting trades to the added 1-tick data series as noted in my previous post?

Please see this forum thread regarding comparing realtime, historical, and playback performance: https://forum.ninjatrader.com/forum/...nce#post100192

If adding that logic to your script caused it to behave differently then you would need to debug the strategy by adding prints to the script to understand exactly why the updated strategy logic is behaving differently compared to before you made changes. The forum thread linked above explains comparing realtime to historical performance.

Below is a link to a forum post that demonstrates how to use prints to understand behavior.
https://ninjatrader.com/support/foru...121#post791121

[tony_28217]

yes. I am using AddDataSeries() to add a 1-tick data series and submitting trades to that 1-tick data series. Precisely as you had laid out.

I had hoped there would be better answers than debugging as such because the root cause is so variable. Different order times, order directions, and ultimately losses on every trade.


I'll take a look at the printing but get the sense this will be extremely manual. Oh well. Gotta spend time to make money, right?