Wildly different results in Strategy Analyzer vs Market Playback

[MatthewLesko]

I'm finding there is almost nothing close in results when I do a Strategy Analyzer and then the same criteria in a Market Playback.
Example is I did NQ 06-21 (2000 tick) with a strategy from 4/11 to 4/23 and the Strategy Analyzer showed it very profitable.
Then I ran the same period, same chart, on market playback between 100, 500 and max speed, and it's like negative thousands.
It is difficult to determine which one to use for testing. because when it comes to a live trade, it's going to be different as well.

[EminiTrader]

I have had similar experience when comparing results in Strategy Analyzer vs live trading (I ran tests where I ran a strategy in the live market for a couple of days, then re-ran in Strategy Analyzer with different results.

I found one source of issue involved intrabar granularity. Unless you select the "high" resolution in SA, the Analyzer doesn't "know" whether the stop or a target was hit first on trades where both the stop and the target are traded within the same bar.

I would suggest trying using "high resolution and set it to "tick". While this slows down the time to run the analysis, this resolved some of my issues. I may be wrong, but I seem to recall that you can't use "high" resolution if your strategy uses multi-timeframe data series - but I may be wrong on this point.

My issues was how strategy analyzer treats the situation where both a stop and a target are contained in the same bar, I believe it uses the direction of the first tick of the bar. If it is "up" then it assumes the higher price is reached before the lower price. (so for a long, the target would be reached first). Obviously this is going to be wrong a good deal of the time depending on the market being traded.

I hope this helps.

I never tested Market Replay, but I assume (dangerous as it is to do so) that it is an exact replica of the live market, tick x tick. I may test this assumption this next week, and if I find it is NOT the same, I will update you on here.

Good luck in your quest!

Some useful links

https://ninjatrader.com/support/help...ime_vs_bac.htm

and

https://ninjatrader.com/support/foru...ive#post773377

[MatthewLesko]

I have had similar experience when comparing results in Strategy Analyzer vs live trading (I ran tests where I ran a strategy in the live market for a couple of days, then re-ran in Strategy Analyzer with different results.

I found one source of issue involved intrabar granularity. Unless you select the "high" resolution in SA, the Analyzer doesn't "know" whether the stop or a target was hit first on trades where both the stop and the target are traded within the same bar.

I would suggest trying using "high resolution and set it to "tick". While this slows down the time to run the analysis, this resolved some of my issues. I may be wrong, but I seem to recall that you can't use "high" resolution if your strategy uses multi-timeframe data series - but I may be wrong on this point.

My issues was how strategy analyzer treats the situation where both a stop and a target are contained in the same bar, I believe it uses the direction of the first tick of the bar. If it is "up" then it assumes the higher price is reached before the lower price. (so for a long, the target would be reached first). Obviously this is going to be wrong a good deal of the time depending on the market being traded.

I hope this helps.

I never tested Market Replay, but I assume (dangerous as it is to do so) that it is an exact replica of the live market, tick x tick. I may test this assumption this next week, and if I find it is NOT the same, I will update you on here.

Good luck in your quest!

Some useful links

https://ninjatrader.com/support/help...ime_vs_bac.htm

and

https://ninjatrader.com/support/foru...ive#post773377

Thanks for feedback. Do I also have to download history before I run a strat analyzer against said date range?

[NinjaTrader_ChelseaB]

Hello MatthewLesko,

Below is a link to a forum post where I have discussed comparing results between playback and the strategy analyzer.
https://ninjatrader.com/support/foru...nce#post100192

1 tick intra-bar granularity, tick replay if using calculate with onpricechange/oneachtick, and testing over the exact same start and end bar is recommended.

Historical data is required for the Strategy Analyzer. This can be pre-downloaded or downloaded from a connected provider.

[IsaacBillion]

Hello MatthewLesko,

Below is a link to a forum post where I have discussed comparing results between playback and the strategy analyzer.
https://ninjatrader.com/support/foru...nce#post100192

1 tick intra-bar granularity, tick replay if using calculate with onpricechange/oneachtick, and testing over the exact same start and end bar is recommended.

Historical data is required for the Strategy Analyzer. This can be pre-downloaded or downloaded from a connected provider.

Hello Chelsea and MatthewLesko.
I've read all the other posts about differences between RealTime vs. BackTest:

https://ninjatrader.com/support/forum/forum/ninjatrader-8/strategy-development/100192-comparing-real-time-historical-and-replay-performance#post100192

https://ninjatrader.com/support/forum/forum/ninjatrader-8/strategy-development/1218724-a-testing-environment-that-closely-mimics-real-time-results-for-strategies
https://ninjatrader.com/support/forum/forum/ninjatrader-8/strategy-development/1148766-why-different-results-on-different-replay-results

https://ninjatrader.com/support/forum/forum/ninjatrader-7/strategy-development-aa/1218702-market-replay-vs-backtest

And also the TickReplay Guide:

https://ninjatrader.com/support/helpGuides/nt8/NT%20HelpGuide%20English.html?developing_for__tick_replay.htm

But I Still don't figure it!

I took the simple CrossOver 2 SMA strategy and enabled TickReplay . Added intrabar granuality 1 tick bar to the secondary series.
Sent the order to the secondary bar series with the method overload with barindex=1 (I know that if I'm adding the 1 tick bar there isn't difference
if i enable or not TickReplay- Correct me if i wrong)
AND I STEEL HAVE MAJOR DIFFERENCES BETWEEN Strategy Analyzer Backtest (with TickReplay Enabled) and PLAYBACK!
And Both of them i use HISTORICAL data (not Replay data because i couldn't find for TQQQ Replay Data - Can someone find me ?)
So i don't understand, it said in the TickReplay guide:
"Tick Replay is used to playback 1 tick historical data to build the bars as if they had been build live, this means that tick data will be thrown as Market Data events in historical and subsequently OnMarketData and OnBarUpdate events will be called as if it was live. "

So why i'm getting total different results between PLAYBACK and Backtest with TickReplay Enabled when the both using Historical data?
I understand there SHOULD be difference between Realtime or Simulated data because of the delay in order fill in RealTime vs. backtest and Playback which
fills the orders immediately.
But why the difference between backtest and Playback??

What do i need to do that Playback and backtest will be the same??
This is Critical because i'm trying to build a strategy and I'm optimizing via Strategy Analyzer to figure out the best settings via backtest with TickReplay
Enabled to simulate market data received tick by tick and using OnMarketData event. But if after i get great result in Strategy Analyzer and now moving to
Playback in the same dates and getting TOTALY DIFFERENT so what is the point?
I need a way to activate the SAME ENGINE of PlayBack as a backtest to the whole 3 months history to figure out the best settings.
How can i do that?
Isn't it the time that NT8 will unite these engines into one backtest and give same results that developer can build by that good Strategy?
Thank you Chelsea for the great Video and demo strategy for diff between BackTest/PlayBack/Realtime :

https://www.youtube.com/watch?v=2iM5BPpcUBg

But I steel don't understand why there is a diff between BackTest with TickReplay enabled and Playback and how to achieve same behavior to closely
resemble RealTime?
Thx So much for Your help.

Issac.

[barbaros]

I am running into this issue as well. What is the point of Strategy Analyzer if the results are not reliable? I added the 1 tick data series, submitted orders on that data series, but results are different as it is shown in the video...mainly playback vs the strategy analyzer. I know the realtime fills will be different but I would expect the Strategy Analyzer have at least match the playback.

What is the NT recommendation on this?

[IsaacBillion]

I am running into this issue as well. What is the point of Strategy Analyzer if the results are not reliable? I added the 1 tick data series, submitted orders on that data series, but results are different as it is shown in the video...mainly playback vs the strategy analyzer. I know the realtime fills will be different but I would expect the Strategy Analyzer have at least match the playback.

What is the NT recommendation on this?

Agree barbaros!
Very Very frustrating!