Market Replay vs. Real-world/Real-time

[ctrlbrk]

If I use the Market Replay feature, and initialize my strategy, and it ends successful - is this truly a real-world representation of what would have occurred?

I am talking about tick level data and tick level strategy.

I ask, because we already know that strat analyzer 'backtesting' is not real world accurate with a lot of precision. It's a great place to start and optimize, but not to be trusted in real world execution.

I am trying to determine if Market Replay should be treated the same, or if it is of a higher caliber of accuracy.

Mike

[NinjaTrader_Ben]

Hello,

This is user pereference.

I will stop short of telling you whether you should trust your results. However two things, amoung a lot of other things, you should consider:

1) Did the test data really represent most market conditions you may see?
2) Did your strategy happen to "fit" the data you used? In other words, if all of the data points were randomly moved slightly higher or lower, would your strategy still work?

These are just tips. I suggest doing some online research. There are lots of articles on this typic online.

[ctrlbrk]

Hello Ben,

I guess what I really mean has to do with the following:

If I run Strat Analyzer, Backtest, on say one specific day say 10/20, with very specific settings ie "10 second bars", I get result set "a".

If I run Market Replay, again on 10/20, with identical settings ie "10 second bars" and set everything identical to strat analyzer, I get a very very different result set "b".

The Market Replay data, at least for my strategies, is always less favorable than the Strat Analyzer. That in itself is a problem I am learning to deal with.

But the bottom line is, if I write my strats to be 'generally' profitable in Strat Analyzer, then I test them using as many days of Market Replay as I can stand, one day a time, and find them to be successful, is there another "gotcha" coming or is the Market Replay data considered more accurate than the backtest results?

Thanks again.

Mike

[NinjaTrader_Ben]

Hello,

The backtest is run off of the historical data that you have stored in your data base, the replay is a recording of the live data. The can be different. I cannot say which is better, it is the users choice.

This link may help with understanding why the data may be different:

http://www.ninjatrader-support.com/HelpGuideV6/HowDoesNinjaTraderBuildChartBars.html

[ctrlbrk]

Ben,

On that point, please help me understand.

I am recording Market Replay data. This is clearly tick-by-tick.

I have tried using backtest params of 1 Tick, a painful experience in patience, but nonetheless the results vary greatly vs the same day of market replay data.

I cannot wrap my head around why.

I am no longer using tick data because of the enormous amount of time it takes, but I have moved to using 10 second data and optimized my strats around that.

Still as I've mentioned, the results are wildly different from backtest to market replay even when you boil down ONE DAY at a time, like yesterday, where I have recorded both tick-by-tick historical data, as well as market replay data.

Why?

Thanks for your patience and helping me understand this very important concept.

I have read the How NT builds bars a few times, and am also reading the MultiTime Frame help guide again to try to get a grasp on this, but I am having a hard time of it.

Mike

[NinjaTrader_Ben]

Hello,

It is also important to note that whenever you are live there are always variables that any kind of simulator cannot account for. Anything can happen at the exchange. The Market replay uses a algorithm to file you in a simulated fashion. This algorithm attempts to simulate a real exchange but it is not the live market so the live market will always act differently in terms of filling orders.

Please review this link on the risks associated with electronic trading:

http://www.ninjatrader-support.com/HelpGuideV6/RisksOfElectronicTradingWithNinjaTrader.html

This link may be helpful as well:

http://www.ninjatrader-support.com/HelpGuideV6/DiscrepanciesReal-TimeVsBacktest.html

[ctrlbrk]

First, thank you again for the quick replies. You guys are awesome and I am happy to be a customer of such a great service oriented company.

Ok, all my orders are market orders. You are saying the Strat Analyzer simulated fill is a different algorithm than the Market Replay simulated fill? Still, we're talking about very minor slippage, if any, on something like ES which is what I am working on.

That wouldn't explain why the first trade of the day is at a completely different time when using strat analyzer vs. market replay, or why such a big discrepancy in the number of trades.

I am assuming the answer to that would be because the bar is built differently. Ok, but why is the bar built differently? NT has tick by tick data available to it. Is this a 'by design' issue that cannot be worked around?

I do not mean to be a pain. But, I want to trade using a automated strategy, and clearly I need to have a fundmental understanding of why the results differ so much. Or perhaps I could forego the understanding, if I could be told how to make the results more consistent with each other.

I have tried 1 Tick intervals, I have tried onbarupdate true/false, I do not know what else to try to make the strat analyzer more consistent with market replay.

Thank you again.

Mike

[NinjaTrader_JoshP]

ctrlbrk,

You need to understand the fundamental differences between backtesting and forward testing instead of foregoing it. Strategy Analyzer is backtesting. Market Replay is forward testing. They will yield different results as per the posts Ben provided. You are trying to reconcile something that is not suppose to be the same to begin with. Results from Strategy Analyzer is an approximation of a result. Results from Market Replay is another approximation of a possible result. Both of which you can argue are not realistic to real world environments. As such, you need to understand this nature of simulation in general and make your decisions with them in mind going forward.

As a side note, backtesting does NOT have tick by tick data available to it. You are running off OHLC values of bars. You do not know any of the tick by tick information within those bars. This is a critical factor in performance differences.

[ctrlbrk]

Thanks Josh.

[ctrlbrk]

As a side note, backtesting does NOT have tick by tick data available to it. You are running off OHLC values of bars. You do not know any of the tick by tick information within those bars. This is a critical factor in performance differences.

Josh, when using backtesting with a data series of less than 1 minute (for instance, select seconds or ticks), can you confirm it is using tick data to calculate the OHLC. Yes I understand I cannot reference the individual tick data within the strategy, but I am looking for confirmation the backtesting is using the tick data to form the bars.

Thank you.

[NinjaTrader_JoshP]

From tick data provided by your data feed.

[ctrlbrk]

OK thought so.

[masterjaz99]

A followup if I may...

Guys,
Thank you for this exchange, still have one lingering question. What if I add an instrument with a 1-tick bar? First, is that possible, and if so, then will it a) backtest correctly, and b) forward test correctly? I ask this for two reasons. One is for back/forward testing, second is that I need a real-time trailing stop that updates every tick (without manually tracking the stop using OnMarketData).

Here would be the code, strategy started with YM 15-min bar, then Add("YM xx-xx, PeriodType.Tick, 1). So with CalculateOnClose = true, I set up entry orders based on the close of 15min bar, but implement the trade using the tick bar.

Example order (within the BarsInProgress == 0):
EnterLongStopLimit(1,contracts[BarsInProgress], IHigh[BarsInProgress] + TickSize, IHigh[BarsInProgress] + TickSize, "Long YM");
SetProfitTarget("Long YM", CalculationMode.Ticks, targetTicks[BarsInProgress]);
SetTrailStop("Long YM", CalculationMode.Ticks, stopTicks[BarsInProgress], false);

Hope that makes sense, any help would be great. I'll actually be testing come Sunday when the futures markets open...

Aaron

[NinjaTrader_JoshP]

Hi Aaron,

It is possible and it will work the way you code it in backtesting and in real-time. 1 tick time frames are resource intensive though.

If you are simply trying to get your trail stop to update on every single tick, you do not need to add a secondary time frame. It automatically does this for you regardless of your CalculateOnBarClose setting.

[beastment]

I would add that the market replay data takes into account the bid/ask spread, which the historical data does not. The spread can ruin many a good trading plan.