Trying to understand a gap between replay testing and Live Sim ...

[Learning1]

Hi There,

For the past few days, I have been testing a new strategy in Live Sim and then the same day in replay. I have made certain that all delay settings are 0 in the simulator, but I have left "Enforce Immediate Fills" (EIF)Unchecked. My understanding (Which may be mistaken so I want to confirm it) is that as long as all delays are set to 0, the results between Live Sim and Replay testing over the exact same period should be relatively the same.

In the case of the strategy I am testing, LIVE SIM from this last Wednesday produced a Net Loss of -1213 / contract on 369 trades (Long and Short). On the other hand, the REPLAY of the exact same day, same times, same settings (No EIF, 0 delays on all else) produced a net gain of +2767 / contract on 652 trades (In both cases, there is no position sizing applied yet - All trades are 1 contract). As you can see, the gap between Live Sim and Replay is not a couple of trades and a hundred or so dollars / contract. It is enormous so I am trying to understand the reason for this.

So with that background, here are my questions:
1) Is the NinjaTrader Fill Algorithm Logic used in Live Sim (No EIF, 0 Delays) the same as the fill alogorithm in replay (No EIF, 0 delays)?

2) I notice that if I run the replay at 4x, it starts out at that speed, but after an hour or 2, begins to slow down ... to the point where it is maybe running at 1/4x speed near the end of the day instead of the 4x that the replay slider is set at. This seems to be happenning as the trade count increases (the strategy uses a fair bit of processing ... CalcOnBarClose == false & it runs 5 timeframes and about 10 indicators). When running in Live Sim (Zenfire datafeed) there have been periods of 3-20 seconds where the chart (ES) isn't moving. It is as though NT isn't processing the data as fast as it is coming. Is this indicative of Machine Performance / Software performance issues?

3) Are there any other possible reasons you can think of for this gap in the 2 testing methods?

Thanks for your help,

Learning1

[NinjaTrader_Ray]

Someone will respond on Monday, we are actually officialy closed today.

[NinjaTrader_JoshP]

The fill algorithms are the same, but there is no guarantees they will play out the same. There are differences in the replay that allow you to replay it at different speeds that can affect the fills. There just isn't anything you can do.

If your replay is slowing down and you also see this in live it is likely your strategy using up all of your computer's computing power. Please see these general performance tips: http://www.ninjatrader-support.com/H...rformanceTips1
You may also need to tweak your strategy. If this is not possible there may be a positive impact if you run on an as-strong-as-possible computer.

[Learning1]

Thanks for the reply Josh and for providing the perspective on the fill algorithm in Replay and Simulation with Live Market Data. Can you clarify my understanding on the statement, "If this is not possible there may be a positive impact if you run on an as-strong-as-possible computer."

I have already followed the majority of the suggestions in the performance tips, but the strategy is unquestionably computationally intensive. It is being run on a Quad Processor with 2 GB of Ram, but it also uses 5 data series in a Multi-Timeframe approach with CalculateOnBarClose == false. To partially offset the calculation load the calculations / order submission only runs when (BarsInProgress == 0 and FirstTickofBar).

With that said, your earlier comment caught my attention. If the computer / NinjaTrader combination is having trouble keeping up to the data frequency/SystemIO/Calculation Load, what would you expect the impact to be on the stratey results in Simulation with Live Data vs. Replay and why would it be "a positive impact"? Also, did you mean "a positive impact" in Simulation with Live Data, or in Replay?

Thanks again for you perspective...

Learning1

[NinjaTrader_Bertrand]

Hi Learning1,

This means you should try to optimize the efficiency (code wise) of your strategies as much as possible to minimize the use of your system resources. If this is not possible at this point, then using the strongest possible computer setup may improve your issue. Personally, I would add 4 GB of quickest possible RAM to your system, a bargain these days.

Have a great day!

[Learning1]

Thanks for the input ... However I don't think RAM is the sole issue. I have watched the System Resources closely during both replay and Simulation with Live data and the Ram is not restricted (Generally running under 1 GB).

I guess the core question I am trying to understand is: Assuming computer hardware is not constraining a strategy running in Simulation with Live Data (noEIF fill algorithm, zero delays), what would you expect the Replay (noEIF fill algorithm, zero delays), results (Net Profit, Drawdow, etc) to be over the exact same period?

[NinjaTrader_JoshP]

Learning1,

Impossible to say. There are just too many factors that can play out here. In terms of the replay being representative of real-time. For sure I would consider it a very good measure, but like all measures it will not be able to 100% capture the real world environment. Replay being different than sim on live feed can happen due to differences in the replay architecture. One way you can try to address it is to run at 1x, but even then there is no guarantee it will play out exactly the same.

Also, all of this is very highly variable dependent on your code. The bottom line is it is what it is. Just like backtest results, replay results and sim with live feed results should always be taken with a grain of salt because real-time environments on a real trading account will be guaranteed to be different.