Market Replay, inconsistent results

[lookOutBelow]

I'm having a recurring problem that is hard to reproduce, but is happening frequently enough to concern me.

If I'm using Market Replay to test a strategy, and I'm making a bunch of changes, and then rerun the strategy over and over across different days, I could get consistent results on a particular day multiple times. However, after disconnecting from MR and then reconnecting I don't get the same results. The data hasn't change, the strategy hasn't changed. Nothing has changed except disconnecting and reconnecting from MR. The results can sometimes be drastically different.

Is there anything that would explain this? It is pretty frustrating.

Thanks

[NinjaTrader_ZacharyG]

Hello lookOutBelow,

Thank you for your inquiry.

It is expected that there may be slightly different executions due to how the simulation engine works. However, as you have stated, your results are drastically different.

Market Replay data should be consistent every time. Are you noticing any differences in the bars being loaded during your Market Replay runs?

Please provide screenshots of the differences that you see and post them to your response so I may investigate further.

To send a screenshot with Windows 7 or newer I would recommend using Window's Snipping Tool.

Click here for instructions

Alternatively to send a screenshot press Alt + PRINT SCREEN to take a screen shot of the selected window. Then go to Start--> Accessories--> Paint, and press CTRL + V to paste the image. Lastly, save as a jpeg file and send the file as an attachment.

Click here for detailed instruction

[lookOutBelow]

Thanks for the response.

It is one of those problems that is unpredictable and I can't purposefully reproduce it. So, it is hard to answer your questions. Here is an example of what causes the issue.

1) I can run the same day over and over again in MR and results are exactly the same.
2) I do a bunch of changing of strategy code to try different things, as expected, the results are different.
3) I like what I'm seeing. But, the results almost seem to good to be true over multiple days.

so, I leave my strategy alone (no new compiles), disconnect from MR, and sometimes re-start NT.

4) Reconnect to MR on a day I previously ran.
5) Run that day with that same compiled strategy under the same session/settings and the results are now different.


Maybe eventually I'll accidentally figure out specifically what screws it up. But it normally only happens after extensive strategy changes. Trying to figure out what did it (since I don't know immediately when it goes bad) may be impossible. I just didn't know if you've ever heard of this happening.

For the record, I'm using multiple instruments and timeframes (1mins and 1 ticks).

Thanks

[sledge]

The only thing I've had issue with its 2nd item added in code doesn't always start on the first day at open but is delayed.

You should really compare trade results to see if there is differences throughout the day or just say early on.

[NinjaTrader_ZacharyG]

Hello lookOutBelow,

I would suggest to continually run your strategy to see if you continue to have wildly different results each time without disconnecting from Market Replay.

As long as there are not any differences in the bars being loaded each time you run over the same time period in the Market Replay, slightly different executions are expected due to how the simulation engine works.

[lookOutBelow]

Hello lookOutBelow,
I would suggest to continually run your strategy to see if you continue to have wildly different results each time without disconnecting from Market Replay.

It will remain rock solid while it is connected. Same results each time. It is only after I disconnect and reconnect that it can sometimes change.

As long as there are not any differences in the bars being loaded each time you run over the same time period in the Market Replay, slightly different executions are expected due to how the simulation engine works.

This is interesting. Are you saying that if you run the same data without changing any variables in the strategy or market data or session, etc, the simulation engine can have different results? Is this by design? If so, that is a concern. We need to have consistency in these things when developing strategies.

Thanks

[lookOutBelow]

The only thing I've had issue with its 2nd item added in code doesn't always start on the first day at open but is delayed.

You should really compare trade results to see if there is differences throughout the day or just say early on.

In my case, that is hard to do. This strategy doesn't use indicators to drive entries/exits. It quotes both sides of the bid/ask and trades continuously all day long with each entry/exit driving another entry/exit. (It is a market making strategy hedging across multiple instruments.) Basically, if the early on trades are off, the rest of the day will be off.

Thanks

[NinjaTrader_ZacharyG]

This is interesting. Are you saying that if you run the same data without changing any variables in the strategy or market data or session, etc, the simulation engine can have different results? Is this by design? If so, that is a concern. We need to have consistency in these things when developing strategies.

Thanks

It's possible to get different executions, however, the differences will be minor. So, you may get executions that are a tick off. However, wildly different results, such as many ticks off, is not expected.

Just to verify, are you using a sim license key or a live license key on your NinjaTrader platform? When using a live license key, the Market Replay connection will consist of historical data that goes beyond the time span you have specified in the Replay playback window based on the days to load and end date in the chart's Data Series window, whereas the sim license key would have only the replay data loaded on your chart and no historical data loaded, causing a difference.

[lookOutBelow]

I'm using a live license key.

For now, since I don't know what else to do. I'm disconnecting and reconnecting to MR every 15 minutes or so. Hopefully this will get around this problem. Whatever may be causing it.

[NinjaTrader_ZacharyG]

Hello lookOutBelow,

When using a live license key, the Market Replay connection will consist of historical data that goes beyond the time span you have specified in the Replay playback window based on the days to load and end date in the chart's Data Series window, which may be causing these issues.

You could try to run your strategy on a sim key to see if these issues continue to occur. If you do not have the sim key, please email us at platformsupport [AT] ninjatrader [DOT] com and we can provide that for you.

Please let us know if we may be of further assistance.