Identical data sets, Strategy Analyzer different results on diff computers

[Skifree]

I have desktop NT and VPN Desktop NT, and a strategy for MCL, with a short config. Both timeframes are set to CT.

When i copy over my file and the template to the VPS, I run the same timeframe for both. My strategy doesn't have any randomness to it, but I see differences in the results? When i run it locally, i can the same results, and on VPS i get the same results, but the two don't match. What would cause this discrapany?

[NinjaTrader_Jesse]

Hello Skifree,

You will need to debug the strategy on both machines to see why there is a difference. Common differences would be based on different data between tests or different settings. You can see the following post that goes over how to debug a strategy when comparing it against other tests, that would also apply in your case when comparing it against another pc.

https://ninjatrader.com/support/forum/forum/ninjatrader-8/strategy-development/100192-comparing-real-time-historical-and-replay-performance

[Skifree]

I'm not sure how the data could be different. Both is historical data downloaded from the NT UI for MCL Dec 23. The code itself is exactly the same, as are the parameters

[NinjaTrader_Jesse]

Hello Skifree,

Without debugging the script the only information you would have would be to look at the strategies executions to try and determine what was different. That can be a very time consuming process if the test has many executions. The easiest way to compare tests to to add debugging in the strategy so you can get a print out of how the logic runs in each test and that can also help to confirm all data was exactly the same.

[Skifree]

Hi Jesse,

I think I found the issue but i can't explain it - does NT distribute data differently based on machine geo (VPS in EU, I'm in US)? In this image, you can see that even though the time periods are the same, my EMA(200) is below a bar in one case and above in another.

How can I compare the downloaded data for discrepancies?

[NinjaTrader_Jesse]

Hello Skifree

Yes a variety of servers can be connected to for historical data however they should be in sync. If indicator values are different that would indicate there is a difference in data between the tests. You can delete the historical data that you have on both machines and then reload it as one step. Alternatively you can copy the historical data from one machine to another. https://ninjatrader.com/support/help...ta_manager.htm

[AmberBamber1999]

Wow! That should NOT happen imo! Wow! Thanks for the idea. I will test our strat with different people with different locations. They should all come out EXACTLY the same. If not I would be looking at NT for some SERIOUS issues with replay. We run a script live with hundreds of people and 99% get the exact same results except for slippage.

[NinjaTrader_Jesse]

Hello AmberBamber1999,

Replay data is downloaded from a central source so that should not be an issue, if you had a difference in data that means you need to update both PC's to the current download from playback. Both pc's need identical data sets to have the strategy perform the same when using the same parameters. If any data is different they may provide different results based on how your logic executes on that unique dataset. This is why debugging the strategy is an important step wen comparing two pcs to ensure every part of the test was identical. For examples on how to debug a script in tests please use the following link:: https://ninjatrader.com/support/foru...ay-performance

[AmberBamber1999]

OK I've found PROOF and already filed a ticket with NT and they confirmed the data is WRONG. This error has happened on my Replay a LOT and NT always says the same thing about it being my script or code or something and ITS NOT TRUE! The data isnt downloading right into Replay OR the data you have is corrupt. NT now has proof and has submitted it to the data team. So take note traders! IT'S NOT YOU! THIS IS AN NT/DATA ISSUE. Please keep sending them tickets. Replay is USELESS until this is fixed. If you want to run a week, month or 60 back test.

[NinjaTrader_Jesse]

Hello AmberBamber1999,

If you opened a support ticket please follow up in that email chain if you needed further assistance surrounding data. I am unaware of what was discussed in that email to provide any assistance with that. In regard to your original question about different results, if any data was different on the two test PC's you likely will see different results. The datasets need to be identical so testing that would be a first step in making sure the tests align.

[alphatango]

Hah! I was just dropping in here to see what the deal was. I have the exact same issue. I have wildly different outcomes with Backtest and Optimization on the same computer. I took it to another computer and it seemed more sane, but now I have no idea what to trust.

if I backtest from my main computer, in 2 months, I trade some 24million dollars worth of trades and I come out +/-. There's not a chance in heck my algorithm would trade that much in 40 days. What data do I delete and redownload to try and fix this?

[NinjaTrader_Jesse]

Hello alphatango,

If you want to delete your data you can use the historical data manager. https://ninjatrader.com/support/help...HistoricalData

To reload data you can open a chart with the days to load for the period you wanted and then right click -> reload historical data.

When comparing tests and getting different results you may need to debug the strategy to find the reason, the following post goes over how to debug between multiple tests. This applies to the same pc or different pc tests. https://ninjatrader.com/support/foru...ay-performance

[AmberBamber1999]

Hah! I was just dropping in here to see what the deal was. I have the exact same issue. I have wildly different outcomes with Backtest and Optimization on the same computer. I took it to another computer and it seemed more sane, but now I have no idea what to trust.

if I backtest from my main computer, in 2 months, I trade some 24million dollars worth of trades and I come out +/-. There's not a chance in heck my algorithm would trade that much in 40 days. What data do I delete and redownload to try and fix this?

there isnt a fix yet... we need good data in the first place and it seems NT isnt getting good data for every day.. most days they do but when you get an error or different results on different pc's this may be the cause.. NT is looking into the issue.. I have got Replay to work great on a 60day test then the next day i would get a few days of errors.. its very inconsistent and they are now aware its them and/or their data.. once we get good data then I think Playback will work a LOT better.. Good luck..

[alphatango]

Can anyone answer the question for me: What is the difference between Historical data and Market Replay data? Both are the prior days/weeks/months.

[NinjaTrader_Jesse]

Hello alphatango,

Playback data is recorded live data and historical data is just OHLCV datapoints which is less granular. Playback data allows you to play back through a period of time just as it would have been seen in realtime.