Historical Data keeps changing under the same conditions

[pvroberts]

Would there ever be cases where it isn't at all the logic in my Strategy, but the data it is receiving to which it reacts?

Can that data change from computer to computer and/or day to day?

For example, will the data it receives today, live, be different than the data that it assesses for that same day, but evaluated tomorrow?

[NinjaTrader_Jesse]

Hello pvroberts,

Data is a big factor between tests, if the data isn't identical the tests likely won't be identical. Your price conditions in a strategy rely on math and data points, if the datapoints are not the same the result likely wont be.

Data can be different from computer to computer which is why debugging is important.

The data from today should be the same as historical data however there can be changes on the data providers end which is another reason debugging is important.

[pvroberts]

Let's assume we are well aware that the data needs to be identical.

What are some of the ways debugging can be useful if the data is changing?

[NinjaTrader_Jesse]

Hello pvroberts,

Debugging is useful to let you know what specifically was different. If you cannot visually see from the results the exact reason for the differences that would be how you approach finding out.

[pvroberts]

Jesse,

What is useful about knowing what specifically was different?

In the video I watched provided by NT, the logs had differences of milliseconds...milliseconds of tick data is apparently changing the results?

The logic in my Strategy isn't changing, it isn't very complex, it requires no granular data, and is being run on 15m candles.

These are On Bar Close concepts, on purpose, with lots of price overlap occurring that should trigger the logic without variation.

If I run the Strategy in real time against the real, current, evolving market, I get result A.

When I backtest that exact day with the exact Strategy, whether it's using the Strategy Analyzer (High Order Fill Resolution), or watching it in Market Replay, I get result B.

The very FIRST time I used Tick Replay, I was able to get result A on the backtest and thought everything was solved, but when I ran it a second time, it changed to result B.

I have not been able to reproduce result A in a backtest situation since the first time using Tick Replay, and NOTHING was changed.

Why would this be?

How can I create an environment where the backtest data will match the real time data perfectly?

[NinjaTrader_Jesse]

Hello pvroberts,

If you know what is different then you will know why the results were different. If you want to know why results are different that would be the way to find out.

I understand the logic is the same but that does not mean it will execute identically, that is where debugging comes in to find out of it did or did not. If it did not you likely will get different results.

There is no way that anyone could answer why two test results were different without doing that kind of investigation, without details on how the logic executed it would be a guess as why they may have differed. The platforms user interface shows values that are relevant to the result set so those are only useful to know results based on execution data. To know how that was actually produced you would need to know how the logic worked which is why debugging is important.

There is not any specific guidance that I could suggest to match realtime perfectly, there will more than likely be some differences in some use cases. This highly depends on how your strategy logic works historical vs realtime, what properties you are using, how indicator or price values you use are being used etc. the only way to identify points where that may differ would be to debug the code in both use cases.

[pvroberts]

Jesse,

Thank you for taking time with me about this.

Taking on the long and arduous process to build in logs and scrub those logs to see how the market moved a tick in milliseconds, and conclude that "that's why the results are different" doesn't seem either practical or (and please forgive my blunt remarks) useful. Why the results are different isn't my goal at this point. It seems clear the data the logic is reacting to in the live market is different to the data the logic is reacting to in the NT provided Strategy Analyzer or Market Replay features. I am not comforted to know that the data changes as it basically destroys everything I've been working on.

Seeing that it's clear the data changes from the live market to how it is settled as "history", I'll have to rely on other tools that make more sense for my goals.

Thanks again for your time and input.