Historical Data keeps changing under the same conditions

[Tidan2]

I have multiple SIM trading accounts. Each account has an automated strategy tied to it.

When I look at Strategy Performance - Historical for each of the strategies, it gives me data that never stays consistent between days. One day the numbers are one set of values and then 24 hours later those same days have different results. This should not be happening and is very worrying in regard to back-testing where inconsistent data is a nightmare. All of my strategies use "on bar close" so it's not a case of the historical data not know what the price was. I think I've had this problem in the strategy analyzer as well where the same backtests and optimizations suddenly pump out different results, but it's harder to confirm there whereas here it was quite easy to confirm all the numbers were different.

Any idea what the issue could be? Is there something in my strategy code that relies on something that doesn't remain consistent?

[NinjaTrader_Jesse]

Hello Tidan2,

Are you restarting the platform or restarting the strategies between viewing the results? Its possible more data was included if you restored the strategies which could result in different results. It is also possible this relates to your code, I would suggest trying the SampleMACrossOver and see if you see the same result.

[Tidan2]

Good suggestions, thanks.

I'll reset all the accounts and log the strategy performance data before turning them back on next week. And I'll add 1 or 2 that use the SampleMACrossOver strategy for comparison. Hopefully it's just me and they start aligning or at least it shows me more specifically where the issue is.

[Ttrade12]

Hello Everyone,

I'm having the same issue. Something weird is going on with the historical data for the values of my indicators and the values sometimes changes when I restart NinjaTrader. I use My NinjaTrader Continuum for my real-time data. I look at the 25, 50, and 100 tick charts. My indicators are set to calculate on bar close. So, all of the values for each indicator should already be set and fixed. However, sometimes the values for the indicators slightly change.

For example, the value for the BOP indicator, setting at 14, on the 25 tick at 12:59:24 on 8-2-23 will be .102 for the MYM 12-24. And, for whatever reason, the next day I restart NT and reconnect to Continuum and now that same BOP bar's value at 12:59:24 is .098.

The above example has happened several times. Shouldn't the historical data on the bars stay the same whether I restart NT or not? Why does this happen? Is there anything I can do to make sure I'm always getting accurate historical and real-time data?

Thanks for any explanation and understanding you can provide!!

[eDanny]

Could this be because tick bars are built on tick data and if you start on another bar, the tick data starts building the bars from the new bar? If your charts starts twos days back historically at the open, the bars start building at the open. The next day you are starting at the open but 24 hours later (the next day) and tick bars start building. This is why tick bars vary. Try setting your data to RTH and then compare to a setting of ETH and the bars will be completely different.

[Ttrade12]

Thanks eDanny for your explanation. But, it seems like indicator values are what they are and should be fixed. If I use the same session time template all the time, how can the indicator value on a bar's data from 2 months ago change just because I restarted NT?

[eDanny]

Does your template allow the beginning historical bar to be at the same time and same date every time? On the next day does your historical start bar move forward one day?

[Ttrade12]

I use the default session template 'use instrument settings' for the MYM contract, so I see all of the data for each trading day. The data is ongoing as opposed to starting and stopping at a specific time each day.

[Ttrade12]

I also have only noticed this issue with the BOP (balance of power) bar values.

[pvroberts]

My experience so far with backtesting and forward testing my strategy has been incredibly frustrating to date.

First, the backtest results changes for the strategy based on if I choose Standard Order Fill Resolution vs High Order Fill Resolution at 1 second or 1 tick levels.

I am currently operating under the assumption that 1 tick level is the most accurate, although I get the same results faster if I just use the 1 second setting.

If I run the resolution at 1 second or lower, the backtest results are suddenly capped to 1/3/2022 vs 1/1/2020 in Standard. So I'm now down a couple years of data to reflect on, but at least (I assume), it's accurate at this point.

But now there's a new obstacle -- the real time results are not the same as the current historical results, AND APPARENTLY THEY CHANGE!

Yesterday (01.28.25), I let my Strategy run for the full day and was pleased to see that it produced +$105 on the accounts -- so the real time forward test result was +$105.

Today, (01.29.25), I let the Strategy run for the full day again and was not pleased to see that it produced -$1012.50 -- but...I also ran it on another computer with the exact same settings and data feed where it produced -$565. This was the EXACT STRATEGY with the exact parameters and the exact data feed, but on a different computer. How is this possible??

In addition, the historical backtest is now showing that yesterday (01.28.25) the Strategy produced +$900 instead of +$105!!

What's going on here?

How can anyone work with this?

What can I do to ensure consistency and not see the history changing??

Please, somebody help to explain these differences and what I can do to ensure the data I'm getting to analyze can be dependable!

[NinjaTrader_Jesse]

Hello pvroberts,

When comparing realtime and historical results there are some reasons that would cause the strategy to calculate differently. While we would be unable to provide an exact answer of why that is with your custom strategy you can debug the strategy to find out the exact answer. Because every strategy is different there is not one single answer as to why that would be different, that heavily depends on the settings you used and how the strategy was coded.

We have a guide linked below that goes over how you can add debugging to the strategy to explore differences between tests. The guide goes over comparing historical and realtime which is what you explained.

This page lists some common items which will cause differences between realtime and historical, you can use this page to see if any of these apply to your use case:
https://ninjatrader.com/support/helpGuides/nt8/discrepancies_real-time_vs_bac.htm

This guide goes over how to add debugging and then compare results. The short summary is that you add prints into your code and then collect the output after running the strategy in realtime. Then restart the strategy so it processes historically and then collect those prints as well. You can then compare the prints to see how the strategy logically worked in both use cases. Based on your observations you can make any changes needed to have the strategy results become more close to realtime.
https://support.ninjatrader.com/s/ar...language=en_US

[pvroberts]

I will be sure to look at the links you've provided, thank you for that.

Perhaps you can help to explain how adding prints into the code will help if my problem is as follows:

1) When I built the Strategy I noticed the results were inconsistent with the Market Replay and the Historical Data backtests. For example, I would run the quick Strategy Analyzer for a given week and note the results, then take the time to download the Market Replay data and run the Strategy as a forward test and note the results. I combed the NT Forums to find explanations and adjusted the Strategy and test settings to after many weeks of fiddling around get the Market Replay to match the Analyzer, which essentially came down to running a test using the 1 second or 1 tick Order Fill Resolution setting. So I thought everything was solved and the data was good to use moving forward.

2) I run the Strategy using the live market and track the results. At the end of the day, it produces +$105 (live). Each trade has a unique text and marker that is easily visible on the chart (will provide a screenshot). But... when I download the Market Replay and rerun the day, it produces +$900 (backtest, historical & Market Replay), and I can't get it to match what actually happened.

3) I run the Strategy again the next day and at the end of the day, it produces -$1000+ (live). But...the backtest and Market Replay for the day prior still shows +$900 for the Prior Day.

I could accept the -$1,000 result if I had actually gotten the +$900 as it suggests that the Strategy should have produced, but that wasn't the reality of the trade.

The backtest data is overall cumulative net positive, BUT, and a BIG BUT, what happened in real life isn't reflected in the Historical Data or the Market Replay for the given events.

What kind of debug can I do for my Strategy that would solve this problem?

[NinjaTrader_Jesse]

Hello pvroberts,

1) that depends on how you coded the strategy, there are quite a few priperties or scenarios which will cause a strategy to perform different between tests. Debugging with prints is how you can add custom output to know how your logic works in each test.

2) for results to be the same every aspect of the tests need to be identical. If there are any data differences or differences in how the strategy executed you will see different results. Past or historical results or playback results are not set in stone, those are an estimation. You may see differences in realtime.

3) This is the same answer, you would have to debug the strategy to know what was different that cause different results.

[pvroberts]

What exactly would I be looking for in the debug that would allow me to make the results the same?

I want every aspect of the test to be identical, so how can I produce that environment?

[NinjaTrader_Jesse]

Hello pvroberts,

The sample I linked to provides some examples for that specific script. Your prints will be unique to your script and its logic and what you need to debug. You would have to print out information that is relevant to the test, for example the bar times to make sure data is the same, when conditions become true and relevant variables etc.