Different BackTest Results Same Strategy

[Peppo]

Hi,

I have the same strategy running in backtest on 2 different machines. The time is really important for the strategy because I enter a trade at a specific time.

The machines are reporting 2 different results. The timezone in NT8 for both machines is set to UTC-06:00. I checked the trades and I found out that there are some trades where the time bar is different. This is completely unexpected.

I deleted the database for historical data for both machines, run the backtests on both machines but the problem is still there.

Since the machines had 2 different windows timezones, I also aligned them to the same UTC+01:00. I deleted the databases and run again the backtests but the problem persists.

If I set the same timezone in both NT to UTC-06:00, I expect to have the same timestamp for all the bars but this is not the case.

How is it possible?

Thanks.

[Peppo]

Hi,

sorry just a correction.By having the same timezone for the 2 NT in both machines and the same windows timezone, I get the same results.

Of course, this does not explain why by having the same timezone for NT for both machines I don't get the same results.

The NT timezone should be completely independent from the windows timezone.

Any suggestion?

Regards.

[NinjaTrader_Jesse]

Hello Peppo,

If there are any differences in configuration that will cause differences down the line in your tests. The tests need to be exactly identical to produce the same results.

When comparing any two test results there are some reasons that would cause the strategy to calculate differently. While we would be unable to provide an exact answer of why that is with your custom strategy you can debug the strategy to find out the exact answer. Because every strategy is different there is not one single answer as to why that would be different, that heavily depends on the settings you used and how the strategy was coded.

We have a guide linked below that goes over how you can add debugging to the strategy to explore differences between tests. The guide goes over comparing historical and realtime but would apply in any two tests. The short summary is that you add prints into your code and then collect the output after running the strategy in the firsts test and then do the same process for the second test. You can then compare the prints to see how the strategy logically worked in both use cases. Based on your observations you can make any changes needed to have the strategy results become more close to each other.
https://support.ninjatrader.com/s/ar...language=en_US

[Peppo]

Hi Jesse,

I think I was not clear enough.

Here there is not historical vs real-time. I have taken the EXACT same strategy (copy/paste) and applied on 2 different machines having the same timezone set to UTC-06:00 and run the backtest with the same instrument, same time-frame, same data feed and same configuration.

I compared the trades one by one and found out that the problem lies in the in the interpretation of the timestamp bar. The long entry is done at certain bar time (for example 04:00 pm) and I see that on the second machine the same bar has a different timestamp and this, of course, is causing discrepancy in the results.

How is it possible that having the same timezone in both NT I get different results?

Is NT relying on windows timezone? Because this is the only difference and when I align them the results are exactly the same.

Thank you.

[NinjaTrader_Jesse]

Hello Peppo,

I mentioned that is for any test, not just historical or realtime. That is a general approach at debugging tests. I couldn't tell you what specifically was different in your tests, you have to add debugging to know what was different.

If you find a specific piece of code and have a question about it in regard to a specific step of steps we can test that to see if thats expected.

[Peppo]

HI Jesse,

I attach you the screenshots of the charts for both machines for NQ JUN25 on a 15min timeframe. In both cases the Timezone is set to "(UTC-06:00) Central Time (US & Canada)".

I took a random day (26/05/2010) and as you can see the bars looks completely different (just take a look at the first bar: one is red while the other one in green).

The only difference is that the 2 machines have a different windows timezone.

Could you please do a check also on your side?

Thank you.

[NinjaTrader_Jesse]

Hello Peppo,

If the data was different between the two machines then that would cause different results in the tests. I would suggest to remove all data on both machines using the historical data manager and then redownload the data to make sure they match up. If you are using different timezone settings on the PC's that would be one reason why the bars may look different when built.

https://ninjatrader.com/support/helpGuides/nt8/historical_data_manager.htm

[Peppo]

Hi Jesse,
Indeed, the different results in the strategy are due to visible differences in the chart.
I have deleted the historical data directly in the NT8 folder and repeated the backtest but the results are still different meaning the charts are clearly different.
As I said, the only difference is the windows timezone setting for the 2 machines. The NT timezones for the 2 machines are exactly the same. I don't understand how is possible to have such a discrepancy in the interpretation of the bars timestamps.
My suspicious is that NT is somehow using the windows timezone to build up the bars which is completely wrong for me.
Can you please check it also on your side?
Thank you.

[Peppo]

Hi Jesse,

I send you the pictures for both machines having the same windows timezone. The charts are exactly the same.

There is something not clear with how NT is building bars having different windows timezone.

If this is the case the setting to adjust the timezone in Tools/Options is completely useless.

Regards.

[NinjaTrader_Jesse]

Hello Peppo,

If you are using different time zones on the pc that could affect the platform. Because you are trying to do a test where everything needs to be similar I would suggest changing the PC time zones to match along with having the same settings in the platform. That will let you see everything the same so the test should be able to be similar.

[Peppo]

Hi Jesse,
The test is nothing more than taking a trade at certain time (for example 14:00). That's all.
I am wondering what is the scope of setting timezone in NT if it doesn't apply at all. Whatever windows timezone you have thus shall not have any effect on the bar timestamp. If I set the same timezone in 2 different NT platforms the bars timestamp must simply be the same. It has to be completely independent from windows timezone.
That is a clear bug that has to be fixed.
Regards.

[Peppo]

Hi Jesse,

do you have any news on this topic?

Thank you.

[NinjaTrader_Jesse]

Hello Peppo,

I would still suggest to change the timezone to the same time to avoid any differences.