BackTests in different Time Zone get incorrect start/end time for Trading Hours

[lorien]

When I'm in a time zone other than Central Time (UTC-06:00), the Back Tests in Strategy Analyzer that I have seem to be messed up, i.e. the start time and end times are skewed. So, I tried doing the followings but it still isn't correct:
1) Set the time zone to local time zone via "Control Center" -> Tools -> Options -> General -> "Time zone"
2) create a new template for trading hours and set the "Time Zone" in the template to reflect the local Time Zone instead of the Central Time zone

Please help. I've attached the settings I did.

Regards,
--
Ben.

[NinjaTrader_ChelseaB]

Hello lorien,

The trading hours template should always reflect the time zone of the exchange and not your local computer.

Just as a heads up, if the time zone is changed, all historical data must be redownloaded.

May I confirm you have deleted the historical data and re-downloaded this?

[lorien]

Hi Chelsea,
Thanks for the reply.
Just to make sure I understand what you said:
1) Chelsea> The trading hours template should always reflect the time zone of the exchange and not your local computer.
Ben> So, in this case I should just use the default "Trading Hours" templates like the followings with their respective Time Zones?
i) CBOE US Index Futures ETH: Time Zones set as "(UTC-06:00) Central Time (US & Canada)
ii) CBOE US Index Futures RTH: Time Zones set as "(UTC-06:00) Central Time (US & Canada)

2) And let the time zone being set to local time zone via "Control Center" -> Tools -> Options -> General -> "Time zone". Is this correct?
Actually this is automatically set to local time zone (or set the same as the computer local time) when I invoke Ninja Trader in a different time zone.


Chelsea> "May I confirm you have deleted the historical data and re-downloaded this?"
Ben> Actually I upload a new set of historical data after I am at a new time zone.

Ben> When should I re-download the historical data? Is it every time after I am at a new time zone?

New related question1> If I bought a set of historical data from Vendor A, I assume that I do not need to re-upload the data every time I am at a new time zone, correct?

New related question2> If I've uploaded a set of historical from Vendor A to Ninja Trader, and when I have a different set of historical data from Vendor B, do I have to delete the "historical data" on Ninja Trader before I upload the Vendor B historical data, or will it just overwrite them?

Thanks and Regards,
Ben.

[NinjaTrader_ChelseaB]

Hello lorien,

Yes, the trading hours should be time zone of the exchange.
Yes, CBOE US Index Futures ETH and CBOE US Index Futures RTH are (UTC-06:00) Central Time (US & Canada)

I typically recommend changing the PC timezone instead of the Tools -> Options -> Time zone, as there have been issues identified and corrected with having the internal time zone of NinjaTrader set to something different than the PC timezone. However, changing the internal time zone should be alright. Let us know if you find any issues with this.

Historical data is time stamped to your local PC time zone when its received from the data provider. Changing the time zone would make all of that data no longer have the correct time stamp so it must all be removed and re-downloaded to be time stamped with the newly selected time zone.

If you are importing data, yes, you will still have to remove and re-import it, and the data files would need to be in UTC.

When importing, the existing data will be overwritten.

[lorien]

Hi Chelsea,
Thanks for the clarification on the Time Zone for the templates as well as the setting of Time Zones via PC time zone and Ninja Trader internal time zone via Control-Center->Options

I'd like to get further clarification on the followings:
Chelsea> Historical data is time stamped to your local PC time zone when its received from the data provider. Changing the time zone would make all of that data no longer have the correct time stamp so it must all be removed and re-downloaded to be time stamped with the newly selected time zone.

Ben> What if instead of re-download the historical data to get the correct time-stamped data, can I just set my PC time zone to the "original time zone" that I downloaded the historical data? In that case, Ninja Trader Strategy Analyzer would use the PC time zone (internal time zone) with the correct time-stamped historical data, isn't it?

Chelsea> If you are importing data, yes, you will still have to remove and re-import it, and the data files would need to be in UTC.
Chelsea> When importing, the existing data will be overwritten.

Ben> If i don't remove the data, and re-import it, isn't it going to overwrite the data, i.e. I'll have the newly re-imported data and NOT merged data, correct? OR
if I don't remove the existing data, the resulting data will be merged?

Follow-up Questions:
1) If I've imported data when I was in CST time zone, then if I've moved to a different time zone, am I supposed to re-import the data?

2) what if I just set the time zone (via Control Center -> Options) to CST time zone from the "local time zone"?

3) Another issue that I noticed is that when I import some Continuum historical data which I downloaded from Ninja Trader, there are multiple Ticks with the same Time Stamps. Why does this happen? Please see attachment

Thanks,
Ben.

[NinjaTrader_ChelseaB]

Hello lorien,

If you are using the timezone of the data, then that's fine. The issue is if the local timezone doesn't match the data's stamped timezone.

The data will be overwritten for the dates of the data. Not sure what you mean by merge. We won't have two copies of the data an we won't pick and choose data from existing and import. It will just be the new imported data, as it is overwritten.

If the local timezone does not match the data's timezone the data will load incorrectly.

There are always ticks that happen at the same price in the same second. It means multiple trades have occurred at near the same time at the same price, which is typical as trades tend fill where the ask or bid price is and its highly likely multiple traders are placing trades at the same time. Some data providers choose to aggregate this and send one larger tick (which would technically not represent the true activity)

From post 4:
I typically recommend changing the PC timezone instead of the Tools -> Options -> Time zone, as there have been issues identified and corrected with having the internal time zone of NinjaTrader set to something different than the PC timezone. However, changing the internal time zone should be alright. Let us know if you find any issues with this.

Tell Ben they are welcome to join the NinjaTrader forum community!

[jeronymite]

Would it not make sense for all data to be timestamped with GMT (UTC, if you prefer) timestamps, and then all usage of the data adjusts to the local or PC timezone, as appropriate? It would certainly be a far more portable and robust approach. It would also overcome issues related to daylight saving time changes, as well as changes for people who travel across timezones.

Thanks.

[lorien]

Hi Chelsea,
I am Ben, lorien is my login/user name

To make sure I don't misunderstand anything, I've copied the questions I asked and try to match your response for clarification

-------------
>> Ben> What if instead of re-download the historical data to get the correct time-stamped data, can I just set my PC time zone to the "original time zone" that I downloaded the historical data? In that case, Ninja Trader Strategy Analyzer would use the PC time zone (internal time zone) with the correct time-stamped historical data, isn't it?

> Chelsea> If you are using the timezone of the data, then that's fine. The issue is if the local timezone doesn't match the data's stamped timezone.
> Chelsea> If the local timezone does not match the data's timezone the data will load incorrectly.

Ben> I've interpreted your replies into the following statements, please confirm if the followings are correct:
Assumption: When I'm in a time zone other than Central Time (CST), my PC local time will automatically set to the
time zone I'm in, e.g. "(UTC-10:00) Hawaii":

a) When I import the historical data, I have to select "Time Zone of Imported Data" with a Time zone that is identical to
my PC local time, for e.g.
i) By default, when I invoke Ninja Trader, the time zone it selects will be using the PC local time zone,
i.e. "(UTC-10:00) Hawaii"
ii) So, Tools -> Historical Data -> "Time Zone of Imported Data" == "(UTC-10:00) Hawaii"

b) When I import historical data, I have to select "Time Zone of Imported Data" and make sure the local time zone is
identical, for e.g.
i) Tools -> Historical Data -> "Time Zone of Imported Data" == "(UTC-06:00) Central Time (US & Canada)"
ii) Tools -> Options -> General -> Preferences -> Time zone == "(UTC-06:00) Central Time (US & Canada)" OR
per your recommendation from "post 4",
just set the PC local time to "(UTC-06:00) Central Time (US & Canada)" and let Ninja Trader set it accordingly
using PC time.

------------

To wrap up the Time Zone issues with Back Test, once the above settings are confirmed, then when I run Back Test, do I just select the default Trading Hours Templates when I trade the US Index Futures (ES), such as the followings?

- CBOE US Index Futures ETH : with (UTC-06:00) Central Time (US & Canada)
- CBOE US Index Futures RTH : with (UTC-06:00) Central Time (US & Canada)

------------
> Chelsea> The data will be overwritten for the dates of the data. Not sure what you mean by merge.

Ben> just curious, if the data will be overwritten by the newly imported data, then it should be OK if I don't remove the old data before importing the new data, correct?

What I meant by "merge" was that, if some of the old data has different time stamps than the new data, they will just be kept, and so, the resulting data in that particular date will have both new and old data(those that have different time stamps from those new data).

------------

> Chelsea > There are always ticks that happen at the same price in the same second. It means multiple trades have occurred at near the same time at the same price, which is typical as trades tend fill where the ask or bid price is and its highly likely multiple traders are placing trades at the same time.

Ben> I agree that there might be multiple ticks within a second, and there might be multiple trades per tick. However, I'm a little
surprised that there are so many ticks within a 1/1000 of a second with each tick having multiple trades.
Just to confirm that you're saying it is possible to have multiple, such as 10 ticks, during the same 1/1000 of a second (0.001 second)?

Thanks again for your patience,

--
Ben.

[NinjaTrader_ChelseaB]

Hello Ben,

I've heard there is a setting for having windows 10 automatically change the time zone when you travel 'Set time zone automatically'.
https://www.google.com/search?q=wind...+automatically

I don't really know much about how well that works. But that would cause the data to appear incorrectly on a chart if the time zone from the pc changes and the data is not reloaded. Personally I would advise against using Set time zone automatically and just leave it set to your primary location to avoid having to reload data in NinjaTrader.


Where you have asked

When I import the historical data, I have to select "Time Zone of Imported Data" with a Time zone that is identical to
my PC local time, for e.g.

This is not correct.

From the help guide on importing data:
"Select the time zone of the data you are importing (not the time zone you are importing to as all imported data will always be converted to local PC time). If you are importing data exported from NinjaTrader then this should be left as UTC because NinjaTrader exports are always done in the UTC time zone."
https://ninjatrader.com/support/help...gImportOptions

This is letting you know that the time zone should not be identical to your PC local time. This should be set to the time zone of the data you are importing, so that NinjaTrader can recognize it.

In that case, Ninja Trader Strategy Analyzer would use the PC time zone (internal time zone) with the correct time-stamped historical data, isn't it?

Yes, by default, NinjaTrader will use the time zone of the local PC unless you set this manually to another timezone in Tools -> Options -> General -> Time zone.

The Strategy Analyzer uses the historical data exactly the same way as a chart. The historical data, when downloaded from a connected data provider or when imported is time stamped to the local pc and then stored, ninjatrader calls this when needed and uses this create the charts or use for backtesting.


Where you have asked:

So, Tools -> Historical Data -> "Time Zone of Imported Data" == "(UTC-10:00) Hawaii"

I don't know where you have gotten your data text files from. What is the time zone of the data?


Where you have asked:

When I import historical data, I have to select "Time Zone of Imported Data" and make sure the local time zone is
identical, for e.g.
i) Tools -> Historical Data -> "Time Zone of Imported Data" == "(UTC-06:00) Central Time (US & Canada)"
ii) Tools -> Options -> General -> Preferences -> Time zone == "(UTC-06:00) Central Time (US & Canada)" OR
per your recommendation from "post 4"

Again, I don't know where you have gotten your data from. What time zone is the data text file in?


Where you have asked:

To wrap up the Time Zone issues with Back Test, once the above settings are confirmed, then when I run Back Test, do I just select the default Trading Hours Templates when I trade the US Index Futures (ES), such as the followings?

The Trading Hours are set for each instrument in the instrument settings. We recommend using <use instrument settings> unless you are specifically trying to use a different trading hours template that does not match the hours of the exchange. For example using a custom trading hours template that you have created yourself.

The ES Future uses CME US Index Futures ETH. This is traded on the Chicago Mercantile Exchange, and is not on the Chicago Board Options Exchange.

To view this click Tools -> Instruments -> ES -> select the ES future -> click edit. Note the Trading hours set for the Instrument. Do not make any changes here.
https://ninjatrader.com/support/help...GeneralSection
Then click Tools -> Trading Hours -> and look for the trading template that you found in the instrument settings.
https://ninjatrader.com/support/help...ngHourTemplate


Where you have asked:

just curious, if the data will be overwritten by the newly imported data, then it should be OK if I don't remove the old data before importing the new data, correct?

Yes, it should be OK if you don't remove the old data first. The data will be overwritten. Overwritten means the existing data will be deleted and the new data will in its place. Think of how you would overwrite a sentence in your reply.


Where you have asked:

What I meant by "merge" was that, if some of the old data has different time stamps than the new data, they will just be kept, and so, the resulting data in that particular date will have both new and old data(those that have different time stamps from those new data).

No, for the dates of the data, the data will not be merged. The data will instead be overwritten. This means existing data will be deleted and the new data will be in its place.


Where you have asked:

I agree that there might be multiple ticks within a second, and there might be multiple trades per tick. However, I'm a little
surprised that there are so many ticks within a 1/1000 of a second with each tick having multiple trades.
Just to confirm that you're saying it is possible to have multiple, such as 10 ticks, during the same 1/1000 of a second (0.001 second)

Yes.

This depends on the data provider, and depends on the data connect adapters, and depends on the trading volume, and could be much much higher than you are suggesting.

[lorien]

Hi Chelsea,
Thank you very much for the detailed replies and the reference links, now I've a good handle of how the Back Testing utilize Trading Hours, Historical Data and Time Zones, and how I should set/import them!

By the way, on Windows10 time zone setting,

Chelsea> I've heard there is a setting for having windows 10 automatically change the time zone when you travel 'Set time zone automatically'.
https://www.google.com/search?q=wind...+automatically

Yes, it does automatically set the time zone to wherever I am. However, I've found that one of my laptops does not do that haven't had time to debug that yet.

Regards and have a wonderful weekend,
Ben.