OHLC real-time data differs from manually updated historical data

[Neekolas86]

I am using 64-bit NinjaTrader 7.0.1000.6 in simulation mode with CQG data and a 1500-tick OHLC chart.

My chart style parameters are:
Color for down bars = Transparent
Color for up bars = Transparent
I am using the BetterVolume3 indicator

There appears to be an issue with my OHLC bars. The bars compiled automatically throughout the trading session differ from when I manually reload all historical data (CTRL+Shift+R).

I have attached two pictures. One is taken before I manually reload historical data and one is taken after. Do you see how the default automatic-updating data differs from the data that I manually updated? Is there a solution to this problem?

Thanks,

Nick

[NinjaTrader_Matthew]

Hello,

Can you please test this on a chart without any custom indicators or bar types applied.

I would like to see if the actual values between real-time and historical are different, of if there is a difference in how your current indicators/bar types are being constructed in real-time vs historical.

I look forward to assisting you further.

[Neekolas86]

Thanks for the reply Matthew. I created a new basic chart around 12:00 pm without any custom indicators or bar types applied. I let the data accumulate until approximately 2:50 pm. I took a picture of the data before manually updating historical data and a picture afterward (attached). The data is basically identical besides some minor differences. Is this normal or should data match perfectly?

[NinjaTrader_Matthew]

Hello,

It is expected for the data to change slightly after reloading.

Please review our Help Guide article on 'How Bars are Built > Understanding why a chart can look different after reloading historical data from the server'

http://www.ninjatrader.com/support/helpGuides/nt7/index.html?how_bars_are_built.htm

To ensure you have the most accurate data as possible in real-time, please compare your local PC clock time vs the CQG time stamps by going to File--> New--> Time and Sales.

From the Time and Sales window, please right click--> select "Instrument" and load any instrument.


You will now see the Time and Sales window populate with trades which are time stamped from the CQG servers.


You will want to open your PC clock and compare this time down to the second. Is your PC ahead, behind or equal to the CQG time stamps in the time and sales? Even a discrepancy of 5 seconds or less can contribute to issues, especially in high volume periods.


More information about using the Time and Sales window can be found below:
http://www.ninjatrader.com/support/helpGuides/nt7/index.html?time__ sales.htm

[Neekolas86]

Thanks, that was very informative. I discovered that my Windows clock is slightly slower than incoming Time & Sales orders by almost a full second!

I already have Windows time synchronized to the Microsoft servers so what else can be done to fix the problem?

Nick

[NinjaTrader_Matthew]

Hello Nick,

We have found better luck using the time.nistv.gov servers, rather than the general Windows time server.

Please use the steps below to change your Time Zone and sync the PC clock, note to set the server to time.nist.gov.


You can sync your PC clock by double clicking on the clock in the lower right corner of your desktop. Press 'Change time zone...' Set the Time zone and press 'OK'.


Once you have done that, click on Internet Time tab set the server to time.nist.gov and then click Update. Your PC clock should now be updated.

[Gumby]

My PC clock is consistently 2-3 seconds ahead of (higher than) the CQG time stamps regardless of which server I synchronize with. I don't understand why this should matter anyway though. Since the CQG bars come with their own time stamps, why would my PC clock affect which bar a tick gets placed with? Doesn't your page about this (which you linked below) say that the issue should only happen when a data vendor doesn't time stamp their data? That's my first question. My second is is there anything else I can do to fix this issue? It happens multiple times every day and screws up the signals that my company's indicators provide to our clients.

[NinjaTrader_Brett]

The time stamp does not affect the chart reload, native time stamping ensures this does not matter and on a minute chart you should be able to confirm you see no issues with reload matching almost exact with the reload vs live.

However you are seeing the behavior because you are using a Tick based chart, with tick based charts time is not factored in at all when making the bars. The only thing that is factored is seeing the number of ticks go >= the tick bar size to create a new bar.

What this means is that with tick bars (Range and other tick based bars as well) it does matter where you start the chart as to what your chart will look like. This is the reason why we always start a new tick bar at the beginning of a new session so everyone's charts more or less looks the same and it doesn't matter what your days to load is set too.

In your case you most likely are starting on the same bar at the beginning of the session however what is going on here is the load time of the chart.

->Its 10:30:05 and you start your chart.
->NT makes request for historical data for 10::30:05 back to the days to load you request to be loaded.
->NT starts downloading the data.
->Data download complete at 10:30:08
->NT starts live data feed at 10:30:08.


You can see where we are going with this and that means that you could be missing a few ticks while the chart loads. Depending on how active the instrument is trading when you load the chart and how fast your internet connection and amount of data you are downloading will affect how many ticks are not factored into the current bar during the chart initial load. Which then slightly affects all the remaining bars after the fact.

This effect is normally not seen with any other chart type.

We do have on our suggestion list for our next major version to look into starting the live data stream first then caching the live data while the chart loads. Until that time workarounds to this is to start your chart early or leave it running throught the overnight session and to reduce the amount of historical data you will be loading at one time.

Let me know if any questions.

[Gumby]

Thanks for the in-depth response Brett. I'm sure it will be helpful for someone, however, I'm actually using 1 minute bars, not tick bars. This is why I can't figure out for the life of me why this is happening. And it's not just me; it's our clients too and they keep asking me about it, understandably. Do you have any idea what's going on and what can be done about it?

[NinjaTrader_Brett]

Hello,

No problem.

This only applies to tick bars, your screenshots where tick charts.

I would need new screenshots of what you are seeing on the minute bars unfortunately as I would not expect that.

-Brett

[Eintopf]

Hey Brett,

I saw this thread and I have a question:
I understand that a reloaded tick or volume chart looks not the
same with every candle like the live one,
but what could be the problem if I´m missing really some bars on a reloaded chart ?

I made a screnshot for example , the first one is from the live chart and the second one is
after reloading historical data , you can see from the high at 17.38 to the low at 17.40
there are missing about 15 one hundred volume bars in the reloaded chart......

I think this is not slightly different ...

thanks for support

gerrit

[NinjaTrader_Brett]

Hello,

Thanks for the forum post.

Who is your data feed provider, are you only connected to one feed or multiple? If multiple what connection order?

-Brett

[Eintopf]

Hey Brett,

my connection is Mirus / Zenfire... , only

greetings

[NinjaTrader_Brett]

Hello,

In this case the data should match, we did have a NinjaTrader Historical Data Server item we saw last week. We did some maintenance this weekend I would not expect the issue moving forward past today. Note: Previous days may not be corrected, I want to know if you see it on historical data from today moving forward.

Thanks.

-Brett

[Eintopf]

Hi,

okay I checked some examples on CL for today , my first impression was livestream and reloaded historical data seemed nearly matching ....

Do you know if it was just the last week this problem ?

gretings

gerrit