Timestamp Issues With Kinetick Data - Live versus Historical

[traderUS]

Hello NT8 Support,

When analyzing ES 06-22 tick data from Kinetick, I have noticed some discrepancies in live data versus historical data and a possible error (possibly my own coding error).

In my export script, I write the OnMarketData MarketDataEventArgs 'e' elements to a text file for later analysis in Excel. When I write the data to a file, I use e.Time.ToString("MM/dd/yyyy HH:mm:ss:fffffff"), e.Price, e.Volume, e.Bid, and e.Ask.

Comparing live data to historical data, I noticed the timestamps are different between the live and historical data exports while the tick sequence and volume, price, bid, and ask are all equal. For a specific example, on April 8, 2022 at 12:11:54 (Pacific Time), I have a series of ticks that have different timestamps. Upon closer inspection, it appears that the sub-second values in the live data have a leading zero while the historical data pulled from Kinetick do not have this leading zero.

Attached is a picture showing the data (pdf of my excel file that I cannot upload per website attachment restriction). The left table contains the tick data recorded live while the right table contains the tick data pulled as historical data from Kinetick (both live and historical data are coming from Kinetick). I have highlighted the timestamps that match in green and the timestamps that don't match in red. Also, the column "Milliseconds" in both tables shows the milliseconds part of the timestamp. As stated earlier, it appears that the timestamps recorded live have an extra leading zero compared to the timestamps recorded with historical data.

Have you seen this issue before, and do you know what is going on? Is my time formatting of the MarketDataEventArgs 'e' correct (i.e. e.Time.ToString("MM/dd/yyyy HH:mm:ss:fffffff"))?

Thank you for your help, and please let me know if anything is unclear.


Edit on May 8, 2022:
To add, I've looked further into this. In live recorded tick data, I see plenty of timestamps with HH:mm:ss:0ff timestamps (millisecond begins with a zero), but this rarely occurs in historical tick data. Out of the 209,661 records of tick data that I recorded live from 11:24am PT to 1:00pm PT on April 8th, 2022, 27,166 records had millisecond timestamps beginning with 0 (e.g. 026, 094, 055). However, looking at the same time segment in the historical tick data, only 54 records had millisecond timestamps beginning with 0 (e.g. 026, 094, 055).

Statistically, assuming that the trades' timestamps are coming in randomly, I would expect 100 out of 1000 trades (or 10%) of trades to have a timestamp with the milliseconds component being between 000 and 099 (ranging from 000 to 999 for all timestamps). For my live tick data, 27,166 records with timestamps having milliseconds between 000 and 099 out of 209,661 total records was about 13% of the total. For the historical tick data, 54 records with timestamps having milliseconds between 000 and 099 out of 209,661 total records was less than 0.03% of the total.

This supports that there is something wrong with the timestamps when they come in live versus when they come in with historical data, or my formatting string e.Time.ToString("MM/dd/yyyy HH:mm:ss:fffffff") is adding a leading zero when it shouldn't.

Could you please help me with understanding what is happening here?

Thank you again for your time and help.

Best regards,
traderUS

[NinjaTrader_ChelseaB]

Hello traderUS,

I want to confirm the issue.

The issue is that historical tick data does not have sub millisecond granularity?

[traderUS]

Thanks for the response to this. No, the data has millisecond data. The problem is that live millisecond timestamps do not match the historically pulled millisecond timestamps. Kinetick live timestamps have 000 to 099 timestamps while the historically pulled timestamps virtually never have 000 to 099 millisecond timestamps.

[NinjaTrader_ChelseaB]

Hello traderUS,

Thanks for clarifying that.

It may come down to a difference of your PC clock verses the servers clock.

Is your PC Clock in sync with internet time to the second?

Compare your PC Clock with https://www.time.gov/

If you are not in sync:

• Shutdown NinjaTrader
• Navigate to the Windows Control Panel
• Select Clock and Region -> then click 'Set the time and date'
  (Or click 'Date and Time' if this is visible)
• Click on Internet Time tab
• In the Server: drop-down select time.nist.gov
• Click Update
• If the message that appears says successful your PC clock should now be updated
  If not, select a different server from the Server: drop-down and try again (repeat until one of the servers is successful)
• Restart NinjaTrader
• Connect to a data feed, then right-click the chart and select Reload All Historical Data

Below are a few links to 3rd party educational sites with steps to synchronize with internet time.
https://www.windowscentral.com/how-m...ers-windows-10
http://mintywhite.com/windows-7/7mai...t-time-how-to/
http://www.windows10update.com/2015/...t-time-server/

Then record some new ticks and compare with new historical data.

[traderUS]

Hello Chelsea,

I am not pulling the timestamps from my PC; I am pulling the timestamps from the tick data supplied by Kinetick and called by OnMarketData. The MarketDataEventArgs provides a timestamp. How would my pc alter the timestamp provided by Kinetick that is supposed to represent the timestamp of the trade?

Also, I compared the timespan against Continuum historical data, and the Continuum timestamps are not a problem at all. About 10% of Continuum's historically pulled tick data has timestamps with milliseconds between 000 and 099 (as would be expected statistically).

There is a problem with Kinetick's data where it doesn't send the leading zero of milliseconds when pulling the data historically. For example, if the live timestamp was 12:24:31:045 (HH:mm:ss:fff), Kinetick would report it as 12:24:31:45X when reporting it historically (where X is some extra number added on).

I hope this clears up the issue I'm describing.

Could you please let me know what's going on?

Thank you,
traderUS

[NinjaTrader_ChelseaB]

Hello traderUS,

All data is translated to your local computer time using your PC clock.

Where you have stated "I am not pulling the timestamps from my PC; I am pulling the timestamps from the tick data supplied by Kinetick and called by OnMarketData", this would be incorrect.

The time stamps are coming from your local PC. Having a clock that is out of sync will cause the bar timer to work improperly, orders to appear on the wrong bar, and other unexpected behavior, because data is time stamped by your local PC after it is received.

[traderUS]

Ok, let's put that PC timeclock issue to the side for a moment. Did you open the PDF attachment I posted in my first post? The live table on the left shows that there were timestamps exactly matching the table on the right with historically pulled data (at the top). The top of the table shows that the timestamps match, then begin to not match as you go further down the table, then match up again at the end, which all transpired all around less than 1 second later. When I pull the historical data from Continuum, I get no issues (historically pulled timestamps match live data). This tells me that the problem is Kinetick, not my PC clock or Continuum data. However, if you're right, then that means my PC clock is wrong and Continuum is also wrong.

I'm leaning more towards Kinetick having an issue when they send timestamps with less than 100 milliseconds in the millisecond component of the timestamp.

Additionally, I found another site seeming to have a similar problem with MultiCharts data. Maybe the issues are related (https://www.multicharts.com/discussi...ic.php?t=47611).

[traderUS]

The biggest signal to me that something is wrong with Kinetick's tick data timestamps is the fact that my historically pulled tick data has less than 0.03% of trades that occur from 0 to 99 milliseconds each second. that means that 99.7% of the trades occur from 100 to 999 milliseconds. That doesn't seem realistic nor reasonable, but when I pull Kinetick data live and Continuum live or historical, I see about 10% of trades that have timestamps occurring from 0 to 99 milliseconds as expected (out of 1000 milliseconds, I'd expect 100 or 10% to take place between 0 and 99).

[traderUS]

Hello Chelsea,

Using May 9, 2022 data around US market open time period, I compared Continuum's live and historical tick data for this time period, and then I compared Kinetick's live and historical tick data for this same time period. Continuum's trade timestamps from historically pulled data were almost exactly the same as the live recorded data. Also, descriptive statistics of the data were identical. Unfortunately, Kinetick's trade timestamps from historically pulled data were inconsistent anywhere that the millisecond component began with a 0 compared with the live recorded data. Descriptive statistics of the data were also completely different. From this analysis, I cannot trust Kinetick historical tick data and can only hope that the live data is correct.

Please help create a bug report for the Kinetick live data feed. I appreciate your help with this.

Best regards,
traderUS

[NinjaTrader_ChelseaB]

Hello traderUS,

I am looking into the issue. I appreciate your patience.

[JimBro]

Dear traderUS and Chelsea,

I am struggling with this same issue now. It is urgent to have someone with a high level of expertise from CQG and NT looking at this. If historical and live data don't match then the platform is junk. I cannot imagine that CQG and NT would be doing well if this was the case, so help is needed to sort this out.

I am looking at Kinetick, IQFeed (same as Kinetick) and CQG Continuum. I am finding that both have issues with regard to timestamps not matching. Moreover, I find that the number of ticks for CQG live data does not match the CQG historical data.

I am pulling the data by running a "GenerateTickFile" strategy from the chart that has a secondary data series that is a 1-tick series. When OnBarUpdate is called for the tick-series the strategy writes the tick data to the file with both a Unix timestamp (long integer number of milliseconds since 1/1/1970) and a DateTime timestamp. Both have millisecond resolution. These timestamps are coming from the market data provider and are not generated by my computer. CQG and Kinetic both have native timestamps so earlier assertions that the PC time affects this are incorrect. I am concerned if NT8 uses PC time to generate bars from market data. This would introduce massive errors. I am 99% sure this is not the case because I can look at trade timestamps and find the tick they happened on using the time stamp info.


Please look at the attached visualization that shows an analysis of the unix timestamp differences between CQG live and CQG historical data for 5/6/2022 RTH. This tool can show all 1.55 million rows of tick data in a single view. This visualization shows the timestamp differences in milliseconds between live and historical data. Ideally we want a straight line on zero. Clearly, we don't get that here.

Perhaps the biggest issue is that the live data has 7229 more ticks. So at some point the data gets out of sync and the timestamp deltas become huge.

So first I am wondering if I am doing something wrong. As traderUS says, a volume or tick bar based strategy will not work if the live and historical data don't match. Moreover, volume and tick bars have very low autocorrelation and are ideal for algorithmic trading compared to time based bars.

I look forward to figuring this out.

[NinjaTrader_ChelseaB]

Hello JimBro,

I am looking into the issue. I appreciate your patience.

[JimBro]

I've been looking at this some more today. I downloaded live data for today's ES 6-22 from the open up until around 10:00 PT. Then I used my GenerateTickFile to write the tick data in the chart to a file and compared it to the live data. It matched exactly because the data in the chart was still the live tick data. Next I deleted this data from the database by removing it in the historical data management tool. Then I downloaded it from CQG and the data was not the same. The timestamps were shifted. So the result is that the difference between historical and live ticks is a problem.

Next I wanted to see if the time zone conversations were somehow causing an issue. So I set my time zone in NT to central time and reloaded the data after restarting NT. Again the timestamps were different from the live data and from the first pull of historical data just a few hours earlier.

However, I did notice something in the most recent historical data. There seems to be something forcing the tick timestamps to have a resolution of 4ms. I see all ticks have a minimum increment of 4ms. Sometimes it is 8ms or 12ms, but always the tick timestamp is rounded to the nearest multiple of 4 ms. For example, I have time stamps of one tick at :015ms the next will is at :019ms and the next is at :027ms (2 x 4ms) and the next is at :031ms, you get the idea. Just to reiterate, none of these timestamps match the live data.

Where is this 4ms rounding of the timestamp coming from? Is it from CQG or from inside the NT database management? Maybe someone thought it doesn't make a difference. Probably true in time based bars, but this could be a problem with tick and volume bars.

[traderUS]

Thanks for your analysis and support on this topic, JimBro.


Hello Chelsea, any update on your end? Appreciate you looking into this.

[NinjaTrader_ChelseaB]

Hello traderUS,

We have the behavior reported to our development but we do not yet have a tracking ID or any further information at this time.

Once we have more information we will update this thread.

We appreciate your patience.