Problem with Kinetick data

[NutFlush]

Hello,

today I started using Kinetick as my data feed for live data. Unfortunately there is something wrong, because some of the bars that are shown in realtime change when I reload the data for the chart.

I have been using TradeStation for some years and the chart I get from my TradeStation application in realtime is correct (inserted into screen shots).

Please see the 2 charts:
a) created live (based on Kinetick)
b) reloaded historical data (based on Kinetick

Could it be that some ticks are sorted into the wrong bar while they come in in realtime? Isn't there a time stamp included in each tick that tells NT into which 1-minute bar each tick belongs?

Maybe I did something wrong in the setup?

Thanks,
NutFlush

[NinjaTrader_ChristopherJ]

Hello,

Thank you for your note.


This can occur if your PC clock is out of Sync which would cause data to shift real time however correct itself after reloading based off the time stamp from the Kinetick servers.

Please close NinjaTrader and sync your PC clock by double clicking on the clock in the lower right corner of your desktop. Once you have done that, click on Internet Time tab-->Change Settings-->Select time.nist.gov from the drop down menu and then click Update.

The PC clock should now be updated and you may restart NinjaTrader.

Please let me know if the issue persists after this procedure.

[NutFlush]

This can occur if your PC clock is out of Sync which would cause data to shift real time however correct itself after reloading based off the time stamp from the Kinetick servers.

Please close NinjaTrader and sync your PC clock by double clicking on the clock in the lower right corner of your desktop. Once you have done that, click on Internet Time tab-->Change Settings-->Select time.nist.gov from the drop down menu and then click Update.

The PC clock should now be updated and you may restart NinjaTrader.

Hi Christopher,

Thanks for the explanation, but the "solution" is not what I expected: I don't want to use my PC clock for time stamping the ticks, because that opens up a potential for errors (as demonstrated).

Could you please clearify whether:
a) Kinetick doesn't send a time stamp with each tick
b) NT doesn't use the time stamp of the ticks but uses the system clock instead

Thanks,
NutFlush

[NinjaTrader_ChristopherJ]

1. Kinetick timestamps data natively.

2. Real time bars are built based off the local PC clock time as data is received from Kinetick therefore if the PC clock is out of sync bars may not include all data that is sent for that specific bar.

[NutFlush]

1. Kinetick timestamps data natively.

2. Real time bars are built based off the local PC clock time as data is received from Kinetick therefore if the PC clock is out of sync bars may not include all data that is sent for that specific bar.

Hi Christopher,

if Kinetick timestamps the data then why doesn't NT use that information for realtime bar building?

Even if my local PC clock is perfectly timed there are delays until the tick is actually transmitted, received and processed. It is therefore inevitable to have differences to the bars when retrieved as "historical".

So why doesn't NinjaTrader simply use the time stamp as provided by Kinetick?

Thanks,
NutFlush

[NinjaTrader_ChristopherJ]

Currently there is not an option to change the bars to be built based off the native time stamp.

When you see this behavior is it only on the 1st real time bar or all bars from that point forward?

[NutFlush]

Currently there is not an option to change the bars to be built based off the native time stamp.

- Is this true for any data feed or just Kinetick?
- Will this be introduced in future versions?
- Is there a way to access the time info that Kinetick sends from within a strategy?

When you see this behavior is it only on the 1st real time bar or all bars from that point forward?

It happens all the time. The bars in my screenshots were built long after the last "refresh".

Cheers,
NutFlush

[NutFlush]

One more question:

What happens if my PC time is behind for 3 seconds (as an example): Will the first ticks of the session be ignored for 3 seconds until the time on my PC matches/exceeds the session starting time (as defined in the session template of the symbol)?

Thanks,
NutFlush

[NinjaTrader_ChristopherJ]

You can view the data time that is being sent within a Time & Sales window which can be accessed by going to File-->New-->Time and Sales then to add an instrument right click within the window-->Instruments.

This data can then be compared to your local PC clock to verify if this is in sync with the data coming thru.

I am looking into the issue further and will follow up as soon as I gather additional information.

[NutFlush]

You can view the data time that is being sent within a Time & Sales window which can be accessed by going to File-->New-->Time and Sales then to add an instrument right click within the window-->Instruments.

This data can then be compared to your local PC clock to verify if this is in sync with the data coming thru.

I am looking into the issue further and will follow up as soon as I gather additional information.

Hi Christopher,

the method that you mentioned (Time & Sales window) is for manual inspection, but not accessible from a strategy, right?

I also just noticed another strange behaviour: I had a 1-minute bar chart open for SPY long before the session started. Then at 9:30 exchange time (25 minutes ago) the first tick came in and was put into the bar from yesterday's session. This pulled down the bar and also the bar's CLOSE. After reloading historical data the final bar from yesterday looked "normal" again (as it did before today's session started).

This is really annoying!

Ticks should go into bars based on the exchange time, not my PC time (which I synched as instructed an hour before the session started).

Thanks for looking into this further,
NutFlush

[NinjaTrader_Matthew]

NutFlush,

Please allow me to clarify as there was a bit of a misunderstanding here.

The data from Kinetick is time stamped from their servers. The PC clock time that is used is only a timezone conversion. As far as the minute/second bar time you see should be exactly what comes from Kinetick.

You can obtain the native bar time by simply using Time[0], which would only give you the minute interval, or you can open a market data stream in OnMarketData() and use the time down to the second:

		protected override void OnMarketData(MarketDataEventArgs e)
		{
			    if (e.MarketDataType == MarketDataType.Last) 
					Print(e.Time);
		}

With all of that said, the time you see on the charts should be exactly equal to the time that Kinetick sends with their data. This should in theory ensure that your historical and live bars will match up.

However, there are a few cases where we could foresee that this data could change. For example, since Kineticks data source actually has about 20 different trades that come through, they only report last-qualifying trades on the real-time stream. When you reload from the server, these non-last qualifying trades are contained in the bar.

Below i'm quoting some documentation we've received from the Kinetick backend when we have looked into similar reports before:

Examples of non-last-trade-qualifying trades that do not come through streaming data (only one that does make it through are Form T trades).

a = acquisition
b = bunched trade - average price
c = cash trade
d = distribution
e = automatic execution
f = intermarket sweep
g = bunched sold trade - opening/reopening trade detail
h = intraday trade detail
i = basket index on close transaction
j = rule 127 trade
k = rule 155 trade
l = sold last
n= next day
o = opened
p = prior reference price
r = seller
s = split trade
t = form t trade - pre/post market trade
u = extended hours trade - reported late or out of sequence
w = average price trade
y = yellow flagged regular trade
z = sold - out of sequence

Please let me know if you have any questions on this information and I'd be happy to help further. If you want to look into your particular trades that are being missed, please email me at support[at]ninjatrader[dot]com and I'll be happy to open an investigation on a particular incident.

[NutFlush]

NutFlush,

Please allow me to clarify as there was a bit of a misunderstanding here.

The data from Kinetick is time stamped from their servers. The PC clock time that is used is only a timezone conversion. As far as the minute/second bar time you see should be exactly what comes from Kinetick.

Hi Matthew,

thanks for chiming in! Unfortunately you now see me a bit puzzled by your clarification, so please allow me a few questions to get this straight for me:

1. When a new tick arrives from Kinetick it has a time stamp on it that goes down to at least the level of a second (or even fractions of a second)?

2. Does NT add the incoming tick to a 1-min candle based on the Kinetick time stamp?
a) same minute as current bar => change OHLCV of bar accordingly
b) different minute as current bar => define current bar as closed and put tick into new bar at a different minute

3. Why was I told that my PC clock needs to be correct then? For the time zone conversion it shouldn't matter if my PC is off by a few seconds, right?

4. If the answers to 1. & 2. are YES then how can you explain that the first tick of SPY today was put into yesterday's last bar (see above)?

5. Is there any reason why the tick data from Kinetick is filtered (they advertise that they DON'T filter )? But maybe there's a good reason. All I can see is that in my TradeStation chart the bar data is different than what I see on my NT chart - and when I refresh NT it then has the same data as TradeStation had right from the beginning.

I know these are a lot of questions, but I really need to understand what's happening here so I can react or program accordingly.

Thanks!!!
NutFlush

[NinjaTrader_Matthew]

Hi Matthew,

thanks for chiming in! Unfortunately you now see me a bit puzzled by your clarification, so please allow me a few questions to get this straight for me:

1. When a new tick arrives from Kinetick it has a time stamp on it that goes down to at least the level of a second (or even fractions of a second)?

The smallest time increment we currently support is a second. Adding sub-second support is an enhancement we're looking forward to in the next major release.

2. Does NT add the incoming tick to a 1-min candle based on the Kinetick time stamp?
a) same minute as current bar => change OHLCV of bar accordingly
b) different minute as current bar => define current bar as closed and put tick into new bar at a different minute

Yes, it's based on the Kinetick timestamp. And you're correct, different minute tick as current bar, that bar would then be closed and we'd build a new bar

3. Why was I told that my PC clock needs to be correct then? For the time zone conversion it shouldn't matter if my PC is off by a few seconds, right?

This was a misunderstanding. Your PC clock could be several seconds off and it would not affect a native provider like Kinetick. In the case of something that is locally timestamped, your PC time settings would be important

4. If the answers to 1. & 2. are YES then how can you explain that the first tick of SPY today was put into yesterday's last bar (see above)?

This would have been a form T trade, which is the pre/post market trades which do come through the real-time feed.

5. Is there any reason why the tick data from Kinetick is filtered (they advertise that they DON'T filter )? But maybe there's a good reason. All I can see is that in my TradeStation chart the bar data is different than what I see on my NT chart - and when I refresh NT it then has the same data as TradeStation had right from the beginning.

It's unfiltered in that the ticks are unpackaged/not sent in snapshots. It will provide you with all of the last trades (and the one non-last qualifying trade) directly as it comes from the exchange.

I know these are a lot of questions, but I really need to understand what's happening here so I can react or program accordingly.

Thanks!!!
NutFlush

I'm happy to help here as I know it's a very important concept to grasp. We do apologize for the misunderstanding from earlier and are taking steps to address this understanding internally.

[NutFlush]

Hi Matthew,

thanks for replying to all of my questions! Too bad (for you ) that they generated some new ones. But the number decreases!

This was a misunderstanding. Your PC clock could be several seconds off and it would not affect a native provider like Kinetick. In the case of something that is locally timestamped, your PC time settings would be important.

Is there anything that would be "locally timestamped" even if I use Kinetick? I suppose you are refering to datafeeds from other providers that don't timestamp?

4. If the answers to 1. & 2. are YES then how can you explain that the first tick of SPY today was put into yesterday's last bar (see above)?

This would have been a form T trade, which is the pre/post market trades which do come through the real-time feed.

Shouldn't those ticks be blocked by the session setting (timestamp prior to session start I would assume)? Even if they come through why are the ticks sorted into yesterday's bar (different date than timestamp)? Is there any way I can prevent this from happening?

With all of that said, the time you see on the charts should be exactly equal to the time that Kinetick sends with their data. This should in theory ensure that your historical and live bars will match up.

However, there are a few cases where we could foresee that this data could change. For example, since Kineticks data source actually has about 20 different trades that come through, they only report last-qualifying trades on the real-time stream. When you reload from the server, these non-last qualifying trades are contained in the bar.

Are you aware of other data providers where the bars don't change as much between live and historical? TradeStation obviously sends the data in a way that leads to virtually no changes when reloading them later, so it is definitely possible to do it.

Cheers,
NutFlush

[NinjaTrader_Matthew]

NutFlush,

Is there anything that would be "locally timestamped" even if I use Kinetick? I suppose you are refering to datafeeds from other providers that don't timestamp?

The "BarTimer" indicator uses your local PC clock, and is often where some confusion comes from in these cases. A native stamped data provider may see a bar close a few seconds before the bar timer indicator reaches 0 (or the other away around as well). Syncing the PC clock will help ensure the BarTime indicator is in sync with the data feed time.

Shouldn't those ticks be blocked by the session setting (timestamp prior to session start I would assume)? Even if they come through why are the ticks sorted into yesterday's bar (different date than timestamp)? Is there any way I can prevent this from happening?

No, this happens outside of the session template. It's the underlying core processing that the tick is received. If anything comes through on the real-time feed, it will be counted as a tick. Unless you're using the real-time tick filter, but that is only in response to price, and not time. I can see if it would be possible to use this user-case as an example of creating a time based tick filter as well, however I'm not sure what other affects this may have.

Are you aware of other data providers where the bars don't change as much between live and historical? TradeStation obviously sends the data in a way that leads to virtually no changes when reloading them later, so it is definitely possible to do it.

I'm not aware of how TradeStation handles these trades or another other programs. The Kinetick and IQFeed data feeds are going to work the same and we've seen reports from eSignal connections as well. The only other data provider you may consider would be BarChart.com