Resolve a mistery

[frankduc]

Hi,

I just found out why the volume doesnt match in particular between contract during rollover time.

It appears that the time and sale of NT and CME are wrong because of a glitch somewhere in the software.

Please to NT developer open a Time and sale from IB and one from NT for the emini sp500.

Notice that every trade where the price ending with an extra decimal doesnt appear in the time and sale of NT.

For exemple: 1454.5833 was a block trade of 254 contracts and it doesnt appear in NT time and sale. I suppose its because of the extra decimal.

Am i wrong? Thats the only reason i see, because there is a difference between the volume show in IB, NT and the CME total which give for yesterday 1794880 when NT time and sale sum only give 1205909.

For some reason NT doesnt receive all data sent by IB.

[NinjaTrader_AdamP]

frankduc,

I am not seeing the similar decimal dropping behavior here. Please see the screenshot attached.

You may want to make sure your settings are as they expect when you right click the ToS window in NinjaTrader > go to "Properties".

[frankduc]

As you can see in my screenshot there's some decimal in IB TS at 1455.4583 6 contracts. Sometimes they dont appear in NT TS. Is that the problem im not sure but there's a lost of data somewhere.

In the time and sale window of NT i see total volume at top 1061950 contracts at 11:54 but if i go in hist data mang in edit ES 12-12 minute sept 19 and i open in excel and sum the total volume i only get 794 508 contracts. Its like if some trade has been lost in the time and sale.

Why it does that?

By the way im connected with kinetick and it does the same with IB.

[NinjaTrader_AdamP]

frank,

Brian contacted me about a ticket you have open with him. I would like to clarify a few things.

The ES shouldn't be trading at decimals such as 1455.4583 since 1 tick is 0.25 unless its an OTC trade perhaps being reported. We are checking with IB now on that.

How are you comparing this? Are you comparing Kinetick data in NinjaTrader to IB data in TWS? Or are you comparing IB data in NinjaTrader to IB data in TWS?

For your volume reporting issue with Kinetick :

Daily Volume can be different than the summation of all the tick volumes in real-time. The reason is Daily Volume contains volume of Last trade types and all non-last-qualifying trades whereas adding tick volumes is just adding Last trade type volumes. Non-last-qualifying trades would not be reported real time.

[frankduc]

first there is many of these trades trading at decimals, i dont know if its usual but during the in-between contract from sept 12 to sept 21 there will be a few with good volume.

Second forget the connection is from IB or Kinetick it produce the same issue. Usually during those two weeks prior to the end of the contract there's missing volume.

According to IB TWS the volume match everyday with the CME total. If TWS say 2.6 millions at the end of the day than i download from NT the time and sale in excel and make the sum and the total equal IB and Globex. Except two weeks prior the end of the contract.

If its true that your are connected to the source which i think would be Globex for kinetick then there's a problem with Ninjatrader cause it doesnt make sense that everyday of the month i collect the data in Hist manager opening an excel sheet of the time and sale in edit and during those two weeks we lose data cause the sum of the time and sale dont match up with the CME globex volume.

I dont know if its Globex or NT but we are missing data during those two weeks i swear!!

The reason is Daily Volume contains volume of Last trade types and all non-last-qualifying trades whereas adding tick volumes is just adding Last trade type volumes.

If it was the case the volume wouldnt match all year long.

If there's 2.6 millions contract traded according to Globex it would be logical that the SUM of all contracts in TS is near equal, which is the case most of the time except those 2 weeks prior to the end of the contract.

Anyway at 13:02 the NT T&S window say there's 1207131 contracts traded and the SUM of all contracts in hist mang dont match with 924801 contracts.

Both should be equal like it was the case 2 weeks ago.

[NinjaTrader_AdamP]

frankduc,

Bertrand is checking on the decimals so I will probably have him follow up with you when he gets back early tomorrow. He works out of Germany.

The volume for real-time data will not match the historical volume is basically what I am saying here, due to the Non-last-qualifying trades not being included in real time quotes. You might see a difference if you right click your chart and go to "reload all historical data".

Specifically this issue with the daily volume being reported differently is that it is used in the ToS window. It is reporting Non-last-qualifying trades, however the real time ticks NT is saving as historical data does not report these. That is specifically with Kinetick however, I am not sure if its the same with IB.

So to clarify what you are saying here, and please correct me if I am wrong, is that there is missing volume around the roll over dates on both Kinetick and IB connections?

I'll have to investigate the roll-over times a bit further to see what you are referring to here, as you were mentioning the roll-over dates are where the volumes aren't matching.

[frankduc]

So to clarify what you are saying here, and please correct me if I am wrong, is that there is missing volume around the roll over dates on both Kinetick and IB connections?

YES!

Actually iv noticed the problem more than a year ago. If you look at my past threads you will see that i have reported this issue quite a few time in the past.

Yesterday there was 1794880 ES12-12 but if you gather the volume in the Tos it gives only 1196449. Monday there was 1573163 contracts but if you sum up it gives only 1148701. The volume is taken from 16:15 to 16:15. There is about 600 000 contracts deficit each day since wednesday sept 12th 2012. Its the same issue every rollover since a year in a half. I could by your Non-last-qualifying trades explanation if it was a few thousand lost contracts, but 600k??? come one!

I think its related to the migration from NT 6.0 version to 7.0. Because i never had this problem before.

I asked kinetick to provide me with a free trial cause i wanted to compare with IB and it does the same. The problem still there and repeats itself. So i thought you were lying about your data and the source beeing actually IB, but seems not, its probably CME globex. So if your data are reliable the only explanation is that Ninjatrader version 7.0 disregard some of the trades during that period!

My theory was that maybe NT disregard trades with decimals during the rollover time and is losing data resulting in less volume for the new contracts.

After all if there is many decimals with large block trade of a thousand for exemple it would explain why 600 000 contracts diseapper every day between the thursday of the second week and the friday of the third week.

Im not gonna let go that issue. If we cant find the problem ill come back to hunt you next rollover in december ;-)

[koganam]

So to clarify what you are saying here, and please correct me if I am wrong, is that there is missing volume around the roll over dates on both Kinetick and IB connections?

YES!

Actually iv noticed the problem more than a year ago. If you look at my past threads you will see that i have reported this issue quite a few time in the past.

Yesterday there was 1794880 ES12-12 but if you gather the volume in the Tos it gives only 1196449. Monday there was 1573163 contracts but if you sum up it gives only 1148701. The volume is taken from 16:15 to 16:15. There is about 600 000 contracts deficit each day since wednesday sept 12th 2012. Its the same issue every rollover since a year in a half. I could by your Non-last-qualifying trades explanation if it was a few thousand lost contracts, but 600k??? come one!

I think its related to the migration from NT 6.0 version to 7.0. Because i never had this problem before.

I asked kinetick to provide me with a free trial cause i wanted to compare with IB and it does the same. The problem still there and repeats itself. So i thought you were lying about your data and the source beeing actually IB, but seems not, its probably CME globex. So if your data are reliable the only explanation is that Ninjatrader version 7.0 disregard some of the trades during that period!

My theory was that maybe NT disregard trades with decimals during the rollover time and is losing data resulting in less volume for the new contracts.

After all if there is many decimals with large block trade of a thousand for exemple it would explain why 600 000 contracts diseapper every day between the thursday of the second week and the friday of the third week.

Im not gonna let go that issue. If we cant find the problem ill come back to hunt you next rollover in december ;-)

I would suggest that you check the CME volume for each of the 2 individual contracts that are being traded at rollover. It seems that it is possible that you are seeing an aggregate volume report for /ES, which would actually represent both contracts at rollover, whereas NT is reporting only the specific front month contract.

[frankduc]

I thought about that too but it doesnt seem to be the case, here's why.

Look at the CME daily bulletin here:

http://www.cmegroup.com/daily_bulletin/Section11_Equity_And_Index_Futures_2012181.pdf

for emini es dec 2012 it says 1794880 contracts were traded for the day.

Now if you go in historical manager edit for the 18th sept and you add up volume from 17th 16:15 to 18th 16:15 which is the actual hour took by CME you will end with a total of 1196362 contracts. See file in attachment at the end volume row.

Usually outside of the rollover period i get about the same number between CME daily bulletin and the volume add up in hist manager.

[NinjaTrader_Bertrand]

Thanks for the added details and report Frank, we're awaiting clarification here - I will hopefully get back to you soon.

[NinjaTrader_Bertrand]

Frank, we tested on our end with the API snapshot feed from IB TWS in different applications and could verify NT's Time and Sales output matches the stream we connected to, the block trades are reported correctly as well as IB sends them via this stream - their price would be rounded by NT to fill within the instruments tick-size.

I understand that NT TWS can provide an alternative, less filtered feed as well and my best guess is that TWS is using this feed for internal Time and Sales window, or their window would use a different snapshot feed timing explaining the slight differences seen.

Accessing the less filtered IB feed via API / NT is on our feedback list in product management and I've made sure to add your votes in as well to enhance this area.

[frankduc]

Thanks for the reply.

I'm not sure if i understand what you said.

Do you acknowledge that there is a difference during that period between the volume in NT time and sales and the actual CME globex daily volume?

Cause even if i use kinetick to backfill that period i still get a deficit in the ending daily volume. I asked a connection to kinetick to test that issue and got the same result. How do you explain that?

I can see there's a filtering done by TWS, cause last week i noticed in their time and sale window that the cumulative volume was inferior to the volume shown in my TWS trading window.
What i dont get is why i cant backfill with kinetick the missing data (deficit, gap between time and sale and CME daily volume) ?

BTW if you are planning to resolve that issue when do you think a new NT version will come out?

The situation is that im paying a programmer to code an indicator and i need really good data. I was planning to eventualy buy the licence but if i cant get those data on NT ill have to find another software.

Frank
ty

[frankduc]

I just got an answer from CME and thought to share but im not sure i get what they mean nor if its the right answer.

Here is what the technical team came up with:

Tag 1020 (traded volume) for the last trade of ESZ2 has a value of 1794480 which matches the bulletin. Here is the FIX/FAST market data message:

8=FIXT.1.1^A56=007^A9=XXX^A34=5524895^A35=X^A49=CM E^A52=20120918-20:14:59.959^A75=20120918^A268=2^A1128=9^A22=8^A48 =10113^A83=6317195^A269=2^A270=145275^A271=1^A273= 201459000^A274=2^A279=0^A451=-125^A1020=1794880^A5797=2^A5799=1^A22=8^A48=10113^ A83=6317196^A269=0^A270=145275^A271=198^A273=20145 9000^A279=1^A336=2^A346=32^A1023=1^A10=0^A

Summing up the trade size from the Time and Sales for ESZ2 will not give you the correct Trade Volume unless you include the trade size for spreads that include ESZ2 as a leg. So in saying all that, you would generate a larger volume during role over period, due to traders that might be rolling positions or getting out. Hopefully this has answered your question.


Regards....

[NinjaTrader_Bertrand]

Frank, thanks for posting that update - I think this confirms what Adam posted here under #6 that it depends which volume types are reported in the accessed feed and that there could be differences comparing varying sources - the most accurate source I would of course deem the exchange itself.

For the IB connection - we have providing access to the less filtered feed on our suggestion list for consideration, for now the snapshot feed is used that updates in 250ms intervals per IB. What we access from them matches what other platforms would report, so this is not categorized as a 'bug' type report but potential enhancement to our platform in the future.

[frankduc]

Oh well thanks for trying.

I will have to ask datamine CME how much it could cost me to get their data.