Bad data

[ScottB]

Hi,

A fellow trader and I both have live accounts at the same broker. When checking to see why there was a discrepancy in a custom indicator, we noticed that his bar volume was significantly higher than mine on the ES 0931 bar and that the high of the bar was different as well. The ES 0932 bar was also off but only on volume and only slightly.

How can this happen and what, if anything, can be done about it. This is disconcerting to say the least.

Best Regards,
Scott

[NinjaTrader_Matthew]

Scott,

The most common reason this will occur on minute bars would be that the PC clocks were out of sync when the real-time data was coming in.

Were you able to correct this by right clicking on the chart and select 'reload all historical data'? Do the bars then match?

[ScottB]

Matthew,

Thanks for getting back to me so quickly. Reloading the data didn't help and given that the two bars together had vastly different volumes would seem to eliminate the issue of the clock (we both keep ours synced to nist anyway).

If I reload data, I thought that would come from your servers anyway (or maybe Kinetick) and given it was historical data when we did the reload that seems truly odd. I also have a TradeStation account/data feed and the numbers there agree with my friend's.

I had the chart up at that time and he did not; is there any possibility that the erroneous data was cached somehow and regardless of it appearing it is being reloaded, it is, in fact, being reloaded from the cache and therefore never going to give a different answer? I even tried changing the number of days to load to a longer time frame on the data series itself, no luck.

As I said before, this is truly disconcerting so I would like to get some sort of definitive resolution.

Best Regards,
Scott

[NinjaTrader_Matthew]

Scott,

If you had the chart loading in real-time and he did not, you're seeing a discrepancy between the real-time and historical feed.

You and your partner should come up with the same exact value upon reloading the chart give you are connected to the exactly historical data servers- however there may be some data cached in your end that is preventing it from updating. To correct this, simply shut down NinjaTrader and navigate to Documents\NinjaTrader 7\db\ and delete your "Minute" and "Cache" folder.

Once these have been deleted, restart NinjaTrader and reload the data.

As long as you are getting the data from the same exact data source, it should match exactly. Please double check that you're not loading one from the NT server while your partner is loading from Kinetick, as it's entirely expected to see different values from different providers.

[ScottB]

Matthew,

That fixed things but leads to another question; how did that much data go missing in real time in the first place. The other data feeds agree on the volume of 27,502 and mine showed a little over 20,000 contracts and a different high for the bar. The high I can understand, missing even 1 tick would do that.

However, I was connected the entire time and so I am trying to determine how reliable my data feed is. If I can miss over 7,000 contracts traded in a 1 minute bar, what else might I be missing. Any thoughts on that or is it best to chalk it up to gremlins?

Best Regards,
Scott

[NinjaTrader_Matthew]

Scott,

There are several things that can cause this. I doubt it's your data provider as the historical data connection we record the data from comes from the same exact technology. The difference being our servers are co-located, while your feed is local to you. I agree 7000 contracts is a lot to miss, but I suspect many of those were aggregated into other bars. It could be that the clock was out of sync, your network/CPU was lagging, or your ISP was dropping packets - any number of these items can cause this sort of discrepancy.

I'd suggest revisiting our Help Guide article on 'Understanding why a chart can look different after reloading historical data from the server' which should help explain why this may occur:

http://www.ninjatrader.com/support/helpGuides/nt7/how_bars_are_built.htm

[ScottB]

Matthew,

Thanks for your help with this issue. I assumed the data connection was TCP and not UDP so I didn't think packets could get dropped but as you say there are a number of reasons that could have happened. The important thing is it is fixed now.

Best Regards,
Scott