Wrong Tick Data

[joebshehata]

I attached some screenshots of an issue I experienced when testing with my tick data. This is CL ##-## (though the error is exactly the same when I pull up the contract month). The Bid bounces/alternates in a range of 50-60+ points all within the same second. It actually does this at random throughout the historical dataset (for that day). It usually lasts for about 15 minutes worth of historical data. The top price remains flat while the bottom price fluctuates.

How am I supposed to trust that the data I am testing with is accurate or even somewhat reliable when I see things like this?

My data provider is CQG. I know this can't be a CQG issue, because no one would use CQG if there data was this unreliable.

I've seen all kinds of weird issues. I've seen times where the Bid equals the Ask for minutes straight. I've seen times where the Last bounces between two prices for easily over 1000 bars all on the same second. I’ve also tried to export a moth of data (using the Historical Data Manager) 1/1/2011 – 1/31/2011. It will export part of the first day in February. How can all of this be?

You need to be made aware of all of these bugs in the processing of data. I’ve been quiet about these issues for some time because, for the most part, I’ve found workarounds. But at this point, I can’t trust the data I’m seeing anymore.

Please address my concerns. I like using NT, but it makes me nervous when I see the data displayed like this. Data is the most important thing in testing. It doesn’t matter how you code your Indicators or Strategies if you have “bad” data.

[NinjaTrader_Brett]

Hello,

Thanks for the forum post.

Is it only with the BID or ASK you see this? Do you see it with the last as well?

Do you only see it during certain times? Such as high news times?

Finally is it only on the ##-## contract or do you see it on the normal contract as well?

-Brett

[joebshehata]

Hello,

Thanks for the forum post.

Is it only with the BID or ASK you see this? Do you see it with the last as well?

Do you only see it during certain times? Such as high news times?

Finally is it only on the ##-## contract or do you see it on the normal contract as well?

-Brett

Thanks for the response Brett.

I've seen this with all data, be it the Bid, Ask or Last.

I've seen it all times of the trading day, even in the middle of the night/early morning trade.

It doesn't matter if its ##-## or normal contract. The good news is, my data for ##-## matches exactly the data for the normal contract.

[NinjaTrader_Brett]

Thanks for the post.

The ##-## is built with data from the actual contract therefor it is good and expected that they match.

Can you give me the time and date of the last time you saw something like this if possible in the current contract or previous contract back. I would need the time and date and chart type and I will compare it with the data on my side to ensure the data is the same.

Please take a screenshot of this same area if possible as well.

-Brett

[joebshehata]

I don't understand your question. I provided screenshots in my initial post that should satisfy your request. Are you asking for another screenshot?

[NinjaTrader_Brett]

Hello,

This will work I missed the screenshots. Thanks for that, I will check into it.

[NinjaTrader_Brett]

What time zone are you in and finally, do you have another example date where you see this? I'm reviewing this with development and need to know if this is an isolated case or if you see it in other areas as this is our first report on it.

Thanks.

[joebshehata]

I feel a little abandoned here. I haven't gotten a response. I've been doing more testing, and wouldn't you know it... I ran into more data problems. I have attached screenshots.

Look at the files that end in 'HISTORICAL'. Look closely at the time at the bottom of the chart. Now remember what the price chart looks like in the picture.

Open the files that end in 'REPLAY'. Do the same. You will notice that the Replay chart is one hour ahead if look at the time at the bottom and the price chart above it.

The trade you see is entered at 9:22 on the historical chart and 8:22 during replay. They both close the trade at 13:15 (the strategy exits all trades at that time of day), but you will see just how different they look because the replay data stays in the trade one hour longer. In order for it to look the same, it would have to exit the trade at 12:15.

This is an extremely serious issue for me. I don't know which set of data is "right" and which is off by an hour. If this issue (and the others I've raised) cannot be resolved, then I will be leaving NinjaTrader. The bad data is too serious to just let slide.

You wanted to know my time zone... I am in Central Standard Time.

P.S. I will have to continue this post because of the five file attachment limit.

[joebshehata]

Part 2.
I attached screenshots of the Historical Data Manager and a Price Chart. You can clearly see data on April 16, 2011, which was a Saturday!

(note: on the price chart, the hash mark right next to the big red candle on 4/17 is the data from 4/16)

You will notice that the price jumps to Sunday's opening price which is also incorrect. Look at the screenshot of Sunday 4/17 (Historical Data Manager). You will see that there is data before the open. I didn't grab a screenshot, but all of them are 110.22. Right at 5:00 PM (the real open) there is one entry of 110.22; then they are all "normal" after that.

Once again, I've seen these kinds of issues for months now. I could give you examples until I'm blue in the face. I need some answers here. I want to know who, once and for all, provides my data.

[NinjaTrader_Brett]

Hello,

Thank you for your patience while I checked into this for you.

We have isolated the time frames referenced in your post and during these times either the data recording failed or it recorded incorrect data. If we have a server outage or other issue with our historical data servers that record the data from CQG this unfortunately can happen and is why we do not recommend utilizing this free data for critical testing. The included free historical data is provided as a complimentary service, and as such we do not guarantee the data for complete data continuity. As you have stated CQG does offer a historical data service, however their data service is part of their CQG specific package which costs hundreds of a dollars to use each month. Alternately for clients in your situation that require cleaned and scrubbed data we recommend using a 3rd party market data service.

NinjaTrader supports multiple high quality 3rd party market data services but the least expensive service is from Kinetick. Using the Kinetick premium historical data service you can load their historical data inside of NinjaTrader and use it for all of your testing purposes. To do so you will need to disable historical data on your CQG connection in NinjaTrader, and then make a second connection to Kinetick. This will cause live data to come from CQG and historical data requests to be routed to Kinetick who does monitor their data for data accuracy and fixes any market data discrepancies. If you have any questions on Kinetick, please email sales at Kinetick dot com and they will be sure to get your questions answered in a timely manner. You can also find details on the Kinetick website.

www.kinetick.com

Thank you again for working with us on your issue and please let me know if I can be of further assistance.