Data Void - CQG - CME FX and Index Futures

[DayTraderinAZ]

I am missing CME Index Futures and CME FX Futures from the open yesterdat (Sunday Afternoon 5 PM Central) to 5 AM Central this morning.

I haven't changed my charts in weeks, but I've verified that all the data series are set to ETH.

I've reloaded historical data and re-downloaded the data, and I still have the data void.

I've restarted NinjaTrader several times. Still missing the data.

HELP!!!!!!!!!!!!!!!!!!!!!

[NinjaTrader_Joydeep]

Hello DayTraderinAZ,
Welcome to the forum and I am happy to assist you.

Unfortunately our CQG server was unable to save historical data from yesterday night and thus you are witnessing the missing ticks. This is however affected only select instruments.

Please refer to this post which further discusses the matter.

http://www.ninjatrader.com/support/forum/showthread.php?p=304612#post304612

[DayTraderinAZ]

Simple. Restore the data.

If you can't provide dependable data, I'll have to look at going back to TradeStation. I need to be able to trust your company.

Also, why wouldn't you verify the data is being recorded as soon on the Globex session opens on Sunday?????

Total Bush League.

[NinjaTrader_Brett]

It is unfortunate that this occurred and I apologize for the inconvenience. I can say that this is the exception to the rule when this happens.

NinjaTrader is dynamic in that we allow multiple connections, I do not suggest you do this as I mention this is the exception to the rule. However I did want to give you all available options.

You do have the option to use another data feed service (Or backup data feed service) for historical data that combs and checks their data for consistency and connect to this at the same time as your broker connection, an example of this type of service is www.kinetick.com.

Otherwise for the CQG historical data servers will be checked and if data can be restored in this weekends maintenance the data would be restored.

-Brett

[DayTraderinAZ]

Was the problem solely on CQG's end, or was it on the servers that NinjaTrader uses to record the data?

I'm really hoping you are able to restore the data.

Is there anyway for me to use a 2nd data feed just to restore the missing data that wouldn't cost me any money?

[NinjaTrader_Brett]

Hello,

No information at this time as to the exact cause however we found our NinjaTrader historical data servers not recording data this morning and we had to take corrective action.

I do not know of any method that would restore the missing data at no cost currently as you would need to get the data from someone and I know of no providers that provide free intraday historical data for futures.

-Brett

[DayTraderinAZ]

Why wouldn't NinjaTrader's servers be monitored 24/7 for any issues?

You should at least check your servers at the start of each Sunday session to make sure there wasn't an issue over the weekend.

I can see that NinjaTrader has had this problem before, so to me, it doesn't seem like you guys have implemented the proper corrective action.

In this case, if NinjaTrader had verified the servers were not recording at the ETH open on Sunday afternoon, we probably would have only missed 1 or 2 hours of data instead of 12 hours.

Can you explain to me and all the NinjaTrader community, why you guys would not verify your servers are functioning correctly at the start of each session?

[NinjaTrader_Brett]

Hello,

Thanks for the feedback I will insure it is sent to the server monitoring team.

If you want data servers that are monitored 24/7 and are combed for historical data accuracy, you would need to subscribe to a premium historical data service. The historical data provided to you is provided as a value added service by NinjaTrader at no cost to you. Though we do everything we can to make sure the data is accurate we do not comb or monitor the data consistency.

For this you would need a premium historical data service such as www.kinetick.com

In the rare case of a server issue we would look into restoring any missing data if possible during the next server maintenance period.


-Brett

[rt-trader]

OK,

Heres the thing I dont get about this;

From NT Help Files;

NinjaTrader Converts Real-Time Data into Historical Data

NinjaTrader stores real-time incoming tick data to your local PC if you have a Chart or Market Analyzer (must have an indicator column added) window open. This data can then be used as historical data. For example, if you open a chart and let it run all day long, the data collected today, will be available as historical data when you open the same chart tomorrow.

and later;

To minimize the need to load data and to speed up chart load times, NinjaTrader maintains an internal cache of your prior accessed data. When data is in this cache, NinjaTrader will use it to populate your charts instead of loading from your data provider.

So, if live data was incoming during to my charts while NT Servers took a long weekend, why on NT restart is the data, which supposedly was recorded because a chart was open, no longer available for loading?

Is this 'feature' overridden by the setting in Tools>Options>Data Get data from server?

Also, if to avoid this again in the future what is the performance overhead if in the same tab, Save chart data as historical is ticked? Does this option ensure each incoming tick is written to NT Database, or does it 'flush' say every hour (or whatever)?

Lastly, when rebuilding the data on your servers, please dont overlook the Australian market.

Thanks

[NinjaTrader_Brett]

Hello,

Thanks for the feedback also the cache is maintained based on the data load rules. In normal operation only the current day historical data would be reloaded instead the entire data set. This cuts loading time significantly and works based on these data loading rules:

http://www.ninjatrader.com/support/helpGuides/nt7/historical_data.htm

When does NinjaTrader load historical data?