CL data loaded repeatedly from IB server

[Ralph]

Thanks for confirmation Harry. Gives me some confidence that this is not an issue specific to my personal setup. My experience is a little different than yours: I use a non adjusted merge policy and not all contracts of the crude-chain are affected, appr. the half in my case. Seems to be something special with some of the contracts only.

So, let's try find the reason for that issue with this thread.

Regards
Ralph

[Harry]

Example: Loading data for CL into my standard workspace

Data loading marathon for CL for my default workspace. This took 2 min 40 sec for checking. No data was loaded, because data was already complete!

I counted 20 data requests that were processed, the first two of them were for intraday data (minute) the following 18 requests for daily data. Whenever daily data was asked for an expired contract, NT asked this for the two days prior to rollover date, so for some reason it did not know that this data was already complete. When daily data was asked for the front month, the request was made for the last working day (Friday). I counted 12 identical requests for daily data for May 14 for the contract CL 06-10. Out of these 9 were triggered by pivot indicators in DailyBars mode, as each of these requested daily data. Summary of data requested:

CL 06-09 -> CL 08-09 -> CL 10-09 -> CL 11-09 -> CL 1-10 -> CL 5-10 -> CL 6-10 (2 times) -> CL 5-10 -> CL 6-10 (9 times triggered by pivots or similar). It is a mystery, why certain contracts were requested for updating and others were not.

This is the trace file, which shows the details of the no data loading connection test. I have left out the 2 minute data and the 9 last identical requests triggered by the pivots to focus on the problem.

2010-05-15 20:50:15:109 (Interactive Brokers) Data.Bars.GetBarsNow: instrument='CL 06-09' from='2009-05-15' to='2009-05-17' period=Daily splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Metals / Energy ETH'
2010-05-15 20:50:15:109 (Interactive Brokers) IB.HistoricalDataRequest.Send: last=17.05.2009 instrument='CL 06-09' from='2009-05-15' to='2009-05-17' period=Daily splitAdjusted=False dividendAdjusted=False bars=0
2010-05-15 20:50:15:109 (Interactive Brokers) IB.HistoricalDataRequest.Send: request='CL;FUT;200906;0;;;NYMEX;;USD;;20090518 00:00:00;1 day;11 M;0;TRADES'
2010-05-15 20:50:15:500 (Interactive Brokers) IB.HistoricalDataRequest.Process: itemCount=149 instrument='CL 06-09' from='2009-05-15' to='2009-05-17' period=Daily splitAdjusted=False dividendAdjusted=False bars=0
2010-05-15 20:50:16:015 (Interactive Brokers) Data.Bars.GetBarsNow: instrument='CL 08-09' from='2009-07-17' to='2009-07-19' period=Daily splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Metals / Energy ETH'
2010-05-15 20:50:17:125 (Interactive Brokers) IB.HistoricalDataRequest.Send: last=19.07.2009 instrument='CL 08-09' from='2009-07-17' to='2009-07-19' period=Daily splitAdjusted=False dividendAdjusted=False bars=0
2010-05-15 20:50:17:125 (Interactive Brokers) IB.HistoricalDataRequest.Send: request='CL;FUT;200908;0;;;NYMEX;;USD;;20090720 00:00:00;1 day;11 M;0;TRADES'
2010-05-15 20:50:17:515 (Interactive Brokers) IB.HistoricalDataRequest.Process: itemCount=186 instrument='CL 08-09' from='2009-07-17' to='2009-07-19' period=Daily splitAdjusted=False dividendAdjusted=False bars=0
2010-05-15 20:50:17:531 (Interactive Brokers) Data.Bars.GetBarsNow: instrument='CL 10-09' from='2009-09-18' to='2009-09-20' period=Daily splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Metals / Energy ETH'
2010-05-15 20:50:19:140 (Interactive Brokers) IB.HistoricalDataRequest.Send: last=20.09.2009 instrument='CL 10-09' from='2009-09-18' to='2009-09-20' period=Daily splitAdjusted=False dividendAdjusted=False bars=0
2010-05-15 20:50:19:140 (Interactive Brokers) IB.HistoricalDataRequest.Send: request='CL;FUT;200910;0;;;NYMEX;;USD;;20090921 00:00:00;1 day;11 M;0;TRADES'
2010-05-15 20:50:19:515 (Interactive Brokers) IB.HistoricalDataRequest.Process: itemCount=218 instrument='CL 10-09' from='2009-09-18' to='2009-09-20' period=Daily splitAdjusted=False dividendAdjusted=False bars=0
2010-05-15 20:50:19:531 (Interactive Brokers) Data.Bars.GetBarsNow: instrument='CL 11-09' from='2009-10-16' to='2009-10-18' period=Daily splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Metals / Energy ETH'
2010-05-15 20:50:21:156 (Interactive Brokers) IB.HistoricalDataRequest.Send: last=18.10.2009 instrument='CL 11-09' from='2009-10-16' to='2009-10-18' period=Daily splitAdjusted=False dividendAdjusted=False bars=0
2010-05-15 20:50:21:156 (Interactive Brokers) IB.HistoricalDataRequest.Send: request='CL;FUT;200911;0;;;NYMEX;;USD;;20091019 00:00:00;1 day;11 M;0;TRADES'
2010-05-15 20:50:21:687 (Interactive Brokers) IB.HistoricalDataRequest.Process: itemCount=222 instrument='CL 11-09' from='2009-10-16' to='2009-10-18' period=Daily splitAdjusted=False dividendAdjusted=False bars=0
2010-05-15 20:50:21:703 (Interactive Brokers) Data.Bars.GetBarsNow: instrument='CL 01-10' from='2009-12-11' to='2009-12-13' period=Daily splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Metals / Energy ETH'
2010-05-15 20:50:23:171 (Interactive Brokers) IB.HistoricalDataRequest.Send: last=13.12.2009 instrument='CL 01-10' from='2009-12-11' to='2009-12-13' period=Daily splitAdjusted=False dividendAdjusted=False bars=0
2010-05-15 20:50:23:171 (Interactive Brokers) IB.HistoricalDataRequest.Send: request='CL;FUT;201001;0;;;NYMEX;;USD;;20091214 00:00:00;1 day;11 M;0;TRADES'
2010-05-15 20:50:23:703 (Interactive Brokers) IB.HistoricalDataRequest.Process: itemCount=230 instrument='CL 01-10' from='2009-12-11' to='2009-12-13' period=Daily splitAdjusted=False dividendAdjusted=False bars=0
2010-05-15 20:50:23:734 (Interactive Brokers) Data.Bars.GetBarsNow: instrument='CL 05-10' from='2010-04-16' to='2010-04-18' period=Daily splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Metals / Energy ETH'
2010-05-15 20:50:25:187 (Interactive Brokers) IB.HistoricalDataRequest.Send: last=18.04.2010 instrument='CL 05-10' from='2010-04-16' to='2010-04-18' period=Daily splitAdjusted=False dividendAdjusted=False bars=0
2010-05-15 20:50:25:187 (Interactive Brokers) IB.HistoricalDataRequest.Send: request='CL;FUT;201005;0;;;NYMEX;;USD;;20100419 00:00:00;1 day;11 M;0;TRADES'
2010-05-15 20:50:25:718 (Interactive Brokers) IB.HistoricalDataRequest.Process: itemCount=219 instrument='CL 05-10' from='2010-04-16' to='2010-04-18' period=Daily splitAdjusted=False dividendAdjusted=False bars=0
2010-05-15 20:50:25:750 (Interactive Brokers) Data.Bars.GetBarsNow: instrument='CL 06-10' from='2010-05-14' to='2010-05-14' period=Daily splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Metals / Energy ETH'
2010-05-15 20:50:27:203 (Interactive Brokers) IB.HistoricalDataRequest.Send: last=14.05.2010 instrument='CL 06-10' from='2010-05-14' to='2010-05-14' period=Daily splitAdjusted=False dividendAdjusted=False bars=0
2010-05-15 20:50:27:203 (Interactive Brokers) IB.HistoricalDataRequest.Send: request='CL;FUT;201006;0;;;NYMEX;;USD;;20100515 00:00:00;1 day;11 M;0;TRADES'
2010-05-15 20:50:27:656 (Interactive Brokers) IB.HistoricalDataRequest.Process: itemCount=232 instrument='CL 06-10' from='2010-05-14' to='2010-05-14' period=Daily splitAdjusted=False dividendAdjusted=False bars=0
2010-05-15 20:50:27:687 (Interactive Brokers) Data.Bars.GetBarsNow: instrument='CL 06-10' from='2010-05-15' to='2010-05-15' period=Daily splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Metals / Energy ETH'
2010-05-15 20:50:29:218 (Interactive Brokers) IB.HistoricalDataRequest.Send: last=15.05.2010 instrument='CL 06-10' from='2010-05-15' to='2010-05-15' period=Daily splitAdjusted=False dividendAdjusted=False bars=0
2010-05-15 20:50:29:218 (Interactive Brokers) IB.HistoricalDataRequest.Send: request='CL;FUT;201006;0;;;NYMEX;;USD;;;1 day;11 M;0;TRADES'
2010-05-15 20:50:29:671 (Interactive Brokers) IB.HistoricalDataRequest.Process: itemCount=232 instrument='CL 06-10' from='2010-05-15' to='2010-05-15' period=Daily splitAdjusted=False dividendAdjusted=False bars=0
2010-05-15 20:50:31:937 (Interactive Brokers) Data.Bars.GetBarsNow: instrument='CL 05-10' from='2010-04-16' to='2010-04-18' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Metals / Energy ETH'
2010-05-15 20:50:31:937 (Interactive Brokers) IB.HistoricalDataRequest.Send: last=18.04.2010 instrument='CL 05-10' from='2010-04-16' to='2010-04-18' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2010-05-15 20:50:31:937 (Interactive Brokers) IB.HistoricalDataRequest.Send: request='CL;FUT;201005;0;;;NYMEX;;USD;;20100419 00:00:00;1 min;5 D;0;TRADES'
2010-05-15 20:50:33:750 (Interactive Brokers) IB.HistoricalDataRequest.Process: itemCount=6971 instrument='CL 05-10' from='2010-04-16' to='2010-04-18' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0

[Ralph]

Interesting Harry,

looks similar for my installation, although not exactly the same. Here are the CL contracts always reloded when I connect:

06-09 -> 08-09 -> 10-09 -> 11-09 -> 02-10 -> 06-10

I am wondering if the issue has to do with the definition of the rollover dates. You could compare yours with the rollover dates I use, if you like to (some posts earlier).

Regards
Ralph

[NinjaTrader_JoshP]

Please note I have forwarded to development kdoren's case. It may be the same case you guys are facing as well. Thank you.

[Harry]

Hi Josh,

had a very special case today: Opening NT and connecting my default workspace (5 connected charts for ES and one tick chart) to IB.

Loading ES: one data request for minute data, one for daily data -> no problem

Toggling to FDAX: Two data requests for minute, two data requests for daily charts -> no problem (made separate requests for yesterday and today)

Toggling to CL: NinjaTrader fired off exactly 58 (!!!) data requests for the same simple workspace, causing numerous data throttling popups. I enclose the trace file, so that you can confront the developpers with their deeds.

[Harry]

Windows Blue Screen

Had this event again, 20 noisy popups, in the end this produced a PC crash (windows blue screen, my second this year), trace file does not show anything related to the crash.

Howver, I went more into details now: All other futures contracts do not show this behavior. I also tried BC, which has the same funny symbol to fix that confusion between contract and expiry month. BC (symbol COIL||||1) loaded orderly.

So it might be related to the fact that I changed the default rollover date for CL 07-10 from May 19 to May 18, as the last trading day is too late to roll CL. Also the offsets of CL did not update after instrument reset, as trace file shows UpdateRolloverOffsets: .... offset=0

[NinjaTrader_JoshP]

We have applied a fix for load requests on non-intraday charts for B16. Unfortunately the fix does not cover CL and is still a WIP.

[Harry]

No problem for me, this is still a Beta.

If it does not fire 58 data requests in the end, I am happy. Could you reproduce the problem? Why does it only happen with CL? BC has also that particular symbol mapping. Is it related to the rollover date setting as per today?

We have applied a fix for load requests on non-intraday charts for B16. Unfortunately the fix does not cover CL and is still a WIP.

[NinjaTrader_JoshP]

Harry,

Unfortunately I do not know the details as to why CL behaves the way it does right now. Development is still investigating. As such I cannot comment on rollovers being a factor or not.

[Harry]

Additional detail on the CL problem. My data base contains data that goes back to CL 12-08. When opening a chart, NT tries to load data for every single contract since 2008, and obviously all trials a rejected and generating popups.

[NinjaTrader_Christian]

Hi Harry !

The problem you reported (redundant reloads on daily charts) is now also fixed for CL.

Thanks for reporting,
Christian

[Harry]

Hi Christian,

good to know, waiting for the next release then.

Hi Harry !

The problem you reported (redundant reloads on daily charts) is now also fixed for CL.

Thanks for reporting,
Christian