CL data loaded repeatedly from IB server

[Ralph]

Hi,

NT7.0.0.15 loads historical daily data for CL whenever I re-connect to the IB-server. The concerning data is stored in the local data base and therefore there is no need to do that again and again.

I opened one workspace only for CL. It contains 3 chart windows (Daily:CustomRange:2009-03-18; 120Minute,Days:150; 15Minute,Days:20). Contracts are merged without back-adjustment. Used session template is: Nymex Metals / Energy ETH.

When I connect to the IB server, then for some contracts some days are reloaded again every time, although they are stored in the data base already. This behavious seems to be CL-specific, these reloads do not occur for GC (for example).

Here is a snippet from the trace file. You can see that daily data for CL 08-09 is reloaded from 2009-07-17 to 2009-07-19. Notice that Jul 18/19 is the weekend:

2010-05-14 11:39:41:369 (InteractiveBrokers) Data.Bars.GetBarsNow: instrument='CL 08-09' from='2009-07-17' to='2009-07-19' period=Daily splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Metals / Energy ETH'
2010-05-14 11:39:42:751 (InteractiveBrokers) IB.HistoricalDataRequest.Send: last=19.07.2009 instrument='CL 08-09' from='2009-07-17' to='2009-07-19' period=Daily splitAdjusted=False dividendAdjusted=False bars=0
2010-05-14 11:39:42:751 (InteractiveBrokers) IB.HistoricalDataRequest.Send: request='CL;FUT;200908;0;;;NYMEX;;USD;;20090720 00:00:00;1 day;11 M;0;TRADES'
2010-05-14 11:39:43:281 (InteractiveBrokers) IB.HistoricalDataRequest.Process: itemCount=186 instrument='CL 08-09' from='2009-07-17' to='2009-07-19' period=Daily splitAdjusted=False dividendAdjusted=False bars=0

The trace file contains such messages also for other contract months, but not for all.

In addition I attached a snapshot of the Historical Data Manager for CL 08-09. You can see that the concerning data is available. However, If I reconnect to IB, then the same procedure happens again.

Regards
Ralph

[NinjaTrader_Jason]

Hello Ralph,

Unfortunately this is as expected. NinjaTrader will load data from your data feed as per the conditions listed at the link below.

http://www.ninjatrader-support.com/HelpGuideV6/WhenDoesNinjaTraderLoadHistoricalData.html

The link refers version 6.5, however the same conditions apply in version 7.

[Ralph]

Thanks for the link Jason, good starting point. Here is a snippet from the link content:


Examples of when NinjaTrader will fetch data if the database contains data from 1/3/08 to 1/6/08:

• 1) Chart of 1/3/08 to 1/7/08 -> load data
• 2) Chart of 1/3/08 to 1/6/08 -> load data
• 3) Chart of 1/3/08 to 1/5/08 -> use database data
• 4) Chart of 12/28/07 to 1/5/08 -> load data

In my case 3) applies which means NT should use the content of the database. In the snippet from the trace file you can see data is loaded from 2009-07-17 to 2009-07-19. In the attached database screenshot you can see that date is available untill 2009-07-21 for that contract.

... following the doc, database is expected to be used

Regards
Ralph

[NinjaTrader_Jason]

Hello Ralph,

What date is entered for 'End date' in the chart? If the current date is entered, historical data will be loaded from the connectivity provider.

"“Format Data Series” contains the current day"

[Ralph]

Jason, you are correct, 'end date' is today. That means, 2) applies (load data). And NT does that, it loads the data for the current contract (CL 06-10).

But why does it re-load partial data from some of the previous contracts? The concerning data series is a merged series of contracts and they are stored in the data base entirely.

Regards
Ralph

[NinjaTrader_Jason]

It will reload the historical data that is requested. Since you automatically merge the data, data from previous contracts are loaded (to merge the historical data into the current contract).

I will forward the suggestion to have NinjaTrader not reload historical data from the connectivity provider while the data is already present in the database.

[Ralph]

Well, I rather think there is something wrong, Jason.

Because:
1) Historical data is not re-loaded for every CL-contract in the merged chain.
2) I have similar setups for other commodities and there it works as expected (data is reloaded for the front month only).

Regards
Ralph

[NinjaTrader_Jason]

Do you perhaps request less historical data in these charts? Could it be the instrument is not rolled over as frequent as the CL, so all requested historical data comes from the front month?

Once the data requests contains the current date, historical data will be loaded from the connected connectivity provider.

[kdoren]

FYI there is a bug in NT7 which I have already reported:

Charts or strategies which use daily data (or higher) are always reloaded from the feed, even if the data is all in the history database.

[NinjaTrader_JoshP]

kdoren,

1. Start NT
2. Go to Historical Data Manager and delete all daily data for ES 06-10, ES 03-10, ES 12-09, ES 09-09, ES 06-09.
3. Connect
4. Open ES 06-10 daily chart for 365 days back (be sure to be using MergeBackAdjusted for your merge settings)
5. Note how long it takes to load the chart. Note in Control Center status how it cycles through all the old expiries downloading data and saving it.
6. Now close the chart and open the same chart up again
7. You will see it loads much much faster without ever going to the old expiries. It will flash you ES 06-10 really briefly to tell you it is pulling up the data.

The data is in fact being loaded from your local cache and not being accessed from the data provider.

[kdoren]

There is for sure a bug in NT7, I hope it got onto your list when I reported it before.

If you open any new chart with daily data, and select the dates to be all in history, (i.e. load 30 days ending March 1, after checking to see that they are in history) it will load the data from the feed even though it should not. Same thing with strategies. This is very easy to verify, I can see it in the IQFeed logs but it also loads if the connection is IB.

Minute-based charts work correctly, the bug affects only daily (and higher) charts and strategies.

[Ralph]

Folks, that's very confusing because my experience is somehow different compared with yours.

Generally the data loading with merged contracts works as expected on my end (Also for ES, Josh). The only exception is Crude Oil. If I open my CL workspace without a data connection, then the daily data is loaded on the chart completely and merged as expected. But when I connect to the IB server, then NT tries to download not only the most recent data but also some of the historical data - not all historical data but some days for some contacts (not all contracts). Let's analyse a little mor in detail.

(I kept my comments bold to get not lost inbetween the logs)

Here is the report from the NT trace file:

2010-05-14 22:20:41:536 (InteractiveBrokers) IB.HistoricalDataRequest.Send: last=17.05.2009 instrument='CL 06-09' from='2009-05-15' to='2009-05-17' period=Daily splitAdjusted=False dividendAdjusted=False bars=0
2010-05-14 22:20:41:536 (InteractiveBrokers) IB.HistoricalDataRequest.Send: request='CL;FUT;200906;0;;;NYMEX;;USD;;20090518 00:00:00;1 day;11 M;0;TRADES'
2010-05-14 22:20:42:467 (InteractiveBrokers) IB.HistoricalDataRequest.Process: itemCount=149 instrument='CL 06-09' from='2009-05-15' to='2009-05-17' period=Daily splitAdjusted=False dividendAdjusted=False bars=0
2010-05-14 22:20:42:557 (InteractiveBrokers) Data.Bars.GetBarsNow: instrument='CL 08-09' from='2009-07-17' to='2009-07-19' period=Daily splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Metals / Energy ETH'
2010-05-14 22:20:43:539 (InteractiveBrokers) IB.HistoricalDataRequest.Send: last=19.07.2009 instrument='CL 08-09' from='2009-07-17' to='2009-07-19' period=Daily splitAdjusted=False dividendAdjusted=False bars=0
2010-05-14 22:20:43:539 (InteractiveBrokers) IB.HistoricalDataRequest.Send: request='CL;FUT;200908;0;;;NYMEX;;USD;;20090720 00:00:00;1 day;11 M;0;TRADES'
2010-05-14 22:20:44:130 (InteractiveBrokers) IB.HistoricalDataRequest.Process: itemCount=186 instrument='CL 08-09' from='2009-07-17' to='2009-07-19' period=Daily splitAdjusted=False dividendAdjusted=False bars=0
2010-05-14 22:20:44:180 (InteractiveBrokers) Data.Bars.GetBarsNow: instrument='CL 10-09' from='2009-09-18' to='2009-09-20' period=Daily splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Metals / Energy ETH'
2010-05-14 22:20:45:542 (InteractiveBrokers) IB.HistoricalDataRequest.Send: last=20.09.2009 instrument='CL 10-09' from='2009-09-18' to='2009-09-20' period=Daily splitAdjusted=False dividendAdjusted=False bars=0
2010-05-14 22:20:45:542 (InteractiveBrokers) IB.HistoricalDataRequest.Send: request='CL;FUT;200910;0;;;NYMEX;;USD;;20090921 00:00:00;1 day;11 M;0;TRADES'
2010-05-14 22:20:46:303 (InteractiveBrokers) IB.HistoricalDataRequest.Process: itemCount=218 instrument='CL 10-09' from='2009-09-18' to='2009-09-20' period=Daily splitAdjusted=False dividendAdjusted=False bars=0
2010-05-14 22:20:46:343 (InteractiveBrokers) Data.Bars.GetBarsNow: instrument='CL 11-09' from='2009-10-16' to='2009-10-18' period=Daily splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Metals / Energy ETH'
2010-05-14 22:20:47:545 (InteractiveBrokers) IB.HistoricalDataRequest.Send: last=18.10.2009 instrument='CL 11-09' from='2009-10-16' to='2009-10-18' period=Daily splitAdjusted=False dividendAdjusted=False bars=0
2010-05-14 22:20:47:545 (InteractiveBrokers) IB.HistoricalDataRequest.Send: request='CL;FUT;200911;0;;;NYMEX;;USD;;20091019 00:00:00;1 day;11 M;0;TRADES'
2010-05-14 22:20:48:406 (InteractiveBrokers) IB.HistoricalDataRequest.Process: itemCount=222 instrument='CL 11-09' from='2009-10-16' to='2009-10-18' period=Daily splitAdjusted=False dividendAdjusted=False bars=0
2010-05-14 22:20:48:456 (InteractiveBrokers) Data.Bars.GetBarsNow: instrument='CL 02-10' from='2010-01-15' to='2010-01-17' period=Daily splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Metals / Energy ETH'
2010-05-14 22:20:49:547 (InteractiveBrokers) IB.HistoricalDataRequest.Send: last=17.01.2010 instrument='CL 02-10' from='2010-01-15' to='2010-01-17' period=Daily splitAdjusted=False dividendAdjusted=False bars=0
2010-05-14 22:20:49:547 (InteractiveBrokers) IB.HistoricalDataRequest.Send: request='CL;FUT;201002;0;;;NYMEX;;USD;;20100118 00:00:00;1 day;11 M;0;TRADES'
2010-05-14 22:20:50:449 (InteractiveBrokers) IB.HistoricalDataRequest.Process: itemCount=230 instrument='CL 02-10' from='2010-01-15' to='2010-01-17' period=Daily splitAdjusted=False dividendAdjusted=False bars=0
2010-05-14 22:20:50:529 (InteractiveBrokers) Data.Bars.GetBarsNow: instrument='CL 06-10' from='2010-05-14' to='2010-05-14' period=Daily splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Metals / Energy ETH'
2010-05-14 22:20:51:550 (InteractiveBrokers) IB.HistoricalDataRequest.Send: last=14.05.2010 instrument='CL 06-10' from='2010-05-14' to='2010-05-14' period=Daily splitAdjusted=False dividendAdjusted=False bars=0
2010-05-14 22:20:51:550 (InteractiveBrokers) IB.HistoricalDataRequest.Send: request='CL;FUT;201006;0;;;NYMEX;;USD;;;1 day;11 M;0;TRADES'
2010-05-14 22:20:52:261 (InteractiveBrokers) IB.HistoricalDataRequest.Process: itemCount=232 instrument='CL 06-10' from='2010-05-14' to='2010-05-14' period=Daily splitAdjusted=False dividendAdjusted=False bars=0
2010-05-14 22:20:52:682 (InteractiveBrokers) Data.Bars.GetBarsNow: instrument='CL 06-10' from='2010-05-14' to='2010-05-14' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Metals / Energy ETH'
2010-05-14 22:20:53:553 (InteractiveBrokers) IB.HistoricalDataRequest.Send: last=14.05.2010 instrument='CL 06-10' from='2010-05-14' to='2010-05-14' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2010-05-14 22:20:53:553 (InteractiveBrokers) IB.HistoricalDataRequest.Send: request='CL;FUT;201006;0;;;NYMEX;;USD;;;1 min;5 D;0;TRADES'
2010-05-14 22:20:56:547 (InteractiveBrokers) IB.HistoricalDataRequest.Process: itemCount=6921 instrument='CL 06-10' from='2010-05-14' to='2010-05-14' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0

So, NT reloads data for 06-10 (that is ok) and in addition for 06-09, 08-09, 10-09, 11-09, 02-10. For the historical contracts Nt reloads only some days and not all data.

Here is the related IB report:

GF 22:20:41:491 JTS-DeadlockMonitor: Memory:total=506,816KB free=392,273KB
GF 22:20:42:494 JTS-DeadlockMonitor: CPU:cur=0.00% avg=0.84% 30 min avg=0.62% 5 min avg=0.45% 1 min avg=0.50%
GF 22:20:42:494 JTS-DeadlockMonitor: GC:called=677 times CPU used=0.03%
GF 22:20:42:494 JTS-DeadlockMonitor: Finalizer:cur=0.00% avg=0.00% 30 min avg=0.00% 5 min avg=0.00% 1 min avg=0.00%
GF 22:20:42:882 JTS-EServerSocket-2173: SECDEF Sending reqId=FixSecDefReqBySymbol2115 symbol=CL 0 NYMEX
GF 22:20:43:093 JTS-CCPDispatcher-26: SECDEF Handling reqId=FixSecDefReqBySymbol2115
GF 22:20:43:150 JTS-CCPDispatcher-26: CONFIG | WARNING | Invalid time period, using default | -1 | 0 | null
GF 22:20:43:150 JTS-CCPDispatcher-26: Getting hmds route with exchange=NYMEX group=-1 secType=FUT currency=USD
GF 22:20:43:801 JTS-EServerSocketNotifier-2172: The EBuffer buffer has grown to 7346 bytes
GF 22:20:44:880 JTS-EServerSocket-2173: SECDEF Sending reqId=FixSecDefReqBySymbol2116 symbol=CL 0 NYMEX
GF 22:20:45:068 JTS-CCPDispatcher-26: SECDEF Handling reqId=FixSecDefReqBySymbol2116
GF 22:20:45:069 JTS-CCPDispatcher-26: CONFIG | WARNING | Invalid time period, using default | -1 | 0 | null
GF 22:20:45:070 JTS-CCPDispatcher-26: Getting hmds route with exchange=NYMEX group=-1 secType=FUT currency=USD
GF 22:20:45:470 JTS-EServerSocketNotifier-2172: The EBuffer buffer has grown to 9394 bytes
GF 22:20:46:886 JTS-EServerSocket-2173: SECDEF Sending reqId=FixSecDefReqBySymbol2117 symbol=CL 0 NYMEX
GF 22:20:47:077 JTS-CCPDispatcher-26: SECDEF Handling reqId=FixSecDefReqBySymbol2117
GF 22:20:47:079 JTS-CCPDispatcher-26: CONFIG | WARNING | Invalid time period, using default | -1 | 0 | null

.......

GF 22:20:55:025 JTS-CCPDispatcher-26: Getting hmds route with exchange=NYMEX group=-1 secType=FUT currency=USD
GF 22:20:57:883 JTS-EServerSocketNotifier-2172: The EBuffer buffer has grown to 399223 bytes

You can see that all the NT requests are responded at the related time stamps. So, something is wrong with this particular CL setup.

Don't know if that helps: I modified the rollover dates for all Cl contracts.

Regards
Ralph

[NinjaTrader_JoshP]

Please provide the exact dates being used by the chart. Days to load = ? End date = ? PC time zone? Session template? Your CL rollovers settings will be needed to reproduce and evaluate your exact scenario as well. Thanks.

[Ralph]

Instrument: CL
Chart type: Day
Load data based on: CustomRange
Start date: 2009-03-18
End date: Current day
Session template: Nymex Metal / Energy ETH
Session template time zone: GMT-5, New York
My PC time zone: GMT+1, Germany

Contract / Data in database / Rollover date
05-09 / 2009-02-02 2009-04-21 / 2009-03-18
06-09 / 2009-03-02 2009-05-19 / 2009-04-17
07-09 / 2009-01-04 2009-06-22 / 2009-05-18
08-09 / 2009-05-01 2009-07-21 / 2009-06-19
09-09 / 2009-06-01 2009-08-20 / 2009-07-20
10-09 / 2009-07-01 2009-09-22 / 2009-08-18
11-09 / 2009-08-03 2009-10-20 / 2009-09-21
12-09 / 2009-09-01 2009-11-20 / 2009-10-19
01-10 / 2009-10-01 2009-12-21 / 2009-11-19
02-10 / 2009-11-02 2010-01-20 / 2009-12-17
03-10 / 2009-12-01 2010-02-22 / 2010-01-18
04-10 / 2010-01-04 2010-03-22 / 2010-02-19
05-10 / 2010-02-01 2010-04-20 / 2010-03-19
06-10 / 2010-03-01 2010-05-14 / 2010-04-16

Regards
Ralph

[Harry]

Just to confirm the problems

For many of the futures contracts NT repeatedly and unnecessarily tries to reload existing data from expired contracts. This causes

- longer loading periods
- unnecessary data throttling by IB

So I confirm that NT tries to load where nothing is to be loaded. This only happens when merge policy is set to MergeBackAdjusted.

The reason that this behavior is more disturbing for CL, is that CL contracts are monthly rolled. So compared to index futures there are 3 times as many contracts that are phantom-loaded, so there is 3 times more hazzle and waiting time. Same applies to Brent Crude, which also has monthly rollovers.