Data loading with NT 7.0.0.17

[uncogs]

my problem is solved, thanks for your help Harry.
Unfortunately I don't know how it happens. I logged yesterday and the problem was still there. I didn't have time to try your different solutions.
I log again today and data for CL 08/10 load normally

Now I'm wondering if there is not a kind of hidden bug when asking a chart and beeing close to the rolling date ? (even when DoNotMerge)
So I suppose I'll wait for the next roll to know

Anyway thanks

[Harry]

NT 7.0.0.17, Interactive Brokers, FOREX, updating data base.

When updating the data base via toggling through a simple default chart with no indicators, NinjaTrader calls every instrument twice, thus causing data throttling by Interactive Brokers.

This slows down the process of loading historical data, as each instrument requires an extra 14 seconds of throttling time. Example below:


010-06-22 12:26:44:718 (Interactive Brokers) Data.Bars.GetBarsNow: instrument='$EURUSD' from='2010-06-21' to='2010-06-22' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0 session='Forex'
2010-06-22 12:26:44:718 (Interactive Brokers) IB.HistoricalDataRequest.Send: last=22.06.2010 instrument='$EURUSD' from='2010-06-21' to='2010-06-22' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2010-06-22 12:26:44:718 (Interactive Brokers) IB.HistoricalDataRequest.Send: request='EUR;CASH;;0;;;IDEALPRO;;USD;;;1 min;5 D;0;BID'
2010-06-22 12:26:47:453 (Interactive Brokers) IB.HistoricalDataRequest.Process: itemCount=6492 instrument='$EURUSD' from='2010-06-21' to='2010-06-22' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2010-06-22 12:26:47:578 (Interactive Brokers) Data.Bars.GetBarsNow: instrument='$EURUSD' from='2010-06-22' to='2010-06-22' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0 session='Forex'
2010-06-22 12:26:47:578 (Interactive Brokers) IB.HistoricalDataRequest.Send: last=22.06.2010 instrument='$EURUSD' from='2010-06-22' to='2010-06-22' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2010-06-22 12:26:47:578 (Interactive Brokers) IB.HistoricalDataRequest.Send: request='EUR;CASH;;0;;;IDEALPRO;;USD;;;1 min;5 D;0;BID'
2010-06-22 12:26:47:578 (Interactive Brokers) IB.HistoricalDataRequest.Send: waiting 13,1 seconds
2010-06-22 12:27:02:453 (Interactive Brokers) IB.HistoricalDataRequest.Process: itemCount=6492 instrument='$EURUSD' from='2010-06-22' to='2010-06-22' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2010-06-22 12:27:22:250 (Interactive Brokers) Data.Bars.GetBarsNow: instrument='$GBPUSD' from='2010-06-21' to='2010-06-22' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0 session='Forex'
2010-06-22 12:27:22:250 (Interactive Brokers) IB.HistoricalDataRequest.Send: last=22.06.2010 instrument='$GBPUSD' from='2010-06-21' to='2010-06-22' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2010-06-22 12:27:22:250 (Interactive Brokers) IB.HistoricalDataRequest.Send: request='GBP;CASH;;0;;;IDEALPRO;;USD;;;1 min;5 D;0;BID'
2010-06-22 12:27:23:890 (Interactive Brokers) IB.HistoricalDataRequest.Process: itemCount=6491 instrument='$GBPUSD' from='2010-06-21' to='2010-06-22' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2010-06-22 12:27:24:062 (Interactive Brokers) Data.Bars.GetBarsNow: instrument='$GBPUSD' from='2010-06-22' to='2010-06-22' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0 session='Forex'
2010-06-22 12:27:24:265 (Interactive Brokers) IB.HistoricalDataRequest.Send: last=22.06.2010 instrument='$GBPUSD' from='2010-06-22' to='2010-06-22' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2010-06-22 12:27:24:265 (Interactive Brokers) IB.HistoricalDataRequest.Send: request='GBP;CASH;;0;;;IDEALPRO;;USD;;;1 min;5 D;0;BID'
2010-06-22 12:27:24:265 (Interactive Brokers) IB.HistoricalDataRequest.Send: waiting 14 seconds
2010-06-22 12:27:39:781 (Interactive Brokers) IB.HistoricalDataRequest.Process: itemCount=6491 instrument='$GBPUSD' from='2010-06-22' to='2010-06-22' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2010-06-22 12:27:41:437 (Interactive Brokers) Data.Bars.GetBarsNow: instrument='$USDCAD' from='2010-06-21' to='2010-06-22' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0 session='Forex'
2010-06-22 12:27:41:437 (Interactive Brokers) IB.HistoricalDataRequest.Send: last=22.06.2010 instrument='$USDCAD' from='2010-06-21' to='2010-06-22' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2010-06-22 12:27:41:437 (Interactive Brokers) IB.HistoricalDataRequest.Send: request='USD;CASH;;0;;;IDEALPRO;;CAD;;;1 min;5 D;0;BID'
2010-06-22 12:27:42:875 (Interactive Brokers) IB.HistoricalDataRequest.Process: itemCount=6491 instrument='$USDCAD' from='2010-06-21' to='2010-06-22' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2010-06-22 12:27:43:015 (Interactive Brokers) Data.Bars.GetBarsNow: instrument='$USDCAD' from='2010-06-22' to='2010-06-22' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0 session='Forex'
2010-06-22 12:27:43:453 (Interactive Brokers) IB.HistoricalDataRequest.Send: last=22.06.2010 instrument='$USDCAD' from='2010-06-22' to='2010-06-22' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2010-06-22 12:27:43:453 (Interactive Brokers) IB.HistoricalDataRequest.Send: request='USD;CASH;;0;;;IDEALPRO;;CAD;;;1 min;5 D;0;BID'
2010-06-22 12:27:43:453 (Interactive Brokers) IB.HistoricalDataRequest.Send: waiting 14 seconds

[NinjaTrader_Bertrand]

Harry, will try to reproduce on my end - do you normally do this update for the DB daily?

[Harry]

Yes, I do daily updates of my database. My habit is loading the data via an open chart, so I can see what the market has been doing. For convenience I link a chart to the DOM and then toggle through the instruments by using arrows.

As the procedure above causes data throttling by Interactive Brokers, I now download data via the download manager, then disconnect and only toggle through the charts, when disconnected.

But I would like to come back to my original habits, because I think there is no need to double-load all sets of data and cause the throttling.

Harry, will try to reproduce on my end - do you normally do this update for the DB daily?

[NinjaTrader_Bertrand]

Harry, we looked into and those requests are expected and technically needed by the IB API as they are split in historical and realtime data for your current day, thus you see this double request.

[Harry]

Hi Bertrand,

thanks for your answer. However, the answer is not satisfying, at least I do not understand it.

-> to update a chart, you only need historical data
-> from the trace file it is clear that two historical data requests were made
-> NinjaTrader 6.5. needs only one request to load the minute data and it works and is fast

So NT 7.0 unnecessarily causes data throttling as opposed to NT6.5

NinjaTrader 6.5.

2010-06-23 18:15:25:625 (Interactive Brokers) Data.Bars.GetBarsNow: instrument='$EURUSD' from='2010-06-21' to='2010-06-23' sessionBegin='00:00:00' sessionEnd='00:00:00' excludeWeekend=False period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2010-06-23 18:15:25:640 (Interactive Brokers) IB.HistoricalDataRequest.Send: last=23.06.2010 instrument='$EURUSD' from='2010-06-21' to='2010-06-23' sessionBegin='00:00:00' sessionEnd='00:00:00' excludeWeekend=False period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2010-06-23 18:15:25:640 (Interactive Brokers) IB.HistoricalDataRequest.Send: request='EUR;CASH;;0;;;IDEALPRO;;USD;;;1 min;5 D;0;BID'
2010-06-23 18:15:27:156 (Interactive Brokers) IB.HistoricalDataRequest.Process: itemCount=6841 instrument='$EURUSD' from='2010-06-21' to='2010-06-23' sessionBegin='00:00:00' sessionEnd='00:00:00' excludeWeekend=False period=1 Min splitAdjusted=False dividendAdjusted=False bars=0 

versus NinjaTrader 7.0

2010-06-23 18:22:20:718 (Interactive Brokers) Data.Bars.GetBarsNow: instrument='$EURUSD' from='2010-06-22' to='2010-06-23' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0 session='Forex'
2010-06-23 18:22:20:718 (Interactive Brokers) IB.HistoricalDataRequest.Send: last=23.06.2010 instrument='$EURUSD' from='2010-06-22' to='2010-06-23' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2010-06-23 18:22:20:718 (Interactive Brokers) IB.HistoricalDataRequest.Send: request='EUR;CASH;;0;;;IDEALPRO;;USD;;;1 min;5 D;0;BID'
2010-06-23 18:22:22:437 (Interactive Brokers) IB.HistoricalDataRequest.Process: itemCount=6848 instrument='$EURUSD' from='2010-06-22' to='2010-06-23' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2010-06-23 18:22:25:078 (Interactive Brokers) Data.Bars.GetBarsNow: instrument='$EURUSD' from='2010-06-23' to='2010-06-23' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0 session='Forex'
2010-06-23 18:22:25:078 (Interactive Brokers) IB.HistoricalDataRequest.Send: last=23.06.2010 instrument='$EURUSD' from='2010-06-23' to='2010-06-23' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2010-06-23 18:22:25:078 (Interactive Brokers) IB.HistoricalDataRequest.Send: request='EUR;CASH;;0;;;IDEALPRO;;USD;;;1 min;5 D;0;BID'
2010-06-23 18:22:25:078 (Interactive Brokers) IB.HistoricalDataRequest.Send: waiting 11,6 seconds 

Harry, we looked into and those requests are expected and technically needed by the IB API as they are split in historical and realtime data for your current day, thus you see this double request.

[NinjaTrader_Bertrand]

Harry, I tried on my end was unable to reproduce your observation of a difference -

2010-06-23 10:45:04:587 (IB) Data.Bars.GetBarsNow: instrument='$EURUSD' from='2010-06-18' to='2010-06-23' sessionBegin='00:00:00' sessionEnd='00:00:00' excludeWeekend=False period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2010-06-23 10:45:04:653 (IB) IB.HistoricalDataRequest.Send: last=6/23/2010 instrument='$EURUSD' from='2010-06-18' to='2010-06-23' sessionBegin='00:00:00' sessionEnd='00:00:00' excludeWeekend=False period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2010-06-23 10:45:04:653 (IB) IB.HistoricalDataRequest.Send: request='EUR;CASH;;0;;;IDEALPRO;;USD;;;1 min;5 D;0;BID'
2010-06-23 10:45:17:774 (IB) IB.HistoricalDataRequest.Process: itemCount=4963 instrument='$EURUSD' from='2010-06-18' to='2010-06-23' sessionBegin='00:00:00' sessionEnd='00:00:00' excludeWeekend=False period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2010-06-23 10:45:18:038 (IB) Data.Bars.GetBarsNow: instrument='$EURUSD' from='2010-06-23' to='2010-06-23' sessionBegin='00:00:00' sessionEnd='00:00:00' excludeWeekend=False period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2010-06-23 10:45:18:064 (IB) IB.HistoricalDataRequest.Send: last=6/23/2010 instrument='$EURUSD' from='2010-06-23' to='2010-06-23' sessionBegin='00:00:00' sessionEnd='00:00:00' excludeWeekend=False period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2010-06-23 10:45:18:064 (IB) IB.HistoricalDataRequest.Send: request='EUR;CASH;;0;;;IDEALPRO;;USD;;;1 min;5 D;0;BID'
2010-06-23 10:45:18:064 (IB) IB.HistoricalDataRequest.Send: waiting 2.6 seconds 

You see above the same splitup as needed for the IB API as we have discussed on the NT7 side of things.

[Harry]

Just started NT 6.5., connected to IB with my default workspace (instruments YM and Tick). Then opened a default 5 min chart for ES, and toggled to EURUSD to load data. Then I closed down NT 6.5. again.

Attached tracefile for the complete 2-minute NinjaTrader session, and as you can see, there was only one request to load data, and it was lightning-fast (no 14 seconds of waiting time).

[NinjaTrader_Ray]

Harry,

There was a miscommunication in one of our responses below. The change between 6.5 and 7 was implemented by us and not a requirement of the IB API. There can be situations where multiple requests are made. The requests from my understanding are not duplicate requests but they can overlap. This logic is generic and a layer above our market data adapters. The net result of this change is overlall positive but there can be situations where its not optimal.