Problems loading data from IB

[tickling]

Just to add my two cents here, I am also loading daily bars from IB as well as minute bar over several days and NT7 does incur more pacing violations than NT6.5. I had several issues I could not reproduce such as hanging charts and in all of them I found many pacing errors in the log file.

I have suggested in another thread that perhaps NT could implement IB pacing rules in the data adapter. This is not rocket science, it will prevent all the pacing errors and the need to recover from them in NT, avoid occasional historical lockout periods imposed by IB and give IB users smoother and on average faster data loading. Please consider.

[Harry]

Hi Josh,

I will do some more testing and send you a collection of trace files via mail together with a detailed documentation of what I did. Do not want to have permanent crashes.

I have frequent exceptions on the trace files. Some examples:

This happened changing NQ 03-10 from 1 min to 15 min:

2009-12-14 19:54:10:140 (Interactive Brokers) defaultTimeToClose=10.12.2009 00:00:00 flattenAll=False
2009-12-14 19:54:21:890 (Interactive Brokers) Data.Bars.GetBarsNow: instrument='NQ 03-10' from='2009-12-14' to='2009-12-14' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2009-12-14 19:54:21:890 (Interactive Brokers) IB.HistoricalDataRequest.Send: last=14.12.2009 instrument='NQ 03-10' from='2009-12-14' to='2009-12-14' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2009-12-14 19:54:21:890 (Interactive Brokers) IB.HistoricalDataRequest.Send: request='NQ;FUT;201003;0;;;GLOBEX;;USD;;;1 min;5 D;0;TRADES'
2009-12-14 19:54:23:203 (Interactive Brokers) IB.HistoricalDataRequest.Process: itemCount=6405 instrument='NQ 03-10' from='2009-12-14' to='2009-12-14' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2009-12-14 19:54:23:828 (Interactive Brokers) Data.Bars.GetBarsNow: instrument='NQ 03-10' from='2009-12-14' to='2009-12-14' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2009-12-14 19:54:23:828 (Interactive Brokers) IB.HistoricalDataRequest.Send: last=14.12.2009 instrument='NQ 03-10' from='2009-12-14' to='2009-12-14' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2009-12-14 19:54:23:828 (Interactive Brokers) IB.HistoricalDataRequest.Send: request='NQ;FUT;201003;0;;;GLOBEX;;USD;;;1 min;5 D;0;TRADES'
2009-12-14 19:54:23:828 (Interactive Brokers) IB.HistoricalDataRequest.Send: waiting 14,1 seconds2009-12-14 19:54:23:843 UI (re)load complete, took 2,2 secs, 27,3 secs after app start
2009-12-14 19:54:39:312 (Interactive Brokers) IB.HistoricalDataRequest.Process: itemCount=6405 instrument='NQ 03-10' from='2009-12-14' to='2009-12-14' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2009-12-14 19:54:39:500 in OnUnhandledApplicationException
2009-12-14 19:54:39:531 ********* exception trapped *********
2009-12-14 19:54:39:531 Der Objektverweis wurde nicht auf eine Objektinstanz festgelegt.
2009-12-14 19:54:39:531 System.NullReferenceException: Der Objektverweis wurde nicht auf eine Objektinstanz festgelegt.
bei System.Windows.Forms.Control.MarshaledInvoke(Contr ol caller, Delegate method, Object[] args, Boolean synchronous)
bei System.Windows.Forms.Control.Invoke(Delegate method, Object[] args)
bei NinjaTrader.Cbi.SynchronizeInvoke.Invoke(Delegate method, Object[] args)
bei NinjaTrader.Gui.Chart.ChartControl.OnIntervalSelec tInWorkerThread(Object state)
bei System.Threading._ThreadPoolWaitCallback.WaitCallb ack_Context(Object state)
bei System.Threading.ExecutionContext.Run(ExecutionCon text executionContext, ContextCallback callback, Object state)
bei System.Threading._ThreadPoolWaitCallback.PerformWa itCallbackInternal(_ThreadPoolWaitCallback tpWaitCallBack)
bei System.Threading._ThreadPoolWaitCallback.PerformWa itCallback(Object state)

earlier this day when daily data was loaded into an intraday chart

2009-12-14 00:53:24:796 (Interactive Brokers) Data.Bars.GetBarsNow: instrument='ES 03-10' from='2009-12-14' to='2009-12-14' period=Daily splitAdjusted=False dividendAdjusted=False bars=0
2009-12-14 00:53:24:796 (Interactive Brokers) IB.HistoricalDataRequest.Send: last=14.12.2009 instrument='ES 03-10' from='2009-12-14' to='2009-12-14' period=Daily splitAdjusted=False dividendAdjusted=False bars=0
2009-12-14 00:53:24:796 (Interactive Brokers) IB.HistoricalDataRequest.Send: request='ES;FUT;201003;0;;;GLOBEX;;USD;;;1 day;11 M;0;TRADES'
2009-12-14 00:53:25:093 (Interactive Brokers) IB.HistoricalDataRequest.Process: itemCount=134 instrument='ES 03-10' from='2009-12-14' to='2009-12-14' period=Daily splitAdjusted=False dividendAdjusted=False bars=0
2009-12-14 00:53:25:140 (Interactive Brokers) Data.Bars.GetBarsNow: instrument='ES 03-10' from='2009-12-14' to='2009-12-14' period=Daily splitAdjusted=False dividendAdjusted=False bars=0
2009-12-14 00:53:25:140 (Interactive Brokers) IB.HistoricalDataRequest.Send: last=14.12.2009 instrument='ES 03-10' from='2009-12-14' to='2009-12-14' period=Daily splitAdjusted=False dividendAdjusted=False bars=0
2009-12-14 00:53:25:140 (Interactive Brokers) IB.HistoricalDataRequest.Send: request='ES;FUT;201003;0;;;GLOBEX;;USD;;;1 day;11 M;0;TRADES'
2009-12-14 00:53:25:140 (Interactive Brokers) IB.HistoricalDataRequest.Send: waiting 15,7 seconds
2009-12-14 00:53:25:234 (Interactive Brokers) IB.HistoricalDataRequest.Process: itemCount=135 instrument='ES 03-10' from='2009-12-14' to='2009-12-14' period=Daily splitAdjusted=False dividendAdjusted=False bars=0
2009-12-14 00:53:45:343 in OnUnhandledThreadException
2009-12-14 00:53:45:359 ********* exception trapped *********
2009-12-14 00:53:45:359 Der Objektverweis wurde nicht auf eine Objektinstanz festgelegt.
2009-12-14 00:53:45:359 bei NinjaTrader.Data.DataSeries.Get(Int32 index)
bei NinjaTrader.Gui.Chart.ChartControl.CalculateMinMax AndMargins(Graphics graphics)
bei NinjaTrader.Gui.Chart.ChartControl.OnPaint(Object sender, PaintEventArgs e)
bei System.Windows.Forms.Control.OnPaint(PaintEventArg s e)
bei System.Windows.Forms.Control.PaintWithErrorHandlin g(PaintEventArgs e, Int16 layer, Boolean disposeEventArgs)
bei System.Windows.Forms.Control.WmPaint(Message& m)
bei System.Windows.Forms.Control.WndProc(Message& m)
bei System.Windows.Forms.ScrollableControl.WndProc(Mes sage& m)
bei System.Windows.Forms.Control.ControlNativeWindow.O nMessage(Message& m)
bei System.Windows.Forms.Control.ControlNativeWindow.W ndProc(Message& m)
bei System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)
2009-12-14 00:53:47:781 in OnUnhandledThreadException
2009-12-14 00:53:47:781 ********* exception trapped *********
2009-12-14 00:53:47:781 Der Objektverweis wurde nicht auf eine Objektinstanz festgelegt.
2009-12-14 00:53:47:781 bei NinjaTrader.Indicator.IndicatorBase.Update(Int32 index)
bei NinjaTrader.Data.DataSeries.Get(Int32 index)
bei NinjaTrader.Gui.Chart.ChartControl.CalculateMinMax AndMargins(Graphics graphics)
bei NinjaTrader.Gui.Chart.ChartControl.OnPaint(Object sender, PaintEventArgs e)
bei System.Windows.Forms.Control.OnPaint(PaintEventArg s e)
bei System.Windows.Forms.Control.PaintWithErrorHandlin g(PaintEventArgs e, Int16 layer, Boolean disposeEventArgs)
bei System.Windows.Forms.Control.WmPaint(Message& m)
bei System.Windows.Forms.Control.WndProc(Message& m)
bei System.Windows.Forms.ScrollableControl.WndProc(Mes sage& m)
bei System.Windows.Forms.Control.ControlNativeWindow.O nMessage(Message& m)
bei System.Windows.Forms.Control.ControlNativeWindow.W ndProc(Message& m)
bei System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)
2009-12-14 00:53:47:781 (Interactive Brokers) Data.Bars.GetBarsNow: instrument='ES 03-10' from='2009-12-14' to='2009-12-14' period=Daily splitAdjusted=False dividendAdjusted=False bars=0
2009-12-14 00:53:47:781 (Interactive Brokers) IB.HistoricalDataRequest.Send: last=14.12.2009 instrument='ES 03-10' from='2009-12-14' to='2009-12-14' period=Daily splitAdjusted=False dividendAdjusted=False bars=0
2009-12-14 00:53:47:781 (Interactive Brokers) IB.HistoricalDataRequest.Send: request='ES;FUT;201003;0;;;GLOBEX;;USD;;;1 day;11 M;0;TRADES'
2009-12-14 00:53:47:781 (Interactive Brokers) IB.HistoricalDataRequest.Send: waiting 13,5 seconds
2009-12-14 00:53:48:125 (Interactive Brokers) IB.HistoricalDataRequest.Process: itemCount=134 instrument='ES 03-10' from='2009-12-14' to='2009-12-14' period=Daily splitAdjusted=False dividendAdjusted=False bars=0

Both exceptions lead to a crash of NT 7.

[NinjaTrader_Dierk]

We would need reproducible scenarios in order to analyze what led to the exceptions. Thanks

[Harry]

This is not 100% reproducible, but you can see from the trace files that null reference exceptions occured in both cases while loading data from Interactive Brokers.

The null reference exceptions caused NT to crash.

All this is only minimum configuration. Control center + 1 chart with a single data series only.

All I can do is to document the crashes as good as possible, but I cannot order NT7 and IB to produce them.

[Harry]

I have sent a mail to support with more details, screenshot and trace file.

But again:

Adding three pivot indicators to a chart has never been a problem for NT 6.5., this takes milliseconds at any time of the day.

With NT 7 the same task becomes close to impossible. Why do pacing limitations of IB only apply to NT 7 , but not to NT 6.5? This is a serious bug, preventing me from using a whole bunch of proprietary indicators that rely on daily data.

[NinjaTrader_Dierk]

You are incorrect in your understanding that the pacing violation would be different between NT7 and NT6.5:
- the NT IB adapter is exact the same (apart from CL expiry work around which is not related)
- pacing violations are imposed by TWS and are beyond NT's control

As explained multiple times before:
- next beta will comes with some changes in regards to handling pacing violation. This might make a difference
- you might consider using a different data service in case your IB issues would not be overcome

[tickling]

Dierk,

Thanks for your answer.

I have written some production applications natively on IB API and got to know its "moods".

Pacing violations are caused when the requesting client does not follow pacing rules. This is in total control of the client. IB publishes when pacing violation occurs as follows :

The following conditions can cause a pacing violation:
• Making identical historical data requests within 15 seconds;
• Making six or more historical data requests for the same Contract, Exchange and Tick Type within two seconds.
Also, observe the following limitation when requesting historical data:
• Do not make more than 60 historical data requests in any ten-minute period.


NT obviously does not follow these rules in its implementation as can be seen from monitoring TWS API log when this occurs. You can turn on API log to "detailed" and load a 30 day minute chart in NT to see this (and the resulting pacing violation errors).

Apart from being "not elegant" per se I suspect that my and others problems with IB may have to do with the recovering attempts. Maybe implementing this properly will not necessitate recommending users to change data feeds (which are either way more expensive or require changing brokers). Thanks for considering this.

[NinjaTrader_Dierk]

>> This is in total control of the client.
You are correct if you mean by "client" you yourself as NT user. Your usage profile imposes the data requests sent by NT to IB.

Here is a (very simple) sample:
- add all NASAQ100 to a market analyzer
- add an SMA 1 minute column
- connect IB
-> you inevitable will run into pacing violations (provided TWS manages to process 60+ requests in less than 10 minutes), nothing NT could do

I suggest checking out an alternate data provider if you wanted to get rid of IB pacing violations.

Also: not sure what you mean by "recovering attempts"

Edit: We may revisit that issue after next beta, since it would come with some changes in regards to handling pacing violations. This "might" make a difference (we'll see...)

[tickling]

Excellent example. The "Client" I meant TWS client, in this case NT IB connection adapter.

Knowing that duty calls for 100 requests from TWS, it should space them out each 10 seconds. This way no pacing voilation occurs. Yes it takes forever, yes it is the human user usage pattern + IB limits that caused it, but THIS is the best NT could do for its human user. AND it is not such a big penalty as it is one time only. Cost of keeping IB ...

Just firing those requests knowing that they hit a brick wall and then saying it is out of NT control is a bit like hiding ones head in the sand.... Won't you agree ?

[NinjaTrader_Dierk]

>> Knowing that duty calls for 100 requests from TWS, it should space them out each 10 seconds.
You may find that acceptable, but 95% of the other users won't

[tickling]

Well they are not getting their data anyway any faster today. They get blank rows and "loading.." charts.

The "recovery attempts" I was referring to are occasions where the beta hang or just vanish without me being able to reproduce it and after heavy pacing violations.

Ok that was only a suggestion, never meant it to grow so much

It seems that not so many people using NT use it with IB which is something for me to think about. I am very happy with IB as a broker and very unhappy with its TWS. I like NT very much so far (not using it in production as yet) but NT+IB data feed seems to be a mismatch. Any feedback from other users ?

I will wait for the final release.

Thanks Dierk

[NinjaTrader_Dierk]

Sounds good

[Harry]

Thank you for the information.

I will wait for the next release. I am still confident that I will be able to operate a single dataseries intraday chart with pivots that does not lock NT 7.

[Harry]

NT 7 not compatible with IB API pacing rules for historical data

Have checked the log files of TWS (level detailed) and compared NT 7 to NT 6.5 when performing

- Connecting Control Center to TWS
- Open 1 chart ES (15 min, 65 days)
- Add 3 pivot indicators (daily, weekly, monthly)

Findings:

Already when opening the single ES chart, NT 7 may cause a pacing violation, here is the log

YY 15:51:41:114 JTS-EServerSocketNotifier-84: The EBuffer buffer has grown to 425844 bytes
YY 15:51:41:221 JTS-EServerSocket-85: SECDEF Sending reqId=FixSecDefReqBySymbol28 symbol=ES 0 GLOBEX
YY 15:51:41:255 JTS-CCPDispatcher-22: SECDEF Handling reqId=FixSecDefReqBySymbol28
YY 15:51:41:257 JTS-CCPDispatcher-22: Checking market data
YY 15:51:41:761 JTS-Fuse-bomb3-100: Checking market data

This causes a 13 sec delay ... this explains, why sometimes the loading of a simple chart takes 16 sec instead of 2 sec!

YY 15:51:54:761 JTS-EServerSocket-85: SECDEF Sending reqId=FixSecDefReqBySymbol29 symbol=ES 0 GLOBEX
YY 15:51:54:790 JTS-CCPDispatcher-22: SECDEF Handling reqId=FixSecDefReqBySymbol29
YY 15:51:54:791 JTS-CCPDispatcher-22: CONFIG | WARNING | Invalid time period, using default | -1 | 0 | null
YY 15:51:54:791 JTS-CCPDispatcher-22: Getting hmds route with exchange=GLOBEX group=-1 secType=FUT currency=USD
YY 15:51:54:791 JTS-CCPDispatcher-22: ROUTING | INFO | jconnection.service.j@121052 | Could not find route table entry | exch=GLOBEX secType=FUT currency=USD dataType=ConAdj
YY 15:51:54:791 JTS-CCPDispatcher-22: No routes were found for: SecType: FUT, Exchange: GLOBEX
YY 15:51:54:791 JTS-CCPDispatcher-22: No data of type ConAdj is available for the exchange 'GLOBEX' and the security type 'Futures'
YY 15:51:54:792 JTS-CCPDispatcher-22: Getting hmds route with exchange=GLOBEX group=-1 secType=FUT currency=USD
YY 15:51:54:792 JTS-CCPDispatcher-22: Checking market data

When adding pivots with NT 7 the log shows two repeated data queries. For NT 6.5. I only could observe one query.

First query:

YY 15:52:38:996 JTS-EServerSocket-85: SECDEF Sending reqId=FixSecDefReqBySymbol34 symbol=ES 0 GLOBEX
YY 15:52:39:026 JTS-CCPDispatcher-22: SECDEF Handling reqId=FixSecDefReqBySymbol34
YY 15:52:39:027 JTS-CCPDispatcher-22: CONFIG | WARNING | Invalid time period, using default | -1 | 0 | null
YY 15:52:39:027 JTS-CCPDispatcher-22: Getting hmds route with exchange=GLOBEX group=-1 secType=FUT currency=USD
YY 15:52:39:028 JTS-CCPDispatcher-22: ROUTING | INFO | jconnection.service.j@121052 | Could not find route table entry | exch=GLOBEX secType=FUT currency=USD dataType=ConAdj
YY 15:52:39:028 JTS-CCPDispatcher-22: No routes were found for: SecType: FUT, Exchange: GLOBEX
YY 15:52:39:028 JTS-CCPDispatcher-22: No data of type ConAdj is available for the exchange 'GLOBEX' and the security type 'Futures'
YY 15:52:39:028 JTS-CCPDispatcher-22: Getting hmds route with exchange=GLOBEX group=-1 secType=FUT currency=USD
YY 15:52:39:028 JTS-CCPDispatcher-22: Checking market data

Second query (not yet a pacing violation):

YY 15:52:39:371 JTS-EServerSocket-85: SECDEF Sending reqId=FixSecDefReqBySymbol35 symbol=ES 0 GLOBEX
YY 15:52:39:400 JTS-CCPDispatcher-22: SECDEF Handling reqId=FixSecDefReqBySymbol35
YY 15:52:39:400 JTS-CCPDispatcher-22: CONFIG | WARNING | Invalid time period, using default | -1 | 0 | null
YY 15:52:39:401 JTS-CCPDispatcher-22: Getting hmds route with exchange=GLOBEX group=-1 secType=FUT currency=USD
YY 15:52:39:401 JTS-CCPDispatcher-22: ROUTING | INFO | jconnection.service.j@121052 | Could not find route table entry | exch=GLOBEX secType=FUT currency=USD dataType=ConAdj
YY 15:52:39:401 JTS-CCPDispatcher-22: No routes were found for: SecType: FUT, Exchange: GLOBEX
YY 15:52:39:401 JTS-CCPDispatcher-22: No data of type ConAdj is available for the exchange 'GLOBEX' and the security type 'Futures'
YY 15:52:39:401 JTS-CCPDispatcher-22: Getting hmds route with exchange=GLOBEX group=-1 secType=FUT currency=USD
YY 15:52:39:526 JTS-Fuse-bomb3-108: Checking market data

And this is the pacing violation that knocks out NT7 altogether.

Just to add here: With NT 6.5. I have 4 charts open with 15 indicators that require daily data. I have tested this 10 times on both NT 6.5. and NT 7 and it can definitely be reproduced. There was never any JTS-Fuse-bomb3 event with NT 6.5. Only NT7 causes the pacing violations.

Sorry, I am bringing this up again here, but as of today NT7 is a bit too fast when communicating with TWS API. So I am looking forward to some improvements....

[NinjaTrader_Dierk]

Please see my post #23 below
Edit: We may revisit that issue after next beta, since it would come with some changes in regards to handling pacing violations. This "might" make a difference (we'll see...)