IB + Ninja (Pacing Violation)

[TraderSU]

I see this problem is in existence for a long time, yet not addressed by NT team. Why?

I think IB has done it's part by documenting their historical data policy and it's full responsibility of NT team to make sure that their adapter does not violate their policy.

They can easily add some throttling code which will prevent from any such violation. Should be pretty easy fix if dev-team is pushed for it but so far I've seen NT support/management is showing this as IB's problem not theirs...

To other IB+NT combo users -- how do you feel about it?

Thanks,

[NinjaTrader_Brett]

Hello,

Thank you for your forum post.

I contacted my product manager on your post. He let me know that NinjaTrader does not limit the amount of data that a user can request and that a user can request as much data as you want, this is the same with all connection technologies as we are not who places limits on this. This pacing violation is a limitation imposed by IB and is a limitation on how much data you can transfer over a set time frame. This has been discussed internally and the error that is presented when you have requested too much data from IB is currently the best way to go, as other options such as bandwidth throttling have their own issues as well and are not desirable. Especially when trying to get the best performance.

Would recommend that you reduce the amount of days back data that you load on a chart to prevent this error from occurring.

Let me know if I can be of further assistance.

[TraderSU]

Hello,

Thank you for your forum post.

I contacted my product manager on your post. He let me know that NinjaTrader does not limit the amount of data that a user can request and that a user can request as much data as you want, this is the same with all connection technologies as we are not who places limits on this. This pacing violation is a limitation imposed by IB and is a limitation on how much data you can transfer over a set time frame. This has been discussed internally and the error that is presented when you have requested too much data from IB is currently the best way to go, as other options such as bandwidth throttling have their own issues as well and are not desirable. Especially when trying to get the best performance.

Would recommend that you reduce the amount of days back data that you load on a chart to prevent this error from occurring.

Let me know if I can be of further assistance.

Since IB connector is indeed IB specific not a generic one; I see no point in letting it request unlimited amount of data knowing that it will cause the pacing violation.

For example, I start fresh NT7 install and punch ES 06-10 as the instrument with ETH session and would like to pull last 120 days worth hourly data. This is what happens to me.

1. After I click [Ok], NT7 will start bombarding TWS with huge number of historical request which is going back to 2005-06 era... and I get banned immediately... What the heck this this? Why is NT going back to 2005-2006 contracts?

2. After waiting 5 minutes, I try again. This time I can load up to 32-35 days worth data before TWS bans me again and I'm stuck for next 5 minutes.

3. Now I try to go in 30, 60, 90, 120 steps manually. Which works and if I follow this approach from beginning, item #2 can be avoided. #1 is still a problem for at least for first time access or when ever contract rolls over.

Historical data policy is available in public domain and I guess we can not blame IB for this issue.

Lets play blame game here and try to figure out who is responsible for it.

1. NT Users - They are stupid, they don't know what they are requesting. They should behave smartly...

2. NT Team - They are ignoring this problem... This can be fixed easily but hey, I can easily blame IB or our stupid users for this problem and avoid this fix.

----------

Anyway, lets get constructive here.

If you are not fixing the code then please let me know how can I pull last 180 days worth ES, NQ, TF and other futures ETH data without any pacing violation. I need it for back-testing :-(

Thanks for prompt reply :-)

[NinjaTrader_Brett]

Hello,

Thank you for your suggestions.

At this point you have 2 options available to you. You can continue to request data in small chunks to get the data you need and aggregate it in the NinjaTrader database. Then disconnect from IB and run your backtest on the locally stored data.

Or you can connect to a data feed that will provide you the data you need.

I have included information below on how to obtain discounted rates and free trials from the following datafeed providers.
Barchart.com (subscriptions starting at $45/monthly):

https://www.barchart.com/register/realtime/?ID=NJA

IQFeed offers a 7-day free trial for NinjaTrader users.

https://www.iqfeed.net/ninjatrader/index.cfm?displayaction=start&promo=1717496

eSignal promotional rate for NinjaTrader clients:

http://www.ninjatrader.com/webnew/eSignalSpecialNinja08.pdf

Let me know if I can be of further assistance.

[TraderSU]

You can continue to request data in small chunks to get the data you need and aggregate it in the NinjaTrader database. Then disconnect from IB and run your backtest on the locally stored data.

Ok, small question. When I pull ES 06-10 and try to fetch 32 days worth data. How many requests are made to TWS API?

I've written a script (AutoHotkey) that acts on NT Data Series screen and does data loading in small chunks. I can do last 32 days and then keep moving End Date to period month after a wait...

This script works sometime but fails other time...

; load data from TWS

SetPeriod(32)

; request and then wait 6 minutes

SetEndDate("06/01/2010")
Sleep 360000 
SetEndDate("05/01/2010")
Sleep 360000
SetEndDate("04/01/2010")
Sleep 360000

SetEndDate("03/01/2010")
Sleep 360000
SetEndDate("02/01/2010")
Sleep 360000
SetEndDate("01/01/2010")
Sleep 360000

SetEndDate("12/01/2009")
Sleep 360000
SetEndDate("11/01/2009")
Sleep 360000
SetEndDate("10/01/2009")
Sleep 360000

SetEndDate("09/01/2009")
Sleep 360000
SetEndDate("08/01/2009")

SetPeriod(DaysToLoad)
{
	SetTitleMatchMode RegEx
	WinWait, Data Series
	WinActivate 
	MouseClick, left, 620, 240
	Send {BS}{BS}{BS}{BS}
	Send %DaysToLoad%
}

SetEndDate(EndDate)
{
	SetTitleMatchMode RegEx
	WinWait, Data Series
	WinActivate
	; Set [EndDate]
	MouseClick, left, 660, 260
	Send {BS}{BS}{BS}{BS}{BS}{BS}{BS}{BS}{BS}{BS}{BS}{BS}
	Send %EndDate%
	; press [Apply]
	MouseClick, left, 400, 590
}

[NinjaTrader_Brett]

Hello,

Requests can be made for previous contracts based on the merge policy. If you do not want a request made to a previous contract. Goto Control Center->Tools->Options->Data Tab-> and change ther merge policy to DoNotMerge.

Now once you get to the end of the ES 06-10 contract you will have to switch over to the ES 03-10 contract manually then start downloading that contracts data.

Then you will have merge the data for all the contracts into one long continous contract.

[TraderSU]

Hello,

Requests can be made for previous contracts based on the merge policy. If you do not want a request made to a previous contract. Goto Control Center->Tools->Options->Data Tab-> and change ther merge policy to DoNotMerge.

Now once you get to the end of the ES 06-10 contract you will have to switch over to the ES 03-10 contract manually then start downloading that contracts data.

Then you will have merge the data for all the contracts into one long continous contract.

Manual switch is pain and I love this auto-merge feature :-)

But why is this going back to 2005-06 era when I only asked for last 20 days worth intraday? Does it query for the whole chain at the first attempt so that auto-merge can work? Can't we just do query for chain for requested period or past 1 year (as my data provider i.e. IB only have historical data for that duration).

Since it is happening only once per contract, guess I can live with it... but is it some bug or expected behavior?

Thanks a lot for your prompt replies.

[TraderSU]

Ok, small question. When I pull ES 06-10 and try to fetch 32 days worth data. How many requests are made to TWS API?

http://www.interactivebrokers.com/php/apiUsersGuide/apiguide/api/historical_data_limitations.htm

Each request is restricted to duration and bar size values that return no more than 2000 bars (2000 bars per request).

Do not make more than 60 historical data requests in any ten-minute period.

I think NT always pull 1m chart data regardless what is my chart timeframe (5m, 15m, 60m).

So for EHT; 24hr = 1440 * 1m bar ; So there will be 1 request per day....
And for RTH (or stock); 1 request can query 2 days worth data... (not sure)...

If I pull 62 days at once, it will have 40-48 trading days and same number of requests. So for back-fill scripting, I should be perfectly fine with a cycle of 10 minutes wait followed by 62 days data...

Am I missing something.?

[NinjaTrader_Brett]

Hello,

Yes this is correct on minute bars. Minute data is the same weather your building a 5 minute or 1440 minute chart with it its all built by 1 minute data. However Daily charts and volume/tick charts are not built with minute data.

I would head down to 30 days of data at a time as I have not run into any issues when I keep the days back at 30. When you start getting above the for minute charts is when you get into grey territory for pacing violations in my experience.

I will check into the past year request however I believe this is expected for the auto merge to function.

[TraderSU]

I would head down to 30 days of data at a time as I have not run into any issues when I keep the days back at 30. When you start getting above the for minute charts is when you get into grey territory for pacing violations in my experience.

I've already modified my script for 62 days and first round on YM went fine with no issue. Will come to know the final outcome within an hour. I don't mind an hour of loading time if it is un-attended.

Possibly my old script was failing because of other interference, hist data request from another instance of NT connecting to same TWS.

[NinjaTrader_Brett]

Hello,

Are you able to reproduce the request you see for 2005 2006 year. If so can you please let me know what you do to reproduce this as this is not expected. NinjaTrader should only request data for the days that it requests.

Glad to hear your script is running well.


I look forward to assisting you further.

[TraderSU]

Hello,

Are you able to reproduce the request you see for 2005 2006 year. If so can you please let me know what you do to reproduce this as this is not expected. NinjaTrader should only request data for the days that it requests.

This issue comes when you have fresh install (remove the `My Documents\NinjaTrader 7` and let it re-initialize)... Let me know if that helps...

[TraderSU]

The AHK script worked like charm.

Tool needed to run the script.

http://www.autohotkey.com/

; load data from TWS

SetPeriod(62)

SetEndDate("06/01/2010")
Sleep 600000
SetEndDate("04/01/2010")
Sleep 600000
SetEndDate("02/01/2010")
Sleep 600000

SetEndDate("12/01/2009")
Sleep 600000
SetEndDate("10/01/2009")
Sleep 600000
SetEndDate("08/01/2009")
Sleep 600000

SetPeriod(DaysToLoad)
{
	SetTitleMatchMode RegEx
	WinWait, Data Series
	WinActivate 
	MouseClick, left, 620, 240
	Send {BS}{BS}{BS}{BS}
	Send %DaysToLoad%
}

SetEndDate(EndDate)
{
	SetTitleMatchMode RegEx
	WinWait, Data Series
	WinActivate
	; Set [EndDate]
	MouseClick, left, 660, 260
	Send {BS}{BS}{BS}{BS}{BS}{BS}{BS}{BS}{BS}{BS}{BS}{BS}
	Send %EndDate%
	; press [Apply]
	MouseClick, left, 400, 590
}

[NinjaTrader_Brett]

Hello,

I was unable to duplicate this with a fresh install. What specfic settings do you use and what instrument when you see this in the Control Center.

Thank You.

[holgeramort]

Has the pacing violation issue been resolved?