NT with an Interactive Brokers (Live Account) back testing

[szayedoud]

Hi - I am trying to back testing using NinjaTrader with an IB connection. I understand that IB doesn't like when you try to pull down a lot of data (say over 5 years), and throttles the I/O. However, I managed to get a couple of years of test data by breaking start/end dates by 3 months at a time, and the historical data is saved locally. (I deleted all the previous historical data before doing this to ensure I had only IB data).

The error I am getting in some time periods is that "Error Requesting bar series: "IB was unable to locate instrument. Please verify your symbol mapping is correct for NQ-03-20 Globex". (see attachement) Any idea what symbol mapping the error is referring to? I did verify that in Tools -> Options -> Market Data -> Historical -> Global Merge Policy is set to "Merge Back Adjusted" - I found this on a post, that refers to a similar issue. I am out of ideas. I have managed to produce tests for yeas 2018 and 2022, but the error has resurfaced in 2018 -2021, but still works for 2022.

[NinjaTrader_ChelseaB]

Hello szayedoud,

The symbol mapping is correct or you wouldn't be able to load any data all.

By default, the control months in the instrument settings do include 2018 through 2020.
To reset these to defaults click Tools -> Database Management -> Update instrument -> check Future expiries -> click Update.

Are you able to open a chart with the end date set to Jan 31st 2020?

[szayedoud]

No, I am unable to load a chart with the instruments in question. One of them was "ES 03-21 Globex" - so I tried to load a chart of that, around January 15, 2021 which would make it the "current" contract. That gave me the same error in the attachment. And I constantly do Tools -> Database Management -> Update instrument -> check Future expiries -> click Update. And then I repair the DB, shutdown NT, delete cache file at the root directory, and the cache file in the DB directory. That seems to help with some years.

My IB Gateway is 19.19.1k and NT is 8.0.28.0, which according to this page is a "supported" pairing. https://support.ninjatrader.com/s/ar...language=en_US

Here's what I do know from IB. They recently (not sure how recent) switched their "exchange" identifier from "Globex" to "CME" (for ES, NQ, EMD and RTY) but YM remained "CBOT" (These could be symbol mappings on their end?). These are identifiers when instantiating a Futures contract in their own native API. However, currently I am only getting good tests for 2021 and 2022, and its 2018-2020 that are giving me errors. I am not sure if that's related or not. I am currently running my tests in NinjaScript running with with an IB connection.

Any other ideas?

[NinjaTrader_ChelseaB]

Hello szayedoud,

That could be related.

If you change the symbol mapping to NQ|GLOBEX does the Jan 15 chart load?

[szayedoud]

I don't see any NGIBLOBEX in the exchanges listing of the ES instrument. I've tried, Globex, Defualt, Clearpoint (CME), and all give the same error

[NinjaTrader_ChelseaB]

Hello szayedoud,

This would be the symbol mapping in the Instrument settings for Interactive Brokers.

Note the vertical line | is not an I or L. This is the key above the Enter key on most keyboards.

[szayedoud]

So we're getting closer. In "Symbol Map" section of the Instrument, for IB it had ES|CME. I changed it to ES|GLOBEX, and reset the exchange to the default "GLOBEX" - there isn't an option "CME" specifically in the exchange listing.

I still got the same error "Please verify your symbol mapping is correct for instrument ES 03-21 Globex". I also tried ES|Globex (if case mattered). Still didn't work

Are you able to get ES 03-21 on a chart using an IB connection? If so, what versions of NT and IB are you using?

[NinjaTrader_ChelseaB]

Hello szayedoud,

Only the supported version of TWS should be used, currently 10.19.1k.

I am also not able to load this data on my end. This may imply that Interactive Brokers is not supplying this older data through the API.

[szayedoud]

(1) That page says that IB Gateway 10.19.1k (not just TWS) is supported https://support.ninjatrader.com/s/ar...language=en_US That's the version I am using.

(2) This page says that these server mapping changes occurred from last Oct -Nov 2022. https://forum.ninjatrader.com/forum/...xecution-at-ib

If that's the case, then I would assume that any prior mappings using "CME" prior to that date (Oct -Nov 2022) should NOT work. I am getting good test runs for the entire years of 2021 and 2022. So your implication that IB is not supporting older data prior to 03-21 (that seems to be about cut off time where things go wrong) might be correct

Does that make sense with you? Any ideas for other workarounds on the NT end? There has to be other NT/IB users out there that are doing work with contracts that are 03-21 and prior. I can't be the only one.

[NinjaTrader_ChelseaB]

Hello szayedoud,

I would support the theory as Interactive Brokers has had limitations with historical data in the API. Some time ago instead of throttling the download speed it was completely denied instead with an error message when requesting large amounts of data.

Interactive Brokers may have more information about what is allowed or if requesting data from before this date requires further setup.

If complementary data source is necessary you may consider a subscription with Kinetick or IQFeed. This would also provide historical tick data if it's needed.
If you are testing Daily bars the Kinetick End of Day Free connection would have daily historical available for free.
Historical data can also be imported from text file from 3rd parties.

[szayedoud]

Yes, I did hear about "restrictions" being lifted (the errors) but throttling was put in place to discourage. I am amazed they do that and can't support the traffic. My NT CQG connection is great, I can wipe out my entire historical data, run a 5 year test and get minute data for those 5 years in about 1 min max. And I think I once went as far back as 15 years without issue. And neither NT or CQG is getting huffy and puffy about it.

I really need historical IB data, so I will go to them. I am migrating my NT system to IB's platform. I need to prove the system has similar results (in the ballpark) using various combinations of platforms and data providers.

Thanks for your help Chelsea!