New futures stop loading after 360 bars on the most recent day NT 8.0.27.1

[zstheorist]

Another funny bug. Now that the market is closed I started to download from Interactive Brokers several upcoming futures contracts (ES 06-23, NQ 06-23, 6B 06-23, SI 05-23, ZB 06-23, etc.) to get ready for rollovers. In every case I opened a chart to show the last 2 weeks to a month leading up, e.g. 1/29/23-2/10/23 in each of their local ETH trading hours (CME, CBOT, NYMEX and their contract variant: CME FX Futures ETH, etc.). In EVERY case the data loaded, but stopped at 6 AM on Friday 2/10/23 (360 bars). You can see a picture for 6B below. I noticed this last month as well, but thought maybe it was an anomaly. Nope. It is perfectly consistent.
Oddly, enough futures for which I already have data from earlier today, e.g. ES 03-23, loaded to the end of the day. But new data that had stopped at 6 AM, e.g. ES 06-23, would not load any more data unless I went to Historical Data and re-downloaded the entire day from scratch. The market closed at 4 PM (960 bars of data).
I don't know it if matters, but I was trying this after the close since I didn't want live data.

[NinjaTrader_Jason]

Hello zstheorist,

This is because of the default global merge setting of 'Merge back adjusted' at Control Center-->Tools-->Options-->Market data-->Global merge policy. If you create a chart for an expiry month in the future before the rollover date, it loads all historical data from the current expiry month (03-23) and realtime data is received for the future expiry month (06-23).

If you set it to 'Do not merge', it will load all historical data for the future expiry month (06-23). More information on the merge policy setting can be found at the link below.

https://ninjatrader.com/support/helpGuides/nt8/NT%20HelpGuide%20English.html?merge_policy.htm

[zstheorist]

Unfortunately, that is not the answer. I only load and use data with 'Do not merge' set. I check this constantly, because it is useful to 'Merge back adjusted' for some tests, but I am religious about only setting that when I am offline and confirming it is in 'Do not merge' when I load data or run live.
Perhaps this is an IB-only bug, or maybe it would show up in Kinetic. I run 24-hours a day during market hours, so cannot check Kinetic during the week. What happens when you try it on Kinetic? Open a non-expired contract you've never opened before (so there is no database entry yet) and see if it stops at 6 AM. As I mentioned this happens for ALL of them.

[NinjaTrader_Jason]

Could it be the individual merge setting for the instrument is set to 'Merge back adjusted'? Please go to Control Center-->Tools-->Instruments. Double click an instrument and check the 'Merge policy' setting. If set to 'Merge back adjusted', it will override the global merge policy setting. Only if set to 'Use global settings', it will use the global merge policy setting at Control Center-->Tools-->Options-->Market data.

I can't reproduce this when I test it on my end. I connect to Kinetick and create a ES 06-23 15 Minute chart. It loads the requested historical data for the 06-23 expiry. It does not load historical data for just the current day up to 6:00 AM.

[zstheorist]

All futures are set to use Global Settings.
Those are always set to Do Not Merge.
Is this an IB only issue with recent NT 8?
I tried it with BZ 05-23 during open market hours (when live data is coming in). I have record data as live on, and it did not stop at 6 AM.
This then leads back to my original question does something happen when markets are closed for the weekend to limit this?

[NinjaTrader_Jason]

I'm not aware of such a limit during the weekends with Interactive Brokers. I suggest to contact Interactive Brokers and check if they limit the amount of historical data for future expiry months during the weekends.

[zstheorist]

No, they do not limit you to 6AM on the last day of a data request. It is after 4 pm, the futures markets are closed today, and I am still getting the same error with EVERY future I do not already have data for today. Something in the way NT is requesting data from Interactive Brokers is stopping at 6 AM. PLEASE look into this. It is an NT bug. Not an IB bug. Perhaps this is tied to the bug I reported that is deleting the next day data. Is one of your timezone conversion routines wrong, and so cutting the data off?
It is rather suspicious that I am in Chicago (GMT-6) and the data is ending at 6 AM (noon GMT).

[NinjaTrader_Jason]

Can you please check what TWS version you have installed. The required TWS version is 10.19.1c with NinjaTrader 8.0.27.1. If you have a different version, please download and install TWS 10.19.1c from the Interactive Brokers connection guide below and check if the issue persists. You can find it under 'Requirements'.

https://ninjatrader.com/ConnectionGuides/Interactive-Brokers-Connection-Guide

[zstheorist]

As I mentioned, NT is 8.0.27.1. The TWS I was using last Friday was 10.20.1f - the only version that can load data from CBOE ONE as of January 4.
As it is now Sunday (Friday has ended), I can now fill in the rest of the data after 6 AM. This happened the last time as well. I loaded the missing data with 10.20.1, then loaded up the old 10.19.1 and loaded other missing futures data. The other error I pointed out in another bug report (data for next day is deleted on a historical load) was identical between those TWS versions.It is rather unlikely this one is an artifact of 10.20.1f vs 10.19.

[zstheorist]

I tried today to load a chart with ZC 03-23 while the market was closed (1:20-7:00 pm), and data stops at 12 noon (there were trades until 1:20 - I can see them in TWS time and sales). This was done through TWS 10.19.1c (downloaded from NT). So, as suspected, it is not a 10.19 v 10.20 problem. When I went to Historical Data and re-loaded the day, it filled in up to the 1:20 close (after the picture below). This is clearly a problem with data requests while the market in question is closed.

[NinjaTrader_Jason]

Can you please tell me at what time (CT) did you load the chart and at what time did you reload the day in the Historical Data window.

[zstheorist]

Sorry, just saw this. I don't recall the exact time, but it was between 3 and 4 pm CT when I opened the chart (ZC closes at 1:20 pm CT). I tried opening a chart again and it only showed up to the noon time again. A few minutes later I closed the chart and opened Historical Data to take that snapshot. I then immediately tried a historical download (sun 2/26 - Tue 2/28) and it filled in the missing data up to the 1:20 close.

[NinjaTrader_Jason]

Hello zstheorist,

Can you please send me your log and trace files so I can investigate.

• Go to Control Center-->Help-->Email Support.
• Enter your email address in the 'Your email address' field.
• Enter 'Att Jason, 03705137' in the 'Subject' field.
• Make sure 'Log and trace files' is enabled and click Send.

If the 'Email Support' function does not work for you, please follow the steps below to manually attach your log and trace files in an email.

• Open the Documents-->NinjaTrader 8 folder.
• Right-click on the 'log' and 'trace' folders and select Send To> Compressed (zipped) Folder.
• Attach 2 compressed folders in your email.
• Send the email to "platformsupport[AT]ninjatrader[DOT]com". Put 'Att Jason, 03705137' in the subject field of your email.
• Once complete, you can delete these compressed folders.