Missing Historical MarketData when using Indicator Market Analyzer Columns

[ntbone]

If an instrument is added and then removed from a Market Analyzer with the right timing, you can end up with gaps in market data.

• Its evident if using a 15 minute, or higher time frame.
• I am not sure on the exact timing of when they must be added/removed. Its likely within the same day but it could be a span of days in between.
• I have never seen it affect the current day.
• You must be using an indicator in a Market Analyzer column.
• Increasing the # of days back that the Market Analyzer column searches reduces but does not eliminate the problem.
• Making sure the lines for the trading sessions are visible helps easily spot the issue, for US instruments at least, as it can be very clear that bars are missing on the 15 minute chart
• The missing bars on the 15 minute chart always occur at the end of the day. You may have the first 15 minute bar, the first 2, etc... There can be entire days though missing after that partial day.

The current workaround is to reload historical data. This will fill in any missing gaps.

I do not have reproduction steps at this time as it requires live market conditions to attempt to reproduce the issue. I have reported the problem but have not been able to provide concrete reproduction steps. The issue still exists in 8.1.2.1

[NinjaTrader_Jason]

Hello ntbone,

I can't reproduce it so far on my end. I have a 15 Minute chart and a Market Analyzer with the same instrument. I add an Indicator column in the Market Analyzer. When I remove the instrument in the Market Analyzer, it does not remove historical data in the 15 Minute chart. I will keep testing to try and find a reproducible scenario.

Can you please delete the db-->cache folder and check if you still experience the issue.

• Shut down NinjaTrader Desktop and navigate to the following folder.
• Documents-->NinjaTrader 8-->db folder.
• Delete the "cache" folder that is within the "db" folder.

The db-->cache folder is used to quickly load data, so the first time you load historical data after deleting the db-->cache folder, it will take longer to load than usual.

Start NinjaTrader Desktop and check if the chart shows all historical data when removing the instrument from the Market Analyzer.

[ntbone]

As far as I can tell, NinjaTrader isn't removing market data. It is not correctly downloading market data when an instrument is put back into the Market Analyzer. I believe the chain of events happens as follows.

1. Instrument added to Market Analyzer for the first 2 hours of the session. Say its 2023-12-14 10:00 am EST
2. Instrument is removed from the Market Analyzer. No more market data is being downloaded since the instrument is no longer in use. Instrument is not in any chart.
3. At some point later, the instrument is added back to the Market Analyzer. It would have to be the next day, or several days later. NinjaTrader is collecting market data for it, but its starting from the beginning of a day after 2023-12-14. This might be on 2023-12-15, 20213-12-18, or any day down the line. NinjaTrader is downloading data from beginning of day for either 12-15, 12-18 or whichever day its resumed.

The end result is a gap in the market data. All the data from 2023-12-14 10am to beginning of either 2023-12-15 or 2023-12-18 is going be missing.

• The problem has been happening since earlier this year.
• Clearing the DB/Cache folder doesn't solve the problem unless I clear it every day before I start NinjaTrader. This will impede performance as I will be downloading fresh copies of all market data evey day instead of reusing the cahce/market data that may already be downloaded. The problem always comes back within in a few days of not clearing the cache.
• The days missing are usually more then 3 or 4 days back. Its never the current trading day, and it never seems to be the previous day either, as far as I can tell. Often its 5 or more trading days back.
• Sometimes its missing just part of a day, sometimes it happens multiple times over the course of a given instrument and the reload historical data drastically changes the 15 minute chart.
• The 15 minute charts I look at span 15 to 20 trading days.

I have already worked through this with a support person. I gave them a snapshot of market data that reproduced the issue. There was data missing from the minute folder of the db.

At some point I need to look at all the instruments that reproduce it, figure out when they were added/removed from the market analyzer last and this will give some insight into the timing necessary to reproduce it.

[NinjaTrader_Jason]

When you add (back) an instrument in the Market Analyzer with an Indicator column, it loads historical data as per the 'Load data based on' and 'Days back' and 'Trading hours' settings of the Indicator column. If you add it back several days later and only request 1 day of historical data, a gap could be expected.

If you check your historical data at Control Center-->Tools-->Historical Data after step 3, does it show historical data as requested per the 'Load data based on' and 'Days back' and 'Trading hours' settings of the Indicator column?

Could it be you are running NinjaTrader Desktop multiple days in a row? One of the performance tips is to restart NinjaTrader Desktop daily. Do you experience the issue if you restart NinjaTrader Desktop daily?

You mention the issue can be solved by reloading historical data. Please keep doing this whenever you notice missing historical data in your chart.

[ntbone]

I restart NinjaTrader every day

Consider the two scenarios

Scenario with Charts

1. On Monday, I load AAPL on a chart configured to show me 3 days of data. I then switch the chart to MSFT at 12:00pm EST
2. I keep using NinjaTrader each day of the week restarting daily. AAPL is never loaded into any chart or Market Analyzer during this time.
3. I load AAPL on the chart on Friday 10:00 am EST.
4. I load AAPL on a 15 minute chart set to show me 5 days or more.

NinjaTrader loads all data for AAPL on Friday from Monday at 12:00pm to Friday 10 am.

Scenario with Market Indicator Columns

1. On Monday, I load AAPL into a Market Analyzer with indicator column. The indicator column is configured to load 3 days of data. At 12:00pm EST i remove AAPL and add MSFT
2. I keep using NinjaTrader each day of the week restarting daily. I never load AAPL into any chart or Market Analyzer during this time.
3. I add AAPL back to the Market Analyzer with the indicator column on Friday at 10:00 am.
4. I load AAPL on a 15 minute chart set to show me 5 days or more.

AAPL is now missing the rest of the Market Data for Monday from 12:00pm until 4m EST.

I was using charts to do the monitoring and never had any gaps in historical data for instruments until I switched to using a Market Analyzer. The Market Analyzer is more efficient as it doesn't need to draw the chart and takes up less screen real-estate for me as well. For both the chart and the indicator column, I was loading 1 minute data with the same # of requested historical days.

Regardless of how and when I load an instrument, and regardless of the amount of market data I request, I do not expect there to be a gap in the market data when I request to see the instrument on a 15 minute chart and ask for (in this case 5 days or more ) of historical market data. Having gaps when viewing a larger time frame is unacceptable. It will skew the analysis of the instrument. It doesn't matter whether NinjaTrader loads the rest of Monday in Step 3 from above, scenarios OR loads the rest of Monday data in step 4 when viewing the 15 minute chart.

It is vital that when market data is requested for Monday, on the Friday that all the data for the day is loaded, regardless of how it is requested. NinjaTrader does the right thing when using charts, but is creating gaps in historical data when using Market Analyzers with Indicator columns and then loading the instrument in a chart later.