Chart freezes when loading data via Historical Data

[Harry]

I am used to updating historical data via the Historical Data Manager (NT 7), or via Historical Data (NT 8).

When I load historical data via NinjaTrader 7

-> chart and SuperDOM continue to work


When I load historical data via NinjaTrader 8

-> SuperDOM works, but chart freezes while data is being loaded


This is not working as expected.

[NinjaTrader_Brett]

When loading historical data from the historical data window. What instrument and data type are you requesting and what instrument is in the chart that is no longer updating?

Reason why I ask since if they are the same instrument/underlying data type and it is not updating as the 'loading...' text then this is expected in NinjaTrader 8. We took special steps to make sure that loading data in the chart is more resilient in NinjaTrader 8 and this includes 'buffering live data' to make it into in progress bar downloads.

-Brett

[Harry]

I had a chart open for 6E 12-16. I would then update my historical data base for the entire current default list list (about 30 futures contracts). 6E 12-16 is included with the default list.

The SuperDOM would show all transactions for 6E 12-16, while the chart would freeze and show "... Loading ..." in the upper left corner. When all historical data was loaded after about 2 minutes, the chart would unfreeze and move in lockstep with the SuperDOM again.

[alexr]

When loading historical data from the historical data window. What instrument and data type are you requesting and what instrument is in the chart that is no longer updating?

Reason why I ask since if they are the same instrument/underlying data type and it is not updating as the 'loading...' text then this is expected in NinjaTrader 8. We took special steps to make sure that loading data in the chart is more resilient in NinjaTrader 8 and this includes 'buffering live data' to make it into in progress bar downloads.

-Brett

'buffering live data' is not work for me I have a gap in all indicators. All graphics are frozen until calculate is at least one of them. In RC2 everything worked correctly.

[Harry]

'buffering live data' is not work I have a gap in all indicators. In RC2 everything worked correctly.

No problems with live data over here.

[NinjaTrader_Brett]

Harry,

What you see is currently expected, the reason is the live data stream is 'buffered' for the next request and we don't continue to throw ticks to the old bar series which will soon no longer exist since the incoming new bar request will replace it. This is a cleaner design from our perspective under the hood but understand that in your use case it's not optimal.

Alexr,

Your case might be unrelated, we would need to take your case from the top so I understand what is going on.

Who is your provider?
Can you give me an example of a gap in data you see?
What instrument / what time frame?
What chart or all charts?
If possible to post a quick screenshot would help me in understanding what your seeing?

Thanks.

-Brett

[Harry]

Harry,

What you see is currently expected, the reason is the live data stream is 'buffered' for the next request and we don't continue to throw ticks to the old bar series which will soon no longer exist since the incoming new bar request will replace it. This is a cleaner design from our perspective under the hood but understand that in your use case it's not optimal.

But if you don't throw ticks this means that the data loading effectively freezes the chart. I would rather have real-time ticks than nothing,

NinjaTrader 7 continues to load real-time data until the historical backfill is complete. And I believe that this solution is a better one, even if it is not clean.

[NinjaTrader_Brett]

Yes understand. Also, just curious why are you reloading data from the historical data window? Is this common operation for you during the day?

[Harry]

Yes understand. Also, just curious why are you reloading data from the historical data window? Is this common operation for you during the day?

First of all this is a habit that I took with NinjaTrader 7. There are basically two reasons.

(1) With NinjaTrader 7 loading daily data via the Add() method did not work properly, because depending on the template used, daily data was not inserted correctly. Therefore some indicators, in particular those that require the settlement price for futures data such as the pivots, are loading daily data asynchronously. This means that the daily data is not loaded automatically, when a pivot indicator is added to a chart. To avoid false data, it is therefore necessary to load daily data for all instruments in the morning of each trading day.

(2) When there is a gap of a few days in your historical data, both NinjaTrader 7 and NinjaTrader 8 will not detect that gap and fill it. In order to avoid such gaps it is best practice to load all data (daily, minute and tick data) every morning. Otherwise it might happen that I do not open a chart for ZS for 6 weeks and then open a chart with a lookback of 30 days. This practice would create a data gap, which is not detected by NinjaTrader.

Updating historical data every morning avoids

- incomplete loading of daily data
- data gaps in the historical data base

[NinjaTrader_Brett]

Thanks for that information.

On 1) That should be working with NT8 now. No need to reload for that reason if you see something there please let me know.

On 2) True that the data load rules only load the requested date range back and starts loading from the provider from the first date detected that is not in the local repository. In a scenario where you have data 40 days back (and never opened that instrument again) and then load a chart 30 days back you would have a gap of 10 days in your local repo, which would be visible if you ever would now open a chart 40 days back. This is on our radar but felt that was a usage scenario came up uncommonly enough in NT7 that we didn't install any fancier reload detection logic into NT8 and kept it simple for now.

[Harry]

On 2) True that the data load rules only load the requested date range back and starts loading from the provider from the first date detected that is not in the local repository. In a scenario where you have data 40 days back (and never opened that instrument again) and then load a chart 30 days back you would have a gap of 10 days in your local repo, which would be visible if you ever would now open a chart 40 days back. This is on our radar but felt that was a usage scenario came up uncommonly enough in NT7 that we didn't install any fancier reload detection logic into NT8 and kept it simple for now.

I think that this is not an uncommon scenario. I am regularly contacted by other traders who have false indicator values on their charts.

Mostly it is a "garbage in - garbage out" scenario. My task is to find the gaps on their charts and ask them to reload data.

I am also getting trapped from time to time. This can happen quickly. You go on a holiday for 5 days, then open a chart with a lookback period of 3 days. Now your chart has a gap that never goes away again, if you don't detect it.

[NinjaTrader_Brett]

Thanks for letting me know, was not aware at the frequency you ran into that. We likely should start tracking that more seriously. I'm tracking this in ID# SFT-1578 for review when we are at our next stage of development were we could look into this further.