Opening charts often fails to load all data, showing gaps

[adamus]

Since I wrote a routine in my indicator to check that NT7 is loading the bars on my chart correctly (i.e. without gaps except as expected) I see the problem occurs practically every day.

I have 3 charts and they all have the same indicator and the same time-frame and today they all loaded with 5 hours missing from yesterday.

This is a frequent occurence and at the moment it's annoying because my indicator is slow to load meaning it costs me an extra round of time spent waiting for the charts to appear.

[NinjaTrader_Bertrand]

adamus, I believe we already been in direct email contact on this report. The primary data series would be loaded and reloaded according to those rules explained here -

http://www.ninjatrader.com/support/helpGuides/nt7/historical_data.htm

[adamus]

Hi Bertrand,

thanks for answering, and yes you did answer my original email about this issue at the time when the website here was having problems and I couldn't post.

The reason why I am bringing it up again is because it is more serious than I thought.

Originally on the 4th October I emailed you because this problem was only occuring in the secondary time-frame bars in my indicator. This time it is occuring in both those same secondary time-frames and also in the primary time-frame in another chart.

This time it's the same length gap in the same time-frame.

Yesterday it was a different gap.

So I have 4 charts with this instrument, two have the hourly time-frame as secondary bars, and one has it as its primary bars. They all had the same gap.

I re-read your original response to my email and I am now not sure that I understood you completely. I said that an easy work-around is to reload all historical data, but that NT7 should never load data with gaps like that.

So I'm posting again, this time with more info and the experience with the work-around that it is actually time-consuming and therefore annoying!

Maybe in the future I will have a chance to improve the performance of my indicator to speed it up considerably, which would lower the reload times, but until then it seems I am stuck with doing time-consuming reloads regularly because the data isn't loading properly at first.

You did say this will be something that is looked at closer in NT8 but that's why I think I misunderstood you - by my definition, this is definitely a bug that needs registering against NT7 already. Perhaps you already have it recorded.

[adamus]

Actually a correction - I have that instrument and time-frame on 5 charts - I forgot I use it as the basis for an ATR / volatility chart too.

I assume that doing right-click -> "Reload all historical data" will reload it for all chart windows? It seems to be doing that at the moment but I didn't check exactly what I did now.

[NinjaTrader_Bertrand]

Thanks adamus - correct the reload should update all windows where the symbol is used. The loading will happen according the loading rules specified in the helpguide under 'when does NT load data' - this is for your primary / charted series though not for the programmatically added series, if you wish to trigger a reload for those as well either use the Historical Data Manager to Reload All active charts, or work off one true MultiSeries chart where the series is visually present as you trigger the Reload. My comment dones via email still stands in the sense that our next major upgrade will have an enhanced handling here and will allow for more options geared to developers.

[adamus]

OK I just figured out where the misunderstanding is.

We agree that NT7 will load data when I make the connection to my data provider for the first time any trading day, is that correct?

Or do you mean that I should first connect to my data provider, and then open my workspace, rather than the other way round which is the default since my workspace opens when I open Ninja.

This is the situation I am talking about above. These gaps should surely be filled - or are you saying that NT7 will only load the data for today?

In fact the worst case scenario is that I actually have already got the data but somehow Ninja wipes it out - but that's not something I've confirmed yet.

[NinjaTrader_Bertrand]

adamus, what I'm saying is that it will work per the load rules published in the helpguide on the primary series, for the current day it will always reload the data if possible by the provider as it could potentially miss data from the day. If you let NT open the workspace or the other way around, it should not matter on the outcome seen.

[adamus]

Hi Bertrand,

I still think I am not making myself clear on this thread.

I have attached a screenshot. There is a gap for the data from 2013-10-02.

To make this more complicated, this gap only occurs on this chart - obviously where the data is on the primary timeframe.

My other two charts with my indicator that loads the same time-frame as a secondary time-frame using even longer amounts of data do not show this gap. A routine in my indicator checks bar by bar for gaps, and it gave no warning.

This is a situation that is not explained by the Help docs about when data is loaded. The data repository on disk *should* have all data up to today. I have been using the charts every trading day since the start of September and have made no deletions in the HDM.



I can see in the IB logs that Ninja is making multiple requests for data from the 12th and also from the 13th for 5 days of 1min data. Probably I have missed some log statements though.

Perhaps you can answer how much of the chart's days to load are requested when there is history in the HDM that covers the whole period and more? My 60min chart is set to load 20 days, and my indicator requests 25 days.

[Harry]

The gaps usually appear as a result of Interactive Broker's data throttling. The best way to avoid any problems is to load data via the Historical Data Manager.

This is how I proceed when loading data from IB: Start NinjaTrader with an empty workspace or one or two charts. Then connect to Interacitve Brokers. Open Tools -> Historical Data Manager -> Download, then select the default list and download data for the period since my last historical data download (typically 1 or 2 days).

With the gap shown on your chart, please proceed as follows:

-> check via Tools -> Historical Data Manager -> Edit whether the required data is sitting in the data base
-> if data is missing download the data with the Historical Data Manager
-> then go to Documents -> NinjaTrader 7 -> db -> cache and delete all files within that folder
(ATTENTION: Do NOT delete Documents -> NinjaTrader 7 -> cache)

NinjaTrader stores bar series built from minute or tick in that cache folder, in order to increase the speed of loading a chart. If you delete those ready built bar series (which may have been built with incomplete data, due to IB data throttling), NinjaTrader will recreate a new bar series from historical data the next time you open a chart.

[adamus]

Hi Harry,
interesting to hear how you do it. When you point the finger at IB data throttling, are you talking about the occasions when IB throws one of its infamous 'pacing violations'?

That does strike me as very likely. I don't often see pacing violations, but I can't swear I haven't had any earlier this month.

What is inconsistent though is that the data was loaded by my indicator as a secondary time-frame and it didn't complain despite running the bar checking routine I use to flag the situation.

Just for the record, this is the code:

doBarCheck(new TimeSpan(1, 0, 0)); // example

public void doBarCheck(TimeSpan timeSpan)
{
    if (CurrentBar < 1) return;
    if (BarsPeriod.Id == PeriodType.Second
        || BarsPeriod.Id == PeriodType.Minute
        || BarsPeriod.Id == PeriodType.Day
        || BarsPeriod.Id == PeriodType.Week
        || BarsPeriod.Id == PeriodType.Month
        || BarsPeriod.Id == PeriodType.Year) 
    {
        DateTime thisBar = Time[0];
        DateTime lastBar = Time[1];
        TimeSpan difference = thisBar.Subtract(lastBar);
        if (difference.CompareTo(timeSpan) > 0
            && lastBar.DayOfWeek != DayOfWeek.Friday)
        {
            for (int x = 1; x <= (difference.Ticks / timeSpan.Ticks); x++)
            {
                DateTime date = thisBar.Subtract(new TimeSpan(x * timeSpan.Ticks));
                if (date.Day == 25 && date.Month == 12) 
                    return;
                if (date.Day == 1 && date.Month == 1)
                    return;
            }
            if (redAlert == false)
            {
                Log("Chart[" + ChartControl.ParentForm.Text + "] " 
                    + " Indicator: " + logHelper.programName 
                    + " bar#" + CurrentBar 
                    + " " + Bars.Period.Value + Bars.Period.Id 
                    + " @" + Time[0].ToString(logHelper.marketTimePattern)
                    + " is more than 1 bar interval since previous bar! ("
                    + Time[1].ToString(logHelper.marketTimePattern)
                    + "). Should be time span of " 
                    + timeSpan.Days + " days " + timeSpan.Hours + " hours "
                    + timeSpan.Minutes + " mins & " + timeSpan.Seconds + " secs"
                    + ". Try reloading historical data.", LogLevel.Alert);
                cleanUpAllLevels();
            }
            redAlert = true;
        }
    }
}

[Harry]

Hi Harry,
interesting to hear how you do it. When you point the finger at IB data throttling, are you talking about the occasions when IB throws one of its infamous 'pacing violations'?

That does strike me as very likely. I don't often see pacing violations, but I can't swear I haven't had any earlier this month.

What is inconsistent though is that the data was loaded by my indicator as a secondary time-frame and it didn't complain despite running the bar checking routine I use to flag the situation.

Just for the record, this is the code:

doBarCheck(new TimeSpan(1, 0, 0)); // example

public void doBarCheck(TimeSpan timeSpan)
{
    if (CurrentBar < 1) return;
    if (BarsPeriod.Id == PeriodType.Second
        || BarsPeriod.Id == PeriodType.Minute
        || BarsPeriod.Id == PeriodType.Day
        || BarsPeriod.Id == PeriodType.Week
        || BarsPeriod.Id == PeriodType.Month
        || BarsPeriod.Id == PeriodType.Year) 
    {
        DateTime thisBar = Time[0];
        DateTime lastBar = Time[1];
        TimeSpan difference = thisBar.Subtract(lastBar);
        if (difference.CompareTo(timeSpan) > 0
            && lastBar.DayOfWeek != DayOfWeek.Friday)
        {
            for (int x = 1; x <= (difference.Ticks / timeSpan.Ticks); x++)
            {
                DateTime date = thisBar.Subtract(new TimeSpan(x * timeSpan.Ticks));
                if (date.Day == 25 && date.Month == 12) 
                    return;
                if (date.Day == 1 && date.Month == 1)
                    return;
            }
            if (redAlert == false)
            {
                Log("Chart[" + ChartControl.ParentForm.Text + "] " 
                    + " Indicator: " + logHelper.programName 
                    + " bar#" + CurrentBar 
                    + " " + Bars.Period.Value + Bars.Period.Id 
                    + " @" + Time[0].ToString(logHelper.marketTimePattern)
                    + " is more than 1 bar interval since previous bar! ("
                    + Time[1].ToString(logHelper.marketTimePattern)
                    + "). Should be time span of " 
                    + timeSpan.Days + " days " + timeSpan.Hours + " hours "
                    + timeSpan.Minutes + " mins & " + timeSpan.Seconds + " secs"
                    + ". Try reloading historical data.", LogLevel.Alert);
                cleanUpAllLevels();
            }
            redAlert = true;
        }
    }
}

The code does not adjust to the trading day as needed. What is it good for?

[adamus]

The code does not adjust to the trading day as needed. What is it good for?

It checks the time difference between the current bar and the proceeding bar and compares it to the expected time difference.

It allows for gaps preceeding the intra-day session, and weekends and some holidays e.g. Christmas, New Year.

I think I don't understand your question.

[Harry]

It checks the time difference between the current bar and the proceeding bar and compares it to the expected time difference.

It allows for gaps preceeding the intra-day session, and weekends and some holidays e.g. Christmas, New Year.

I think I don't understand your question.

I am mostly trading futures. Futures have a lot of technical breaks intraday, and there are at least 6 days where the exchange is partly closed. Furthermore there are full holidays. But even for FOREX, the trading day does not start at midnight, but at 5:00 PM EST. Without using the appropriate session template and referring to that session template, it is not possible to check any data set for completeness.

But maybe I have not understood what you try to achieve ...

[adamus]

Yes you are right, I am trading forex and I remember I decided to leave out session checks because I really only wanted to use it on hourly or higher time-frames ... so yes there is a lot of work to be done to make it generically useful.

[adamus]

Bertrand,

I have another situation now. Ninja has failed to load the 60min bars from yesterday from 00:00 thro 22:00.

2013-10-15 23:00 is the 50th bar in the bars collection. The 49th bar is also yesterday, but 00:00. So Ninja loaded 49 bars OK, but for reasons unknown, stopped at 2013-10-15 00:00.

This was the secondary time-frame in an indicator on a 3-min chart of the same instrument.

Also on that instrument, I had the same indicator running on 1-min chart, which loaded the full 60min bar history OK, and also on a 60min chart as the primary time-frame, which was also loaded OK.

So how come Ninja left a gap in just one?

Incidentally the last bar from the data provider on the machine was yesterday 2013-10-15 17:00.