Historical Data Manager

[Harry]

Hi Josh, thank you for your answer.

I sent you a mail with the AUDUSD data base and the logfile of February 16.

[NinjaTrader_JoshP]

Harry,

I loaded your exact data and this is what I see.

Looks this way regardless if I connect to IB or not. If I then right click > Reload Historical Data then it fills in the data and I can see the entries in the Historical Data Manager.

Only explanation I can think of for this is maybe you just had some mismatched cache between the 30min and 60min series. Each time you open a chart, it creates a cache. Maybe your 30min cache had data there and 60min was created without it.

[Harry]

Thanks Josh for the reply,

Your data shows the gap according to the data I edited with the historical data manager and the 60 minute chart. But there was data displayed on my 30 min chart, as you said still availabe from the cache. How can that be and where did it come from? Black magic?

By the way had another case yesterday with TF. This was different, as the gap appeared with all types of charts and could also be seen when editing the database. So it least it was a coherent error -> data missing completely, not partially.

Why do these gaps occur, i.e. NT not loading data for a specific period when connected? Can this be related to the session manager, i.e. that NT thinks it has data, whereas actually it hasn't?

[NinjaTrader_JoshP]

Harry,

To clarify, I have data gap in both charts. Which is expected since you don't have the data in your historical data manager. How you achieved your behavior I would not know. You need to reload your historical data to refresh your caches. If you already have a cache it will just use the cache and not load. This is expected behavior.

[Harry]

Reloading is not a problem. I just want to understand, what I can do to avoid this. The gap occured several days ago. If the gap includes a major high or low, all my pivots and fibonacci lines might be false without me even realizing this.....

This happens quite often, so under certain circumstances (which?) Ninjatrader leaves a gap between the historical data in the data base and the new historical data loaded, when I open a chart.

Maybe this occurs because I had loaded the old data using a different session template and there is some internal confusion about the starting point to load? Say I loaded data three days ago with 24/7 now use FOREX -> might cause a gap, but this is mere speculation.

[NinjaTrader_JoshP]

Harry,

Next time this happens, development has informed me to instruct you to delete the contents of the Documents\NinjaTrader 7\db\cache directory and see if it resolves any gap. You will need to close NT first and then restart it when doing this deletion.

[Harry]

Thanks for the hint, will do so and keep you informed.

Harry,

Next time this happens, development has informed me to instruct you to delete the contents of the Documents\NinjaTrader 7\db\cache directory and see if it resolves any gap. You will need to close NT first and then restart it when doing this deletion.

[Harry]

Deleting the cache -> Gaps on both charts.

Reloading data -> All data correctly displayed.

Harry,

Next time this happens, development has informed me to instruct you to delete the contents of the Documents\NinjaTrader 7\db\cache directory and see if it resolves any gap. You will need to close NT first and then restart it when doing this deletion.

[NinjaTrader_JoshP]

Harry,

Unfortunately we are back to square one then. Is there anything you can say about what is happening when NT does not have those bars of data? Were you disconnected for a period of time and reconnected later in the day?

What options do you have checked in Tools>Options>Data?

[Harry]

Thanks for answering, Josh.

I have nothing selected under -> Options ->Data. In particular I am not storing real-time data. I do not remember, whether I was disconnected and reconnected later that. So I understand that you cannot do anything here. I will continue checking my data, and in case that I find any inconsistencies, try to check the log and trace files to find out whether there is a reason.

Harry,

Unfortunately we are back to square one then. Is there anything you can say about what is happening when NT does not have those bars of data? Were you disconnected for a period of time and reconnected later in the day?

What options do you have checked in Tools>Options>Data?

[NinjaTrader_JoshP]

Harry,

What about trying to use "Get data from server"? Maybe that would help?

[Harry]

Sorry Josh,

when I read your reply, I just had NT 6.5. open (which I use for trading) and by mistake I checked -> Data over there. Of course, for NT7 "Get data from server(if available)" is checked. I did not change this.....

Harry,

What about trying to use "Get data from server"? Maybe that would help?

[NinjaTrader_JoshP]

Harry,

Hmm. We will keep our eyes open too. Thanks.

[Harry]

Just connected NT 7.0.0.9 to IB TWS 900/6. Then I tried to toggle instruments for 2 connected charts, toggling from ES to AUDUSD. The charts were a 5 minute and a 30 minute chart, each with several indicators including pivots.

Result 30 min chart loaded: ok
Result 5 min chart loaded: half of today missing, no text below the x-axis.

If you look at the tracefile, you can verify that one request for minute data and 2 requests for daily data have been made. Somehow the different requests interfere in a bad way.

Tracefile

2010-02-23 23:16:58:171 (Interactive Brokers) Data.Bars.GetBarsNow: instrument='$AUDUSD' from='2010-02-23' to='2010-02-23' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0 session='Forex'
2010-02-23 23:16:58:187 (Interactive Brokers) IB.HistoricalDataRequest.Send: last=23.02.2010 instrument='$AUDUSD' from='2010-02-23' to='2010-02-23' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2010-02-23 23:16:58:187 (Interactive Brokers) IB.HistoricalDataRequest.Send: request='AUD;CASH;;0;;;IDEALPRO;;USD;;;1 min;5 D;0;BID'
2010-02-23 23:16:59:578 (Interactive Brokers) IB.HistoricalDataRequest.Process: itemCount=5702 instrument='$AUDUSD' from='2010-02-23' to='2010-02-23' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2010-02-23 23:17:05:843 (Interactive Brokers) Data.Bars.GetBarsNow: instrument='$AUDUSD' from='2010-02-22' to='2010-02-22' period=Daily splitAdjusted=False dividendAdjusted=False bars=0 session='Forex'
2010-02-23 23:17:05:843 (Interactive Brokers) IB.HistoricalDataRequest.Send: last=22.02.2010 instrument='$AUDUSD' from='2010-02-22' to='2010-02-22' period=Daily splitAdjusted=False dividendAdjusted=False bars=0
2010-02-23 23:17:05:843 (Interactive Brokers) IB.HistoricalDataRequest.Send: request='AUD;CASH;;0;;;IDEALPRO;;USD;;20100223 00:00:00;1 day;11 M;0;BID'
2010-02-23 23:17:06:218 (Interactive Brokers) IB.HistoricalDataRequest.Process: itemCount=233 instrument='$AUDUSD' from='2010-02-22' to='2010-02-22' period=Daily splitAdjusted=False dividendAdjusted=False bars=0
2010-02-23 23:17:06:687 (Interactive Brokers) Data.Bars.GetBarsNow: instrument='$AUDUSD' from='2010-02-23' to='2010-02-23' period=Daily splitAdjusted=False dividendAdjusted=False bars=0 session='Forex'
2010-02-23 23:17:07:843 (Interactive Brokers) IB.HistoricalDataRequest.Send: last=23.02.2010 instrument='$AUDUSD' from='2010-02-23' to='2010-02-23' period=Daily splitAdjusted=False dividendAdjusted=False bars=0
2010-02-23 23:17:07:843 (Interactive Brokers) IB.HistoricalDataRequest.Send: request='AUD;CASH;;0;;;IDEALPRO;;USD;;;1 day;11 M;0;BID'
2010-02-23 23:17:08:390 (Interactive Brokers) IB.HistoricalDataRequest.Process: itemCount=233 instrument='$AUDUSD' from='2010-02-23' to='2010-02-23' period=Daily splitAdjusted=False dividendAdjusted=False bars=0

Screenshot below

[Harry]

Continued loading data. For 10 futures contacts this worked, for YM I had a similar problem again as for AUDUSD, as there was a data gap in the chart. ZB 5 min stopped loading at 11AM, ZB 30 min went ahead to the end of the day.

Some observations:

(a) Logfile and trace file showed no errors. For the instruments which were correctly loaded and displayed, the tracefile showed exactly the same messages as for AUDUSD and YM.
(b) Editing the data via the historical data manager showed that data was complete and correctly loaded.
(c) Reloading chart template did not help to display the missing data.
(d) For AUDUSD the 5 min chart was corrupted, for YM the 30 min chart, so it could be either of the connected charts.
(e) When I opened a new 30 min chart, data was correctly displayed. So I had two 30 min charts open at the same time, one correct and one false (see below).


Conclusion: Problem here is not data, but several charts accessing the same correct data in parallel -> I will not use connected charts.