And more data gaps

[NinjaTrader_JoshP]

Harry,

You say between session break and midnight.

Is this as in you start up your chart during that time? Or is it that you have a chart running in real-time through that time and then the next chart you open happens to be that way?

Let us concentrate at one case at a time. For this thread, let us concentrate on the gap instead of volume spike as they are likely two separate things. Kyle should be working with you on the spike one in another thread.

[Harry]

Agree Josh, will continue volume spikes in the other thread. It just that gaps and volume spreads share something: go away if I open the same chart a second time! Volume spikes also occcur really often.... gaps are more difficult to produce.

I cannot fullly answer your question regarding gaps. But usually I trade one instrument and then update all instruments later. I will pay attention to this next time, and asnwer your question then.

Harry,

You say between session break and midnight.

Is this as in you start up your chart during that time? Or is it that you have a chart running in real-time through that time and then the next chart you open happens to be that way?

Let us concentrate at one case at a time. For this thread, let us concentrate on the gap instead of volume spike as they are likely two separate things. Kyle should be working with you on the spike one in another thread.

[Harry]

Comes faster than expected, just opened a chart to test the "Copy and Paste" problem. Here is my chart: one entire day of data missing. I was connected for several hours, and just opened the chart now with template 24/7 for both data series.

The chart is 24/7 for both dataseries, because I wanted to test whether this has an impact on the drawing problem.

Now this is a bit special, because usually I do not change session templates, but here comes a gap with session 24/7. Now today is rollover day for FDAX and FESX, maybe this is too much for NT7.

Data for march 17 is missing, this should be taken from last contract. I edited the data for March 17 - is all complete for both contracts.

[Harry]

And this is the db cache at the very moment, when the chart was opened. But I must say, that this cache was probably newly created, as I did not use the template 24/7 during the last weeks (always take instrument settings which is EUREX).

[NinjaTrader_JoshP]

Harry,

Thank you for your time on the phone. Can you please send me your exported data for that FDAX 06-10 instrument when you get a chance as discussed over the phone? Thank you.

[NinjaTrader_JoshP]

Harry,

Back to the original issue.

Development has provided some guideline questions which *may* help us isolate a little bit. If you could enlighten us as to the answers that would be appreciated.

- Does this happen in the middle of a requested series, or on the edge (like border historical/live session; futures - on different contracts? [If not using futures, then border historical/actual session likely]
- Session definition always your Globex which is now 100% identical to the US Index ETH template?
- Is there more than 1 session definition per day in the user’s session?
- When does that happen? Before session on (local) calendar date, after session begin of calendar date, after session end on calendar date. This is important.
- Does this only happen on requests including actual date, or also historical only requests?

[Harry]

Hi Josh,

here are some answers:

- Does this happen in the middle of a requested series, or on the edge (like border historical/live session; futures - on different contracts? [If not
using futures, then border historical/actual session likely]

=> If you look at posts #1, #4, #11 of this thread you can actually see what is happening:

Usually I update my database after session close by opening a simple chart and toggling through the instruments. Let us take a typical case. Last update for YM was yesterday after the session close (early morning local time), so the historical data already included a few bars of today's session prior to my update. When doing the update, NT loaded all historical data (as checked via Historical Data Manager), but only displayed old data and the historical bars after today's session close.

So only the fraction of today's session was displayed that already was loaded in the early morning.

Possible Interpretation: NT "thinks" that - because only today's data was loaded - it is sufficient to add data to the current session and not the previous one. This I do not know of course.

- Session definition always your Globex which is now 100% identical to the US Index ETH template?

I am not sure that my sessions have been correctly updated after I installed NT 7.0.0.12 -> I am still seeing those funny double sessions, which I do not use. The session template I use for Globex is Sun 17:00 - Mon 15:15, Mon 15:30 - Fri 15:15, all Central Time.

- Is there more than 1 session definition per day in the user’s session?

No. Five sessions per week. All charts use instruments settings.

- When does that happen? Before session on (local) calendar date, after session begin of calendar date, after session end on calendar date. This is important.

It usually happens after Globex session close (22:15 CET) but before midnight local time.

- Does this only happen on requests including actual date, or also historical only requests?

I do not know, because I did not do any historical requests not including today.

[Harry]

Reproducible between session break and midnight

Interesting enough, I can just reproduce now.

Because of different daylight saving rules, session close is at 21:15 CET today. It is now 21:45 CET, so after session close for Globex instruments, but still prior to midnight local time.

So I opened charts for YM, NQ, 6E, 6J, 6B. The charts for the currency futures show no gaps, as the session closes only in 10 minutes.

YM and NQ show a huge gap. Charts attached below. It is now after the session close. Data prior to session close is missing. Again I have checked the Historical Data Manager -> Edit. Data for YM and NQ is complete and covers the gaps as well. If I reload charts for YM/NQ a second time, the gaps disappear.

For ES I had charts open during the last hours, so the data shown was real-time data. To check ES I disconnected NT7 from IB and reconnected.
This procuced quite an interesting result:

1 ES chart (30 min) produced a gap. The gap covers the period starting at connection this afternoon (had loaded historical data for ES then) and ending at session break.

The 2 other charts (5min, 120 min) show no gap. I guess that the 30 min chart was displayed first, so it inherited the gap.

Hope this helps.

[NinjaTrader_JoshP]

I am not sure that my sessions have been correctly updated after I installed NT 7.0.0.12 -> I am still seeing those funny double sessions, which I do not use. The session template I use for Globex is Sun 17:00 - Mon 15:15, Mon 15:30 - Fri 15:15, all Central Time.

You will have to reset your database to get the new templates. What do you mean by B12? Did you mean B11?

So I opened charts for YM, NQ, 6E, 6J, 6B. The charts for the currency futures show no gaps, as the session closes only in 10 minutes.

YM and NQ show a huge gap. Charts attached below. It is now after the session close. Data prior to session close is missing. Again I have checked the Historical Data Manager -> Edit. Data for YM and NQ is complete and covers the gaps as well. If I reload charts for YM/NQ a second time, the gaps disappear

So, chart opened at what time? Prior session close? Or prior midnight but after session close?

For ES I had charts open during the last hours, so the data shown was real-time data. To check ES I disconnected NT7 from IB and reconnected. This procuced quite an interesting result:

Chart closed and then reopened or left running the whole time? Did you right click -> Reload Historical Data or only let it do the loading as it should automatically do when reconnecting. Was there any time gap between disconnect and reconnect?

[Harry]

(1) NT 7.0.0.11 , of course.

(2) Opened charts for YM and NQ after the session close. Did not have any charts open for YM and NQ during the day.

(3) Connected to IB about 19:00 CET, had three charts open for ES since. So charts were displaying historical data prior to 19:00 and real-time data after 19:00 CET. Stayed connected until after the session close.

To check for ES gaps I needed to load historical data. So I disconnected - after session close - from IB (3 charts for ES remained still open) and then reconnected immediately. Upon reconnection NT7 loaded historical backfill for ES from 19:00 CET until the time of disconnection/reconnection. The 5 min and 120 min chart correctly showed the backfill loaded from 19:00 CET until reconnection. The 30 min chart however showed a gap between 19:00 and session close at 21:15 CET (1 hour early because of different start dates for DST), the newly loaded backfill was only displayed for the new session that started at 21:30 CET.

So one of the 3 ES charts showed a gap, 2 were complete.

[Harry]

After resetting the database I stiill have that funny double session template for CME US Index Futures ETH.

You will have to reset your database to get the new templates.

[NinjaTrader_JoshP]

You will need to Reset and restart NT.

[Harry]

Have reset and restarted NT several times. Double session with break for CME US Index Futures is still there....

You will need to Reset and restart NT.

[NinjaTrader_JoshP]

Please delete your database and try again (Documents\NinjaTrader 7\db\NinjaTrader.sdf). If you still have it after this then there is clearly something wrong with your setup and I suggest a 100% clean reinstall.

[Harry]

Thanks Josh, this worked. So I conclude

Tools -> Options -> Data -> Reset DB does not reset session templates.
Deleting NinjaTrader.sdf does reset session templates.

Is this as expected?

Please delete your database and try again (Documents\NinjaTrader 7\db\NinjaTrader.sdf). If you still have it after this then there is clearly something wrong with your setup and I suggest a 100% clean reinstall.