And more data gaps

[Harry]

Globex session, connected after midnight local time. When connecting after midnight for YM, ES and NQ with Globex session, the data is not correctly loaded. To understand (all in local time)

March 8, 22:15: Session end March 8
March 8, 22:30: Session begin March 9
March 8, 24:00: Day break (1)
March 9, 22:15: Session end March 9 (2)
March 9, 24:00: Day break
March 10, 0:10: Connected

The data loading logic of NT 7 did not load any data between Day Break (1) and Session end March 9 (2). I really do not know what is happening here.

To reproduce:

System clock CET, Globex template, load data for ES, YM or NQ.
Then delete data for March 9 and 10 (CET).
Then close down NT 7.
Then start again and connect.
Load 30 min chart with instrument settings (I have lookback 65 days).

[NinjaTrader_JoshP]

Did you have a Cache?

[Harry]

Do not understand this question. I edited the database, but this confused me, because for ES there was no data and for NQ there was data and both showed the gap. So I did not want to communicate this here. I reloaded the data via Historical Data Manager, and problem was solved.

There is a lots of problems if you connect or disconnect around midnight and/or session break. This shows that the whole logic of session management still has a flaw. I think there are now at least five different bugs related to this in the forum....

Did you have a Cache?

[Harry]

Data from data base not displayed

Now connected to IB today. All Forex instruments show data gaps for yesterday 0:00 to yesterday 23:00 with the standard FOREX session.

The chart below shows that there is no data between 0:00 and 23:00 for March 9 (yesterday). At the same time you can see the Historical Data Manager, which shows 1.425 items for March 9 (1440 minutes - 15 minutes session break) which is a complete day. So data is correctly loaded but not displayed.

Reproducible:

If you load the chart a second time for an instrument, it is correctly displayed. The error occurs only once, the first time a chart is loaded.

[NinjaTrader_JoshP]

Harry,

Please do not confuse other issues with what you are discussing here. If you have no data in the HDM and you see it on the chart this is indicative that you have a cache available for that instrument and you are simply viewing the cached data. In the future if you feel you have weird behavior please stop all operations, do not load anything any more. Do not touch anything at all. Please go to Documents\NinjaTrader 7\db and zip up everything keeping it as pristine as possible. Then please send it in outlining exactly what the circumstances leading up to scenario was. Thank you.

Please also include all workspaces used and inform us as to which workspace contained the issue and on which chart. Please keep everything as untouched as possible.

[Harry]

Josh it was just the opposite way round, the data was in the HDM, but not displayed.

The issue is not workspace related. The chart below is a default FOREX chart with nothing on it. No indicators. Standard FOREX session template as per instrument settings.

Occurs when there is a 1-default-chart-nothing-else workspace with correct data in the HDM.

[NinjaTrader_JoshP]

Harry,

As this is unreproducible on our end, please try to keep everything 100% intact the way it is on your system the moment it occurs. (This may require you to save your workspace though.) Do not reload anything, zip up everything. You might as well do the whole Documents\NinjaTrader 7 directory and please send that in for analysis.

[Harry]

Josh,

my system is clean. No problems. Just when connecting, the data is loaded to DB but not displayed on the chart. When toggling back forth between instruments this goes away, so I need to load every chart twice.

I do not load data via my charts with indicators, but usually

(1) either create a default chart and connect this to the DOM -> then using arrows to toggle through all instruments,

(2) open historical data manager and directly download data.

These are the fastest and safest methods.

You will not find anything on any workspace, tracefile etc. It is the internal logic NT 7 uses to display data. When checking for data already available, it obviously confuses session end and daybreak!!!

[NinjaTrader_JoshP]

Harry,

As stated, this is not reproducible through many many attempts. As such the next step is for you to send in all requested files for analysis the moment you hit the issue. Just because you feel there is nothing wrong with your files does not necessarily mean there is nothing wrong. Furthermore, in the event that there is indeed nothing wrong, having your files will help us isolate differences between your end and our end. Thank you for your cooperation.

[Harry]

Cannot send my workspace with all indicators, but I have a second simple installation of NT7 on another PC with a small data base, so I will try to reproduce it there. This has the advantage that you will know that it is not related to hardware or my installation of .NET 3.5 and that it can be reproduced on a second machine. If I cannot reproduce it there, I will give up. Might take a few days, but I will come back to you.

Harry,

As stated, this is not reproducible through many many attempts. As such the next step is for you to send in all requested files for analysis the moment you hit the issue. Just because you feel there is nothing wrong with your files does not necessarily mean there is nothing wrong. Furthermore, in the event that there is indeed nothing wrong, having your files will help us isolate differences between your end and our end. Thank you for your cooperation.

[Harry]

Same data gaps with NT 7.0.0.11

When opening charts, again gaps occur.

Below you can see two charts. When opening a second time, the data gaps will disappear. This is similar behaviour to the volume spikes that I observed - see different thread. Need to open every chart twice.

[jdfagan]

When opening charts, again gaps occur.

Below you can see two charts. When opening a second time, the data gaps will disappear. This is similar behaviour to the volume spikes that I observed - see different thread. Need to open every chart twice.

I was just noticing this same issue earlier this week (with beta 10) after trying to figure out why one of my backtests did not have same results as identical strat on different computer. It ended up being due to data gaps in one or both of the charts actually (but at different times).

Now I'm trying to remember after initially loading up a chart but before running backtests on it, to first right click and reload all historical data (for both the primary chart and any secondary charts that its referring to via multi-timeframe Add calls in Initialize method). This workaround seems to work but it does seem random why it originally happens in first place.

Yet another hazard to be aware of when backtesting..

Was thinking of writing some type of "data gap check" indicator that would write in bold letters on top of chart when it thinks that this chart's history may have gaps in it..

JD

[NinjaTrader_JoshP]

jdfagan,

Your issue is likely you pulling up a chart and it using cached data and continuing along. Please provide a reproducible scenario.

Harry,

Unfortunately like the Volume spike thread, we cannot reproduce. Please provide more information. At this point in time all I can suggest is for you to start with a 100% blank database and zero cache. Then, write down exactly what times you connect to your data provider, exactly what time you open your chart, exactly what time you close the chart, exactly what time you close your data provider connections. All information is absolutely necessary as to when it leads up to the point where you don't get 3/18 loaded. Periods of time when you are not connect will be critical information to note. Periods of time when you don't have the instrument running is critical to note as well. All information as to when you load data is needed as well. Thank you.

[Harry]

I have started with a blank data base and zero cache on one of my PCs. Have this regularly on two different PCs, one is Dell/XP the other is HP/Vista. Also default charts.

The issue is not caused by missing data in the data base. When editing the data base data covering the gap is there. I have documented this. I have no clue why NT does not display the data.

The only hint -> problem occurs on first loading of a chart. Second loading will be usually correct (toggle to a different instrument and back). This point is common between the volume spikes and data gaps.

Just now while answering, I am opening a chart of YM. Volume spike. Then I toggle to another instrument and back. Volume spike gone. See 2 charts below. Other PC shows same behaviour. The data gaps usually occur between session break and midnight, so I cannot produce them now.

jdfagan,

Your issue is likely you pulling up a chart and it using cached data and continuing along. Please provide a reproducible scenario.

Harry,

Unfortunately like the Volume spike thread, we cannot reproduce. Please provide more information. At this point in time all I can suggest is for you to start with a 100% blank database and zero cache. Then, write down exactly what times you connect to your data provider, exactly what time you open your chart, exactly what time you close the chart, exactly what time you close your data provider connections. All information is absolutely necessary as to when it leads up to the point where you don't get 3/18 loaded. Periods of time when you are not connect will be critical information to note. Periods of time when you don't have the instrument running is critical to note as well. All information as to when you load data is needed as well. Thank you.

[JS999]

A similar problem is being discussed here:

http://www.ninjatrader-support2.com/vb/showthread.php?t=27053

I will note that in my case, switching to another instrument and then switching back does not solve the problem. I have noticed the same thing since NT 6.5 and switching instruments did solve the problem there, but for some reason it didn't work for me today.

I have started with a blank data base and zero cache on one of my PCs. Have this regularly on two different PCs, one is Dell/XP the other is HP/Vista. Also default charts.

The issue is not caused by missing data in the data base. When editing the data base data covering the gap is there. I have documented this. I have no clue why NT does not display the data.

The only hint -> problem occurs on first loading of a chart. Second loading will be usually correct (toggle to a different instrument and back). This point is common between the volume spikes and data gaps.

Just now while answering, I am opening a chart of YM. Volume spike. Then I toggle to another instrument and back. Volume spike gone. See 2 charts below. Other PC shows same behaviour. The data gaps usually occur between session break and midnight, so I cannot produce them now.