Data missing after midnight starting Jan 18th for LineBreak chart

[JS999]

I am trying to view data traded after midnight tonight (Monday Jan 18th) using a LineBreak chart for a number of contracts, including 6A, 6B, 6C, FDAX, CL, and GC. It is showing no data past midnight and in fact everything is being displayed as one bar. Other tick chart types such as Range, Renko, and Second charts are accurately displaying data, but the LineBreak type is not for at least the contracts that I mentioned. The Base Period type is Second, Base period value is 20, and Line Breaks setting is 3. Session is set to Default 24/7, looking 300 bars back, with the 18th as the end date. LineBreak was working fine on Friday, but there seem to be issues crossing the midnight barrier for some reason... data from Sunday the 17th is fine until 12:00 AM...

[NinjaTrader_Dierk]

Hmm .. this is what I see around midnight as just tried to reproduce. Looks ok to me. My PC is on Mountain time.

[JS999]

Hmm .. this is what I see around midnight as just tried to reproduce. Looks ok to me. My PC is on Mountain time.

OK, just logged back in now in the morning 8:30 EST... essentially completely random behavior on the historical data. As I am switching from one to another, sometimes it seems to load fine, other times the data is gone after midnight.

For example, I switch to 6E... no data after midnight. I press Reload All Historical Data, I get data now properly to 8:30. I switch to 6J, no data after midnight - I press Reload All Historical Data, still no data (this is all on the LineBreak chart so far). I then switch to a Second chart for 6J - data is there. I switch back to LineBreak - gone. I then switch back to 6E as an instrument - no data after midnight again (LineBreak chart). I press Reload Historical data again - this time, nothing. (The first time it worked on 6E when I did that, now it doesn't.)

Completely random behavior (as I speak, CL is working). I am using the settings that I said, connecting to Mirus futures and Zen-Fire. Can you guys load up a chart with all the futures instruments and try changing amongst them, loading things up as I have described and so on, and see what happens? I have always noticed these types of historical data problems with your servers on version 6.5, especially at the midnight crossover at the start of the week - data missing, etc... looks like some of these issues things may have carried over to 7? My suggestion in any case is to really run some very, very thorough tests on this type of thing to see if you can reproduce any of these issues... especially concentrate on the midnight rollover times, and especially on the first day of a new week...

[Alfred]

JS999...

I was seeing somewhat similar sort of behaviour that you describe (single line charts etc...random..) on other charts (not line break)...things behave OK now (so far anyway) for me since using a custom Globex session that cuts out times the market does not trade weekends and between sessions...and this can also be adjusted to temporily cut out holiday spaces the market does not trade.

Maybe give this a shot and see if it makes a difference. Screen shot below on how to set up...the first three lines duplicate for M/F, but after doing that delete the Friday line from 1:15 on....

I was using equadistant set to false usually...

See if this might make a difference....it may have something to do with the way NT currently calcs non trading weekend and holiday spaces.


Thanks...

[JS999]

JS999...

I was seeing somewhat similar sort of behaviour that you describe (single line charts etc...random..) on other charts (not line break)...things behave OK now (so far anyway) for me since using a custom Globex session that cuts out times the market does not trade weekends and between sessions...and this can also be adjusted to temporily cut out holiday spaces the market does not trade.

Maybe give this a shot and see if it makes a difference. Screen shot below on how to set up...the first three lines duplicate for M/F, but after doing that delete the Friday line from 1:15 on....

I was using equadistant set to false usually...

See if this might make a difference....it may have something to do with the way NT currently calcs non trading weekend and holiday spaces.


Thanks...

I have changed my session as you described, cutting out all times over the weekend... doesn't seem to have changed the behavior so far... I guess I will keep experimenting. If the NT guys can try to load up a connection to Mirus and try this out it would be appreciated... (not sure if the broker/feed makes any difference since we're talking about historical data.)

[Alfred]

JS999...

I use Zen Mirus also.

Maybe try a Repair DB and or Reset DB with the new session stuff...
Have also occasionally done a rename the NT.sdf file also to get stuff working...not sure what worked...but with several various custom sessions cutting out weekends now no longer get the single line chart, and seem (so far anyway) to be able to change period and type without erratic problems.


Thanks...

[JS999]

JS999...

I use Zen Mirus also.

Maybe try a Repair DB and or Reset DB with the new session stuff...
Have also occasionally done a rename the NT.sdf file also to get stuff working...not sure what worked...but with several various custom sessions cutting out weekends now no longer get the single line chart, and seem (so far anyway) to be able to change period and type without erratic problems.


Thanks...

Thanks Alfred... I assume that if I delete/rename the .sdf file then that will work the same way that 6.5 did where it just clears all DB data and creates a new database, correct? In any case, I did a repair and it seems to be working a bit better now with the weekends cut out, but I'm not sure if that is due simply to this being now 10 in the morning EST... in the past I have found that NT 6.5 would have historical data problems that would eventually sort themselves out as time went on, so I'm not sure exactly how well the solution is actually working. I will test again on midnight crossover times, and next weekend...

[Alfred]

JS999...

Maybe check to see how stuff behaves after todays holiday gap.

Yes, renaming NT.sdf creates a new DB....not sure what you lose now when you do that though.

Thanks...

[NinjaTrader_Dierk]

OK, just logged back in now in the morning 8:30 EST... essentially completely random behavior on the historical data. As I am switching from one to another, sometimes it seems to load fine, other times the data is gone after midnight.

For example, I switch to 6E... no data after midnight. I press Reload All Historical Data, I get data now properly to 8:30. I switch to 6J, no data after midnight - I press Reload All Historical Data, still no data (this is all on the LineBreak chart so far). I then switch to a Second chart for 6J - data is there. I switch back to LineBreak - gone. I then switch back to 6E as an instrument - no data after midnight again (LineBreak chart). I press Reload Historical data again - this time, nothing. (The first time it worked on 6E when I did that, now it doesn't.)

Completely random behavior (as I speak, CL is working). I am using the settings that I said, connecting to Mirus futures and Zen-Fire. Can you guys load up a chart with all the futures instruments and try changing amongst them, loading things up as I have described and so on, and see what happens? I have always noticed these types of historical data problems with your servers on version 6.5, especially at the midnight crossover at the start of the week - data missing, etc... looks like some of these issues things may have carried over to 7? My suggestion in any case is to really run some very, very thorough tests on this type of thing to see if you can reproduce any of these issues... especially concentrate on the midnight rollover times, and especially on the first day of a new week...

Let's not get confused here:
- opposed to NT6.5 historical data is not stored in the .sdf (6.5:.mdb) file but in various directories under <my documents>/NinjaTrader7/db
- I just toggled a couple of futures you mentioned on a default chart, default session, no template, using 1 minute and 150 tick series (so basically covering any minute and tick based series) -> no issue
- it's important to understand that any intraday chart is either 1 minute or 1 tick based. So to check is "there really is data" go for either a 1 minute chart or a (150) tick based chart. This also might help to square out that issue might be related to some (e.g. line break) bars building logic, which is NT internal
- I'm not aware of any NT6.5 issue related
- I'm not aware of any NT7 session related to tick/minute data after the weekend
- this morning I read on EliteTrader that people complained about "no trading going on" on the 6X instruments at 18th early morning (across all platforms/brokers). Not sure if that's related. Sorry, can't find the link again...

[Alfred]

Dierk...

Thanks for the info re NT7 .sdf file...did not know...

I have found erratic seconds and tick chart behavior (compression etc) seems to be more prevalent (when changing period, type or instrument) when charts are set to equidistant false...and charts seem to work better when set to true....

Though since cutting out weekends and non trading periods with custom sessions things seem to work better with equidistant false on seconds and tick charts.

Will see how it goes this week...only recently got custom sessions ironed out...

Thanks...

[JS999]

Let's not get confused here:
- opposed to NT6.5 historical data is not stored in the .sdf (6.5:.mdb) file but in various directories under <my documents>/NinjaTrader7/db
- I just toggled a couple of futures you mentioned on a default chart, default session, no template, using 1 minute and 150 tick series (so basically covering any minute and tick based series) -> no issue
- it's important to understand that any intraday chart is either 1 minute or 1 tick based. So to check is "there really is data" go for either a 1 minute chart or a (150) tick based chart. This also might help to square out that issue might be related to some (e.g. line break) bars building logic, which is NT internal
- I'm not aware of any NT6.5 issue related
- I'm not aware of any NT7 session related to tick/minute data after the weekend
- this morning I read on EliteTrader that people complained about "no trading going on" on the 6X instruments at 18th early morning (across all platforms/brokers). Not sure if that's related. Sorry, can't find the link again...

Dierk, thanks for the reply. Let's be clear - the issue is not that the data was not being downloaded. I can vouch for that as a fact, because for example Second-based charts were working, as were Range and Renko charts. So it's not surprising that you were able to download and view data with tick-based charts. The problem as you mentioned is almost certainly related to the LineBreak chart type, and since the data did appear properly using the other types, it must actually be some type of charting issue related to the way that the LineBreak bar type is being built in those cases.

There was definitely trading across all these instruments, as I could see the market moving, so that is not the issue. The issues were not as prevalent later in the morning. In any case, maybe you can try testing this again with the LineBreak chart type specifically, with the settings I outlined in my first post.

[NinjaTrader_Dierk]

I just toggled through the 6X instruments with your exact LineBreak setup and noticed that on the 6M I would get way less that 300 bars, although there is sufficient backfill data available. We're looking into that one.

Is that the issue you're experiencing?

[JS999]

I just toggled through the 6X instruments with your exact LineBreak setup and noticed that on the 6M I would get way less that 300 bars, although there is sufficient backfill data available. We're looking into that one.

Is that the issue you're experiencing?

It is possible. Keep in mind that when I was doing this, it was after the midnight crossover. So I was getting no data whatsoever for the contract, apart from one single LineBreak bar being built after midnight. I was seeing all the data before midnight correctly with no problems, however. There was a couple of times when I loaded data that I got nothing, I think, apart from a single bar. This would make sense if the following were true:

1. No bars were being built after midnight
2. The 300th bar back fell AFTER the midnight boundary

In that case, if it wasn't building any of the bars after midnight, then you would see nothing but a single bar I think. You should make sure that the bars you were seeing on 6M were actually from the 18th and not the 17th, because it could be that they were a few bars from the 17th that just showed up because they happened to fall within the criteria of being in the last 300.

I am positive that there is an issue with the LineBreak chart building bars after the midnight boundary, although it is possible that it could be a beginning-of-week issue. I am not sure if you will see it tonight, but you should check.

EDIT: I notice in another thread that a user was mentioning that bars are always cut off at the midnight boundary with a new bar being started... I wonder if this could issue be related to that fact when using the LineBreak type?

[JS999]

Ok, I think I have a rough idea of what the problem is here. Line Break charts basically rely on past bars' data to decide when to build a new bar, as per the description found here:

http://www.stator-afm.com/three-line-break.html

The problem is, as I read in another thread on this forum, I believe that you guys are doing a "hard break" and automatically starting a new bar at the midnight boundary when a new date starts. You can do that with something like minute-based charts, and even something like range-based charts wouldn't be _too_ screwed up by that... but I think that a LineBreak chart might get seriously screwed up by that type of thing. Bottom line, you can't just start automatically building a new bar on the midnight boundary with this chart type - you have to figure out a way to get it to "span" over that boundary and use data from the previous day's last few bars to decide when to start building a new bar. The data you are currently displaying is just plain wrong if you start building a new bar at midnight when the Line Break formula does not call for it, and that might be why the chart type doesn't work across the midnight boundary. Correct me if I am wrong, but I believe that this is the source of the problem.

[NinjaTrader_JoshP]

JS999,

Reason this hard break is done is to allow for the same chart to be built regardless of your starting date. If you use a 5 day lookback chart on 1/19 it would look completely different than a 5 day lookback chart on 1/20 if there was no hard break.