Incorrect Close on Daily bars

[astrolobe]

My Daily bars are usually incorrect after the session closes.

Environment:

1. Kinetick feed
2. PC clock is Australian Eastern Standard Time (AEST)
3. Session template is the default "Forex" template
4. Using RC1 - b23

Details:
As I operate on AEST time zone, when I look at my daily charts after 9am AEST (5pm EST), the bar that just completed is incorrect. The first attachment, "Incorrect Daily.jpg" shows the close of the circled bar as 6.8023...it should be 6.8314.

After reloading the data for the same chart, the bar is now correct, with a close of 6.8314 (see second attachment, "Correct Daily after Reload.jpg").

Using a minute chart, I discovered that the incorrect close of 6.8023 was exactly at midnight AEST (see third attachment, "Minute close at midnight.jpg"). This means the daily bar stopped updating at midnight AEST, instead of 9 hours later at 9am AEST.

Looking at the fourth attachment ("Minute session close.jpg"), you can see that the correct close was 6.8314 at 9am AEST (5pm EST), which the "Reloaded" daily chart shows correctly.

If I were to guess, I think this error may have something to do with the following bug:

4157 Charts - Live charts did not necessarily roll to the next date if opened during a session gap and left overnight

[NinjaTrader_Jason]

Hello astrolobe,

Do you recall what date was listed in the Data Series menu for 'End date'. Did it list yesterday's date?

When I currently create a USDSEK daily chart, I display the proper close time. Can you please tell me if you use the free Kinetick daily feed or do you use a Kinetick feed you purchased?

Am I correct this issue is reproducible each day? You mention it occurs usually - do you notice any consistencies when the issue did not occur?

Do you receive real-time data in the daily chart after 8 AM EST (midnight AEST)?

[astrolobe]

Hi Jason,

When I currently create a USDSEK daily chart, I display the proper close time. Can you please tell me if you use the free Kinetick daily feed or do you use a Kinetick feed you purchased?

I use the purchased Kinetick feed, and I do receive real-time data after 8am EST. If you look at my first attachment in the first post, you will see that the close time is correct, even though the high and close are incorrect.

To answer your other questions, I need to confirm the behaviour by having NT run for at least 24 hours, without shutting down. When I restart NT on a Monday morning (Sunday evening EST), it downloads the data it needs, and the daily chart look correct. So right now my charts look correct. The problem arises from the second day, and continues for the rest of the week ( I keep NT up 24/5).

I will check the charts again on Monday after 5pm EST, when the daily bar has completed, and update this thread.

[NinjaTrader_Jason]

Sounds good, I will await your report.

[astrolobe]

Hi Jason, This is what I have found so far.

Do you recall what date was listed in the Data Series menu for 'End date'. Did it list yesterday's date?

The date listed is for today, for example, right now it is 9:30 am on the 16th in Sydney (AEST), and the Data End Date is the 16th. FYI, the daily bar that is incorrect is the one that ended on the 16th at 9am AEST (5pm on the 15th EST).

Am I correct this issue is reproducible each day? You mention it occurs usually - do you notice any consistencies when the issue did not occur?

Yes, it happens every day except for Monday morning when I restart NT. On a Monday morning when I connect to the Kinetick feed for the first time, it is all good. Today I checked, EURAUD, AUDCAD, AUDCHF, AUDJPY, EURJPY, EURUSD. All of them exhibited the same behaviour, i.e., the close of the previous daily bar is incorrect, as it stops updating after midnight AEST. A reload of the daily chart for each pair corrects the previous daily bar.

Something else I just noticed, which may or may not be related...when I load my 1 minute chart to see if the close price of the incorrect daily bar coincides to midnight AEST, the 1 minute chart has no data from 8am on the previous Saturday (4pm Friday EST). However, the chart is receiving data as it builds the new 1 minute bars from when I opened the chart, but there is a gap from 8am on Saturday AEST. A reload of that chart backfills the gap correctly.

P.S. when you test this, is your computer set to AEST? It should be to reproduce my environment. Also, it needs to run for at least a 24 hour cycle from when NT was started, or from when the charts were reloaded. Lastly, I do not have separate chart instances for each forex pair, I use the same chart and change the pair.

thanks.

[NinjaTrader_Brett]

Hello,

Thanks for that report.

This is Brett following up for Jason.

Most likely what you are expericning here is the session template.

Heres what occurs, Daily Bars have no session template, we essentially use the OHLC that the data feed provider sends out, as theres no way to look into the bar to see what occured when.

When your recieving live data however NinjaTrader can apply a session template to this since it is live data and it can look into the bar.

Therefor what occurs is you have the historical data on one session template that one in use by Kinetick and the session template in use by NinjaTrader for the current live bar which doesnt match the kinetick session template and your expectations.

The reason this is most likely off is you are using the Default 24/7 Session template. This ends at Midnight. 12:00 to 12:00 AM, this is when NinjaTrader will start a new bar.

Therefor you will need to create a new session template to correspond with the close time you are expecting.

Please see the below guide on how to do this, what I would do is just make a copy of the Default 24/7 template and then alter the times to do what you need.

http://www.ninjatrader.com/support/helpGuides/nt7/using_the_session_manager.htm

Then set the new session template on the chart.

Right Click -> Data Series -> Newly Created Session Template with correct closing time.

Let me know if I can be of further assistance.

[astrolobe]

Hi Brett,
Thanks for the response. However, I am NOT using the Default 24/7 session template. See below from my first post:

Environment
Kinetick feed
PC clock is Australian Eastern Standard Time (AEST)
Session template is the default "Forex" template
Using RC1 - b23

Even if I was using the 24/7 template, your following explanation does not explain the behaviour.

The reason this is most likely off is you are using the Default 24/7 Session template. This ends at Midnight. 12:00 to 12:00 AM, this is when NinjaTrader will start a new bar.

If you read my posts, you would see that I am on AEST time. I should not have to restate the problem, so please review post 1 and post 5. I said previously:

Using a minute chart, I discovered that the incorrect close of 6.8023 was exactly at midnight AEST (see third attachment, "Minute close at midnight.jpg"). This means the daily bar stopped updating at midnight AEST, instead of 9 hours later at 9am AEST.

The problem is that my daily bar stops updating at midnight AEST time, not EST. Therefore, the 24/7 theory does not hold, because 24/7 template goes to midnight EST. I think it would help if NT tested this with a computer set to AEST time, using the "Forex" template.
If you do try to replicate this bug, please note the following from post 5:

P.S. when you test this, is your computer set to AEST? It should be to reproduce my environment. Also, it needs to run for at least a 24 hour cycle from when NT was started, or from when the charts were reloaded. Lastly, I do not have separate chart instances for each forex pair, I use the same chart and change the pair.

[NinjaTrader_Jason]

Hello astrolobe,

Did you ever looked at the chart in real-time during midnight? If so, am I correct the daily bar stopped updating as soon as it turned midnight? The reason I ask is because this is not clear to me yet. You stated that you looked at your daily charts at 9 AM AEST and only then noticed that the close price was incorrect.

You verified the close price using a minute chart - the midnight close price in the minute chart was the same as the close price in the daily chart. Did you verify this for only one instance?

Could it be you experience a connection loss around midnight? Can you please check the Control Center-->Log-tab for any connection issues around midnight.

[astrolobe]

Hi Jason,

Did you ever had the chart opened in real-time during midnight? If so, am I correct the daily bar stopped updating as soon as it turned midnight?

The chart is open 24/5, so yes it was open during midnight. It has just gone midnight in Sydney (AEST), and my daily bar actually closed off/ finished, and a new daily bar was started. This is obviously incorrect, as the daily bar should continue until 9am AEST (5pm EST) as I use the "Forex" session. (Although I understand that the session template may not have much bearing on daily bars.) I had not noticed this before, as I had not looked for it, but this makes sense and confirms my findings.

You verified the close price using a minute chart - the midnight close price in the minute chart was the same as the close price in the daily chart. Did you verify this for only one instance?

No, I verified it for multiple instances, including these pairs; EURAUD, AUDCAD, AUDCHF, AUDJPY, EURJPY, EURUSD.

Could it be you experience a connection loss around midnight? Can you please check the Control Center-->Log-tab for any connection issues around midnight.

I checked the log, and there were definitely no connection issues. I don't think I have ever had a Kinetick connection loss.

I have just discovered a new strange behaviour...I never noticed this before because I do not work on NT between midnight AEST and 9am AEST (so right now I am working overtime for you guys). I went back to the forex pairs listed above (in fact all 20 pairs I follow exhibit the same behaviour), and did a reload for each pair. The new daily bar, which incorrectly started at midnight (9 hours too early) is still there, BUT the current bar (the one that should only end at 5pm EST on Wed 17) has dissappeared altogether. So my last few bars go as follows 16, 17, 19 (remember a bar that would close on day n for EST is n+1 for AEST).

I am going to bed now, but I bet that the missing bar will only reappear when I do a reload after 9am AEST (5pm EST). Then the daily bars will all be correct until midight AEST, etc.

[NinjaTrader_Jason]

Hello astrolobe,

I tested it on my end using the AEST time zone and I am able to reproduce the missing bar for the 18th and it does display the current daily bar as the 19th.

I also tested Perth time zone (GMT +8) and modifying the time a few minutes into the future to check what would happen during midnight. However the current daily bar (18th) kept receiving real-time data after midnight. However I did not have NinjaTrader running 24 hours before testing it.

I will forward your findings to Development to further troubleshoot the issue. For now, I suggest to reload historical data each day after 9 AM AEST. Once there is news regarding the matter, I will let you know.

[astrolobe]

Hi Jason,
It is a relief to know that it is reproducible, and not just my environment. Is it ok if I use the HDM to reload all my forex pairs at the same time, and only reload the last 2 days (to save time and bandwidth)?

[astrolobe]

I just did a reload on a handful of charts at 8am AEST (not 9am AEST when a new daily bar should start), to see what happens. I got a very unexpected result...the bar on the 19th, i.e., the incorrect bar that started 9 hours too early at midnight AEST, has also totally disappeared. So now the last two bars are missing. The 19th bar reappears after the next tick.

So far I tested this on 7 pairs, and they all do the same thing.

I still think that the bars will correct themselves after a reload in 1 hour (5pm EST). Just thought this was a strange development.

[NinjaTrader_Austin]

astrolobe, thank you for sharing. We will continue to look into this.

[NinjaTrader_Jason]

Hello astrolobe,

I have been informed that the issue is fixed. Can you please check if you still see the current bar timestamped 1 day into the future tomorrow.

[NinjaTrader_Jason]

Hello astrolobe,

Please disregard my previous post. Unfortunately, it is not fixed in the current release (RC1).

However, the issue is fixed for the production release (R1).