Historical Data - puzzlement

[mgabriel01]

As a test - I uninstalled NT 7.0.1000.11 and deleted the entire Ninja Trader 7 folder so no historical data would be retained

I then installed NT 7.0.1000.12 and set tools/options/data/'Save Chart Data as Historical' = Yes


Step 1: Connect to Kinetick and opened a daily chart for ES0313 - 1 Day (Feb 19, 2013)
Open historical data manager and export the data:
Date 2/19/2013
Open 1516
High 1529
Low 1515.25
Close 1528
Volume 1498369

Step 2 Disconnect Kinetick and connect to ZenFire
Open a daily chart for the same day ES0313 1 Day (Feb 19, 2013)
Open historical data manager and export the data
Date 2/19/2013
Open 1519.50
High 1528.75
Low 1519.50
Close 1528
Volume 1161624

Step 3 Check the CME provided daily data for ES0313 1 Day (Feb 19, 2013)
Date 2/19/2013
Open 1516
High 1529
Low 1515.25
Close 1528
Volume 1498174


Any thoughts on the discrepancies with the Zenfire data and the volumes on all 3?
Thanks in advance

[mgabriel01]

As a test - I uninstalled NT 7.0.1000.11 and deleted the entire Ninja Trader 7 folder so no historical data would be retained

I then installed NT 7.0.1000.12 and set tools/options/data/'Save Chart Data as Historical' = Yes


Step 1: Connect to Kinetick and opened a daily chart for ES0313 - 1 Day (Feb 19, 2013)
Open historical data manager and export the data:
Date 2/19/2013
Open 1516
High 1529
Low 1515.25
Close 1528
Volume 1498369

Step 2 Disconnect Kinetick and connect to ZenFire
Open a daily chart for the same day ES0313 1 Day (Feb 19, 2013)
Open historical data manager and export the data
Date 2/19/2013
Open 1519.50
High 1528.75
Low 1519.50
Close 1528
Volume 1161624

Step 3 Check the CME provided daily data for ES0313 1 Day (Feb 19, 2013)
Date 2/19/2013
Open 1516
High 1529
Low 1515.25
Close 1528
Volume 1498174


Any thoughts on the discrepancies with the Zenfire data and the volumes on all 3?
Thanks in advance

Apologies - meant to attach the screenshots
Here they are

[NinjaTrader_Lance]

With Zen-Fire daily bars are recorded in RTH (Regular Trading Hours) rather than ETH (Electronic Trading Hours) to our servers. Kinetick is ETH.

Since they are recorded in RTH they can only be provided in RTH.
To be able to view daily bars with Zen-Fire in ETH you have two options:
1) Open a 1440 minute based chart (1440 minutes in a day) and use an ETH (Electronic Trading Hours), Default 24/7, or Default 24/5 Session Template.
2) Use the Kinetick End of Day free feed and use an ETH (Electronic Trading Hours), Default 24/7, or Default 24/5 Session Template.
The Kinetick is as follows:
• From the Control Center -> File -> Disconnect -> your Zen-Fire connection
• File -> Connect -> Kinetick End of Day
• File -> Connect -> Your Zen-Fire connection
This will now make Kinetick be set to retrieve the data before Zen-Fire.

[Crassius]

With Zen-Fire daily bars are recorded in RTH (Regular Trading Hours) rather than ETH (Electronic Trading Hours) to our servers. Kinetick is ETH.

Since they are recorded in RTH they can only be provided in RTH.
To be able to view daily bars with Zen-Fire in ETH you have two options:
1) Open a 1440 minute based chart (1440 minutes in a day) and use an ETH (Electronic Trading Hours), Default 24/7, or Default 24/5 Session Template.
2) Use the Kinetick End of Day free feed and use an ETH (Electronic Trading Hours), Default 24/7, or Default 24/5 Session Template.
The Kinetick is as follows:
• From the Control Center -> File -> Disconnect -> your Zen-Fire connection
• File -> Connect -> Kinetick End of Day
• File -> Connect -> Your Zen-Fire connection
This will now make Kinetick be set to retrieve the data before Zen-Fire.

I wish we had an option to build daily data from intraday data.

Because the daily data is treated as separate form intraday data, we are stuck with the session template of the data provider.

Session template control is one of the strongest features of NT.

The 1440 minute workaround doesn't work that well, because during rollover there is no minute data sent by my data providor, also there are missing minutes from time to time during the inevitable internet disconnects that occur during a week or month.

Those missing minutes throw off the substitute daily bars and the errors compound and buildup over time..... when you want to know what the 200 day moving average is today from 1440 minute bars, the number returned by the average of 1440 minute bars will be several points above/below the actual number the rest of the market is trading from.

If date were taken into account as well as the session template and number of minutes, we could get daily bars that more closely matched the actual daily bar. The only time they would be off would be those very rare instances when the high or low of the day occurred on a missing minute.

What kills me is that my data provider MB Trading's currency daily bars include an unwanted daily bar for the half day Sunday session.... and then draws six daily bars for one week... Since this is not what the market as a whole looks at when calculating things like the 200 day moving average, the MB daily data is useless to me.

[mgabriel01]

Thanks Lance. Understood
Also helpful to know that the Daily bars are not built from the intraday data.
Is that feature planned in a future release?
Thanks again

[koganam]

I wish we had an option to build daily data from intraday data.

Because the daily data is treated as separate form intraday data, we are stuck with the session template of the data provider.

Session template control is one of the strongest features of NT.

The 1440 minute workaround doesn't work that well, because during rollover there is no minute data sent by my data providor, also there are missing minutes from time to time during the inevitable internet disconnects that occur during a week or month.

Those missing minutes throw off the substitute daily bars and the errors compound and buildup over time..... when you want to know what the 200 day moving average is today from 1440 minute bars, the number returned by the average of 1440 minute bars will be several points above/below the actual number the rest of the market is trading from.

If date were taken into account as well as the session template and number of minutes, we could get daily bars that more closely matched the actual daily bar. The only time they would be off would be those very rare instances when the high or low of the day occurred on a missing minute.

What kills me is that my data provider MB Trading's currency daily bars include an unwanted daily bar for the half day Sunday session.... and then draws six daily bars for one week... Since this is not what the market as a whole looks at when calculating things like the 200 day moving average, the MB daily data is useless to me.

Why not just create a custom Session Template that removes the time span that covers the data that you do not want to see?

[Crassius]

Why not just create a custom Session Template that removes the time span that covers the data that you do not want to see?

It is possible that I am doing something wrong, but I have a template that is:

5PM Sunday to 5PM Monday
5PM Monday to 5PM Tues
5PM Tues to 5PM Wed
5PM Wed to 5PM Thurs
5PM Thurs to 5PM Fri

I would think that the session template system would take the daily bars reported by MB and fit them into this template, giving five daily bars in a week.

But it does not work.

It has been awhile since I tried, but I will do it this week and document what happens, but missing bars and incorrectly drawn bars happen as we cross midnight.

If this is different from what you are suggesting please elaborate.

[NinjaTrader_Lance]

Thanks Lance. Understood
Also helpful to know that the Daily bars are not built from the intraday data.
Is that feature planned in a future release?
Thanks again

The period for the daily data is NOT a feature of NinjaTrader, but a feature related to your data provider. You would need to contact them if you wish to have this data sent to include the ETH.

[NinjaTrader_Lance]

It is possible that I am doing something wrong, but I have a template that is:

5PM Sunday to 5PM Monday
5PM Monday to 5PM Tues
5PM Tues to 5PM Wed
5PM Wed to 5PM Thurs
5PM Thurs to 5PM Fri

I would think that the session template system would take the daily bars reported by MB and fit them into this template, giving five daily bars in a week.

But it does not work.

It has been awhile since I tried, but I will do it this week and document what happens, but missing bars and incorrectly drawn bars happen as we cross midnight.

If this is different from what you are suggesting please elaborate.

The session template here would be dependent on the instrument. In your case you also mentioned not wanting to include Sunday daily data. If that's the case your first line could be the cause of your error.

You may want to start with something like the Default 24/5 template and adjust accordingly. However I'd need to know your goal and instrument name to provide more info.

[Crassius]

The session template here would be dependent on the instrument. In your case you also mentioned not wanting to include Sunday daily data. If that's the case your first line could be the cause of your error.

You may want to start with something like the Default 24/5 template and adjust accordingly. However I'd need to know your goal and instrument name to provide more info.

I want the Sunday data, I just want it in a bar that opens at 5PM Sunday and closes at 5PM Monday... that would be the first "day" of my desired five day, five daily bar week.

I want to avoid having a close of bar at Midnight Sunday so that indicators with CalcualteOnBarClose = true don't consider the Open (at 5pm Sun) to Midnight Sunday as one "day".

The instrument is EUR/USD.

Its been a while since I last gave up trying to get them to draw the way I want them to. I will try this week and post here my results, because I don't remember exactly how or when things go awry.

[NinjaTrader_Lance]

I tested our MB Trading Forex Connection and used the default template (Forex) which spans from 5PM Sunday to 5PM Monday and it is properly displaying the open and the first bar of these week has not closed yet.

Let me know if you see different. (please include your local time zone if you do)

[Crassius]

It will get messed up later in the week....

I am taking detailed notes and screenshots to try to get to the bottom of this... I will keep you posted.