Data loading with NT 7.0.0.17

[NinjaTrader_JoshP]

Unfortunately I am unable to reproduce.

Default intraday chart of minutes is to load 5 days back. The resulting data requests is posted below for CL 07-10 on the session template you used on my end here.

2010-06-14 14:41:20:353 (Kinetick Test) Data.Bars.GetBarsNow: instrument='CL 07-10' from='2010-06-08' to='2010-06-12' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Metals / Energy ETH'
2010-06-14 14:41:20:858 (Kinetick Test) Data.Bars.GetBarsNow: instrument='CL 07-10' from='2010-06-13' to='2010-06-14' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Metals / Energy ETH'

I do not see anything back to 2008.

[Harry]

Example today

Cannot reproduce either, but today this happened, when I opened a default chart (30 min , 175 days) to load intraday data. Loading daily bars, although I opened an intraday chart without indicators. Merge policy = MergeBackAdjusted. Session Template NYMEX ETH.

2010-06-14 14:35:06:312 (Kinetick Beta) Data.Bars.GetBarsNow: instrument='CL 07-10' from='2008-07-19' to='2008-07-20' period=Daily splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Energy RTH'
2010-06-14 14:35:07:203 (Kinetick Beta) Data.Bars.GetBarsNow: instrument='CL 09-08' from='2008-07-19' to='2008-07-20' period=Daily splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Energy RTH'
2010-06-14 14:35:07:656 Cbi.MasterInstrument.UpdateRolloverOffsets: instrument='CL' periodType=Day date=2008-07-21 offset=0
2010-06-14 14:35:07:671 (Kinetick Beta) Data.Bars.GetBarsNow: instrument='CL 09-08' from='2008-08-17' to='2008-08-18' period=Daily splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Energy RTH'
2010-06-14 14:35:08:140 (Kinetick Beta) Data.Bars.GetBarsNow: instrument='CL 10-08' from='2008-08-17' to='2008-08-18' period=Daily splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Energy RTH'
2010-06-14 14:35:08:640 Cbi.MasterInstrument.UpdateRolloverOffsets: instrument='CL' periodType=Day date=2008-08-19 offset=0
2010-06-14 14:35:08:640 (Kinetick Beta) Data.Bars.GetBarsNow: instrument='CL 10-08' from='2008-09-17' to='2008-09-18' period=Daily splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Energy RTH'
2010-06-14 14:35:09:109 (Kinetick Beta) Data.Bars.GetBarsNow: instrument='CL 11-08' from='2008-09-17' to='2008-09-18' period=Daily splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Energy RTH'
2010-06-14 14:35:09:562 Cbi.MasterInstrument.UpdateRolloverOffsets: instrument='CL' periodType=Day date=2008-09-19 offset=0
2010-06-14 14:35:09:562 (Kinetick Beta) Data.Bars.GetBarsNow: instrument='CL 12-08' from='2008-11-17' to='2008-11-18' period=Daily splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Energy RTH'
2010-06-14 14:35:10:203 (Kinetick Beta) Data.Bars.GetBarsNow: instrument='CL 01-09' from='2008-11-17' to='2008-11-18' period=Daily splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Energy RTH'
2010-06-14 14:35:10:656 Cbi.MasterInstrument.UpdateRolloverOffsets: instrument='CL' periodType=Day date=2008-11-19 offset=0
2010-06-14 14:35:10:671 (Kinetick Beta) Data.Bars.GetBarsNow: instrument='CL 01-09' from='2008-12-16' to='2008-12-17' period=Daily splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Energy RTH'
2010-06-14 14:35:11:125 (Kinetick Beta) Data.Bars.GetBarsNow: instrument='CL 02-09' from='2008-12-16' to='2008-12-17' period=Daily splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Energy RTH'
2010-06-14 14:35:11:609 Cbi.MasterInstrument.UpdateRolloverOffsets: instrument='CL' periodType=Day date=2008-12-18 offset=0
2010-06-14 14:35:11:609 (Kinetick Beta) Data.Bars.GetBarsNow: instrument='CL 02-09' from='2009-01-17' to='2009-01-18' period=Daily splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Energy RTH'
2010-06-14 14:35:12:093 (Kinetick Beta) Data.Bars.GetBarsNow: instrument='CL 03-09' from='2009-01-17' to='2009-01-18' period=Daily splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Energy RTH'
2010-06-14 14:35:12:562 Cbi.MasterInstrument.UpdateRolloverOffsets: instrument='CL' periodType=Day date=2009-01-19 offset=0
2010-06-14 14:35:12:578 (Kinetick Beta) Data.Bars.GetBarsNow: instrument='CL 03-09' from='2009-02-17' to='2009-02-18' period=Daily splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Energy RTH'
2010-06-14 14:35:13:046 (Kinetick Beta) Data.Bars.GetBarsNow: instrument='CL 04-09' from='2009-02-17' to='2009-02-18' period=Daily splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Energy RTH'
2010-06-14 14:35:13:515 Cbi.MasterInstrument.UpdateRolloverOffsets: instrument='CL' periodType=Day date=2009-02-19 offset=0
2010-06-14 14:35:13:515 (Kinetick Beta) Data.Bars.GetBarsNow: instrument='CL 04-09' from='2009-03-17' to='2009-03-18' period=Daily splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Energy RTH'
2010-06-14 14:35:13:984 (Kinetick Beta) Data.Bars.GetBarsNow: instrument='CL 05-09' from='2009-03-17' to='2009-03-18' period=Daily splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Energy RTH'
2010-06-14 14:35:14:453 Cbi.MasterInstrument.UpdateRolloverOffsets: instrument='CL' periodType=Day date=2009-03-19 offset=0
2010-06-14 14:35:14:468 (Kinetick Beta) Data.Bars.GetBarsNow: instrument='CL 05-09' from='2009-04-18' to='2009-04-19' period=Daily splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Energy RTH'
2010-06-14 14:35:14:937 (Kinetick Beta) Data.Bars.GetBarsNow: instrument='CL 06-09' from='2009-04-18' to='2009-04-19' period=Daily splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Energy RTH'
2010-06-14 14:35:15:406 Cbi.MasterInstrument.UpdateRolloverOffsets: instrument='CL' periodType=Day date=2009-04-20 offset=0
2010-06-14 14:35:15:406 (Kinetick Beta) Data.Bars.GetBarsNow: instrument='CL 06-09' from='2009-05-16' to='2009-05-17' period=Daily splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Energy RTH'
2010-06-14 14:35:15:906 (Kinetick Beta) Data.Bars.GetBarsNow: instrument='CL 07-09' from='2009-05-16' to='2009-05-17' period=Daily splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Energy RTH'
2010-06-14 14:35:16:375 Cbi.MasterInstrument.UpdateRolloverOffsets: instrument='CL' periodType=Day date=2009-05-18 offset=0
2010-06-14 14:35:16:390 (Kinetick Beta) Data.Bars.GetBarsNow: instrument='CL 07-09' from='2009-06-17' to='2009-06-18' period=Daily splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Energy RTH'
2010-06-14 14:35:16:859 (Kinetick Beta) Data.Bars.GetBarsNow: instrument='CL 08-09' from='2009-06-17' to='2009-06-18' period=Daily splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Energy RTH'
2010-06-14 14:35:17:343 Cbi.MasterInstrument.UpdateRolloverOffsets: instrument='CL' periodType=Day date=2009-06-19 offset=0
2010-06-14 14:35:17:359 (Kinetick Beta) Data.Bars.GetBarsNow: instrument='CL 08-09' from='2009-07-18' to='2009-07-19' period=Daily splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Energy RTH'
2010-06-14 14:35:17:812 (Kinetick Beta) Data.Bars.GetBarsNow: instrument='CL 09-09' from='2009-07-18' to='2009-07-19' period=Daily splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Energy RTH'
2010-06-14 14:35:18:281 Cbi.MasterInstrument.UpdateRolloverOffsets: instrument='CL' periodType=Day date=2009-07-20 offset=0
2010-06-14 14:35:18:296 (Kinetick Beta) Data.Bars.GetBarsNow: instrument='CL 09-09' from='2009-08-17' to='2009-08-18' period=Daily splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Energy RTH'
2010-06-14 14:35:18:781 (Kinetick Beta) Data.Bars.GetBarsNow: instrument='CL 10-09' from='2009-08-17' to='2009-08-18' period=Daily splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Energy RTH'
2010-06-14 14:35:19:250 Cbi.MasterInstrument.UpdateRolloverOffsets: instrument='CL' periodType=Day date=2009-08-19 offset=0
2010-06-14 14:35:19:250 (Kinetick Beta) Data.Bars.GetBarsNow: instrument='CL 10-09' from='2009-09-19' to='2009-09-20' period=Daily splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Energy RTH'
2010-06-14 14:35:19:718 (Kinetick Beta) Data.Bars.GetBarsNow: instrument='CL 11-09' from='2009-09-19' to='2009-09-20' period=Daily splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Energy RTH'
2010-06-14 14:35:20:171 Cbi.MasterInstrument.UpdateRolloverOffsets: instrument='CL' periodType=Day date=2009-09-21 offset=0
2010-06-14 14:35:20:171 (Kinetick Beta) Data.Bars.GetBarsNow: instrument='CL 11-09' from='2009-10-17' to='2009-10-18' period=Daily splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Energy RTH'
2010-06-14 14:35:20:687 (Kinetick Beta) Data.Bars.GetBarsNow: instrument='CL 12-09' from='2009-10-17' to='2009-10-18' period=Daily splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Energy RTH'
2010-06-14 14:35:21:156 Cbi.MasterInstrument.UpdateRolloverOffsets: instrument='CL' periodType=Day date=2009-10-19 offset=0
2010-06-14 14:35:21:156 (Kinetick Beta) Data.Bars.GetBarsNow: instrument='CL 12-09' from='2009-11-17' to='2009-11-18' period=Daily splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Energy RTH'
2010-06-14 14:35:21:625 (Kinetick Beta) Data.Bars.GetBarsNow: instrument='CL 01-10' from='2009-11-17' to='2009-11-18' period=Daily splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Energy RTH'
2010-06-14 14:35:22:203 Cbi.MasterInstrument.UpdateRolloverOffsets: instrument='CL' periodType=Day date=2009-11-19 offset=0
2010-06-14 14:35:22:203 (Kinetick Beta) Data.Bars.GetBarsNow: instrument='CL 01-10' from='2009-12-12' to='2009-12-13' period=Daily splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Energy RTH'
2010-06-14 14:35:22:765 (Kinetick Beta) Data.Bars.GetBarsNow: instrument='CL 02-10' from='2009-12-12' to='2009-12-13' period=Daily splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Energy RTH'

continued...... (not enough space here to show all the requests)

[Harry]

A suggestion for treatment of the problem

If I think about this there is a simple explanation for the problem.

I use MergePolicy MergeBackAdjusted.

When an instrument is first loaded NinjaTrader will try to calculate the offsets according to the rollover dates under instrument settings. To calculate the correct offsets it will first look for daily data, and therefore tries to load daily data for all contract months that have been defined under instrument settings.

Now if contract months are defined several years back, but there is daily data for only half a year, this will cause a lot of error messages. This is particularly annoying for instruments which roll every month.

Example CL -> 23 contract months defined back to 2008 , Kinetick offers daily data for only 6 months -> 17 error messages popping up. With daily data from Interactive Brokers this causes a multiple data throttling problem - which also explains that the correct offsets for the available months are never loaded correctly!!!!

The problem here is that NinjaTrader starts with the oldest contract month defined and works its way up to the most recent. If you do it the other way round, you could stop searching data for calculating offsets with the first contract month that is not available and just create a single instead of 17 error messages popping up.

[NinjaTrader_JoshP]

Your analysis of the scenario is accurate on the current expected behavior when using MergeBackAdjusted. Thank you for the suggestion.

[noincomenojobnoassets]

Hi guys,

I have data loading issues with b17.

When trying to start a 100-day-lookback strategy on TF, NT loads those 100 days over and over and over again and never stops. Eventually runs into IB's historical data limitations like this, of course.

Only happens on TF - other equities (ES, NQ, FESX) load fine. Session times are accurate.

Any ideas? Anyone can reproduce?

Thanks,
ninjna

[NinjaTrader_Bertrand]

ninjna, just tried on my end here on IB and was unfortuantely unable to reproduce your findings here - what merge policy and sessions were you using? Were you loading data triggered by a chart with the strategy applied or the Strategy Analyzer? Is this with IB's TWS 900.6?

Thanks

[noincomenojobnoassets]

Hi Bertrand,

Merge policy -> DoNotMerge.

Session -> Sun 6pm - Mon 6pm, Mon - Thu 8pm - 6pm (session according to the Intercontinental Exchange's HP; there was no matching template in NT).

Strategy start -> From "Strategies" tab in NT, no charts open.

TWS -> 903.7.

Thanks for looking into this.

Regards
ninjna

[NinjaTrader_Bertrand]

ninjna, thanks while it was not exactly reproducible I spotted an issue which is related - we will look into this further.

Also: please ensure to be working with the supported TWS 900.6 release as per our connection guide for Interactive Brokers.

Thanks

[Harry]

NinjaTrader cannot correctly read TF data from Interactive Brokers, if you are located GMT +1 or further East. I am not trading TF because of this. Never worked so far.

Problem is timestamp of the daily bar. This is 0:00 CET next day, or if you are in Athens it would be 1:00 next, anyhow it takes the wrong date, so you will already get tomorrows close

I do not know however, whether your problem is linked to timestamps.

[uncogs]

Hi,
I tried today to switch all my charts from CLN0 to CLQ0
I got this chart, where the data in the yellow rectangle are from Q0, all the others data are from N0.
That is obviously wrong.

I'm using NT 7.17
TWS 905.4
Option DONOTMERGE

Any idea?
Thanks

[NinjaTrader_JoshP]

uncogs,

Please downgrade your TWS to 900. You can get it here: http://www.ninjatrader.com/webnew/su...ivebrokers.htm

This is the only version of TWS we can support.

Please also be sure you reload your chart after you make a switch to DoNotMerge. You can compare the data of your various contracts in the Edit tab of the Historical Data Manager to clearly understand what data is being displayed as well.

[uncogs]

Hi Josh

I've downgraded to 900.6 following your link.
I've erased data of CL 06/10 - 07/10 - 08/10
I've repaired the DB, removed and recreated the instrument CL 08/10

CL 08/10 is now built with CL 07/10 data only.
When I choose CL 08/10, I can see in the control center data for 07/10 beeing downloaded.

Same if I create a CL 09/10, ...built with CL 07/10 data.

In the edit tab of the historical data manager there are no data at all for 08/10 and 09/10

next step ?
thanks

[Harry]

Rollover date as per default for CL 08-10 is June 21, 2010. So with merge policy set to MergeBackAdjusted this is expected behaviour.

If you change the rolldate for Cl 08-10 to June 18, 2010 (instument manager -> CL -> edit -> miscelleaneous - contract months, see lower right corner), your chart will display data for Cl 08-10 for June 18, and show backadjusted data for June 17 and prior. Note: Default rollover dates for NT 7 are still not optimal, as they come late. June 18 is the correct rolldate based on volume crossover.

To see data for Cl 09-10 just set Tools -> Options -> Data -> Merge policy to DoNotMerge

[uncogs]

Harry,
thanks for your help, however as said in my first message and showed in the second screenshoot, I'm allready in DoNotMerge.
My probleme is that I keep getting CL 07/10 data with my CL 08/10 chart, despite using DoNotMerge.

[Harry]

Strange, just tested this. My data provider is also Interactive Brokers, that is why I am answering here. With merge policy set to DoNotMerge, I get a correct chart, see chart below. I have five more ideas what you can try, but no more than that

-> check merge policy under instrument settings, should be UseGlobalSettings
-> check symbol mapping for CL, should be CL||||1
-> use the correct template which is Nymex Metals/Energy ETH (see below)
-> open Historical Data Manager and reload data for CL 07-10
-> go to Own Documents -> NinjaTrader7 -> db -> cache and delete the subfolder that relates to the session template used for your chart

Harry,
thanks for your help, however as said in my first message and showed in the second screenshoot, I'm allready in DoNotMerge.
My probleme is that I keep getting CL 07/10 data with my CL 08/10 chart, despite using DoNotMerge.