Merge backadjusted still flawed

[Harry]

Checked MergeBackAdjusted, but unfortunately....

I just checked MergeBackAdjusted with BC, with rollover day today. It is getting better, but there is still a bug.

The old contract is correctly backadjusted, with the exception of the close, which is shown unadjusted. Otherwise it looks good so far, see chart below.

Chart created with default session template IPE Europe futures and system clock set to CET.

[Harry]

Also had this on my second PC. Could you reproduce it?

[NinjaTrader_JoshP]

Harry,

My NT says BC rolls on 4/15 and not 4/13. Please clarify your settings. Thanks.

[Harry]

Hi Josh,

4/15 is the expiry day of the contract and not the roll date. NT's settings for crude are simply false. Volume has already shifted to the new contract. This is the reason, why I changed the settings to 4/13. Rolldates of NT are also false for CL, GC and HG.

So I used 4/13 as roll date and an offset of 0.95.

Harry,

My NT says BC rolls on 4/15 and not 4/13. Please clarify your settings. Thanks.

[NinjaTrader_JoshP]

Harry,

1. I deleted all historical data and all cache.
2. PC clock = CET
3. Start NT
4. Instrument Manager for BC --> 4/13 with 0.95
5. Connect IB
6. Open chart with MergeBackAdjusted

Am I missing a step? Please see attachment where the close looks alright.

[Harry]

Maybe a special case

Yes, that is what I did, except that I did not delete the data in the cache. So the result of my checking this now:

(1) Opened a default chart 65 minutes for BC 06-10. Result is chart one, problem as before.

(2) Reduced the the chart period to 5 days and reloaded the data. This time the close even gets a negative offset instead of the positive one, so it is off by 2* 0.95. The open of the last bar was also moved down.

(3) Closed NT and deleted cache -> db -> cache -> IPE Europe.... -> minute . Reproducible with cache deleted. No offset with 65 days lookback, but false negative offset with 5 days lookback.

(4) I think this is the problem: If I edit the database, the timestamp of the close of BC 05-10 prior to rollover day is 13.04.2010 0:00. This could explain that the offset is not applied. The special case here is that the session ends at 11 PM GMT, which is 0:00 CET local time. The only other instrument that has 0:00 CET as a session close is FOREX, and this does not use MergeBackAdjusted.

However I do not understand why the false red bar moves down further by 0.95, when I switch from 65 days lookback to 5 days lookback.

Everything can be reproduced.

[NinjaTrader_JoshP]

Thanks. Got it to get a big close bar on a similar case as yours from a reload. Forwarding to development to see if they can isolate the overarching issue.

[NinjaTrader_JoshP]

Harry,

I believe we have been able to address this special case scenario. I tested it with BC and GAS with some fake rollovers and it was able to create the last close bar correctly. Should you still have issues in the next beta version, please let us know. Thank you again for your efforts.

[Harry]

Thank you for the feedback, Josh.

Harry,

I believe we have been able to address this special case scenario. I tested it with BC and GAS with some fake rollovers and it was able to create the last close bar correctly. Should you still have issues in the next beta version, please let us know. Thank you again for your efforts.

[Harry]

MargeBackadjusted Not Working

Today, Monday July 19, is rollover day for CL. The new contract is CL 09-10. Global settings are MergePolicy = MergeBackAdjusted. CL uses the global settings.

The contract is not backadjusted, as it should (see chart below)

NT 7.0.0.18. Problem is not related to datafeed, occurs with both Interactive Brokers and Kinetick datafeed. I use Central European Time, it is now 01:00 in the morning.

Explanation

NT does not download the required values for the offsets. The correct offset would be 0.37, however when editing CL, the instrument manager shows an offset of 0 for CL 09-10.

[NinjaTrader_Austin]

Harry, thank you for the report. We will look into this tomorrow.

[aviat72]

Harry, thank you for the report. We will look into this tomorrow.

Please do. I also see what Harry is seeing on my CL charts.

Josh had mentioned that NT is aware of the problem which occurs around roll-over and the next Beta will fix it. As Harry pointed out with the curious nature of CL (rollover in the prior month), perhaps the back-adjustment info is not downloaded on time.

[NinjaTrader_JoshP]

Harry and aviat72,

The offset values are not calculated on our servers the second the values would be available. Calculations are done roughly every 4 hours or so and as such in the beginning of a rollover the offset calculations may or may not have been triggered yet.

Should you need the offset calculations immediately you will want to self calculate them when the servers do not have them available yet. Unfortunately this is a limitation.

[Harry]

This is clearly a bug

As I am not connecting to your data servers but to IB servers, offsets are never downloaded from your servers and are calculated locally from daily data, as explained in the NinjaTrader Documentation -> Editing Instruments. The daily data was available and and despite that there was no calculation of the offset.

My educated guess is that the timing of the offset calculations does not take into account both the time zone setting of the system clock and the session template used for the instrument. So the timing is bad and comes a couple of hours (I would bet it is 6 hours) late.

So if you want to trade the instrument during the first hours of rollover day, you have false data on your charts. This is clearly a bug.

I retested 24 hours later, and the offset is now locally calculated from my daily data as expected.

Harry and aviat72,

The offset values are not calculated on our servers the second the values would be available. Calculations are done roughly every 4 hours or so and as such in the beginning of a rollover the offset calculations may or may not have been triggered yet.

Should you need the offset calculations immediately you will want to self calculate them when the servers do not have them available yet. Unfortunately this is a limitation.

[NinjaTrader_JoshP]

Will get back to you.