Daily Data False when Merge Policy set to MergeBackAdjusted

[NinjaTrader_Dierk]

BTW: this is TWS 900.X

[Harry]

Dierk,

thank you for your help again. I am on TWS 900. Made some further analysis. There are two problems that may not be related.

(1) The shift by 1 day was created by the session template 24/7 (involuntarily applied as part of my default chart template that recolours the candles). To reproduce simply open a daily chart with session template 24/7. Eventually you need to click below the x-axis and uncompress a little, before the change in the DataSeries becomes effective. Why do session templates affect daily charts?

(2) The backadjusted merging bug for daily data also occurs with 100% default settings. Problem here is that the merging also impacts the current contract after the rollover date. ES 03-10 is off by 1.6%!

Yesterdays high is shown as 1102.75 - the correct value would be 1120.00 - this is off by 17.25 points!

What do you get if you merge ES data backadjusted (standard NT settings)?

[NinjaTrader_Dierk]

1) Expected, since daily bars are timestamped at the end of their session. For future: please stick to exact instructions I provide in order to reproduce issue or let me know where exactly you went for something different. This would save both us time in order to reproduce.

2) Where exactly do you feel NT is off? In order to not get lost in different setups again, please do this:
- backup your <my documents>/NinjaTrader7/db directory
- delete all content of that directory
- make sure no Default chart template is active
- make sure DoNotMerge is set
- connect IB
- instrument manager->ES->roll over dates: for 10th of Dec (03-10) you would see an offset of -5.25
- open chart (chart A) with 2 series:
* ES 03-10, 1 day 25 days back, all default
* ES 12-09, 1 day 25 days back, all default

you will see:
- close at 9th for 03-10 is 1090.50
- close at 9th for 12-10 is 1095.75

-> this matches the above offset of -5.25 exactly

- disconnect from IB
- switch to MergeBackadjust
- open another chart (chart B) of ES 03-10, 25 days back

-> you will see that the bar for 9th (and all bars before) is backadjusted by that exact amount

Please don't try anything different until we are in a 100% agreement that what I described above is what you see.

Based on scenario above: where do you feel something is wrong?

[Harry]

Dierk, thanks for your answer.

I will follow your instructions as soon as I can connect to IB, which would be Monday, and then come back with the results.

[NinjaTrader_Dierk]

I just was able to connect using my live account on TWS 900.X.

[Harry]

Test 1

Dierk you are right, this is the first Saturday in 2009 that I was able to connect!

So following exactly the process you defined, I examined ES 03-10 for the dates Dec 2 (1 week before rollover date), Dec 9, Dec 16 (one week after rollover date) and Dec 24 (last day when CME was open). I only looked at the Close of each day.

Loading ES 03-10 chart -> correct data displayed
Loading ES 12-09 chart -> correct data displayed
Disconnecting
Changing option to MergeBackadjusted

Opening ES 03-10 chart (unconnected): chart is attached

Data is correct for Dec 2 (1102.50 = 1107.75 -offset)
Data is correct for Dec 9 (1090.50 = 1095.75 - offset)
Data is false for Dec 16 (1100.5 , correct value would be 1105.75)
Data is false for Dec 24 (1116.75, correct value would be 1122.00)

The problem is here, that the offset was also applied to the data of the new contract, so the whole data series after the rollover data is false.

Opening ES 12-09 chart (unconnected) -> correct data displayed

I now connected once again to IB to find out, whether this would change the data series:

Opening ES 03-10 chart (connected):

Data is correct for Dec 2 (1102.50 = 1107.75 -offset)
Data is correct for Dec 9 (1090.50 = 1095.75 - offset)
Data is false for Dec 16 (1100.5 , correct value would be 1105.75)
Data is correct for Dec 24 (1122.00)

So when connecting to IB the last day of data is rectified, but all the data between Dec 10 (day after rollover) and Dec 23 remains false.

Opening ES 12-09 chart (connected) -> correct data displayed.

This is not the process that led to my original observations, but I followed exactly your procedure with a new data base. So the error detected here is a bit different. Can you reproduce this?

[Harry]

Test 2

Try to reproduce something similar to the original version of the problem.

(1) Create new DB, set Options -> Data to "DoNotMerge"
(2) Connect to IB
(3) Open daily charts (all default) with a lookback period of 330 days for each ES 03-10, ES 12-09, ES 09-09, ES 06-09

Other problems not related to merging here: Loading hangs at 27 Dec for about 10 sec. If I try to load 365 days, will get no data and error message (see below). For this reason I only load 330 days, because this does not violate IB rules...

(4) Close charts and disconnect
(5) Set Options -> Data to "MergeBackAdjusted"
(6) Open a chart for ES 03-10

Close of last day is 1104.75 - correct value would be 1122.00. So I find again a difference of 17.25 points for the close of 24 Dec, which is identical to my first observation.

This seems to be a cumulated error. I also find the same error for the last trading days of ES 12-09 and ES 09-09,whereas only the data of the first contract ES 06-09 remains correct.

[NinjaTrader_Dierk]

I'm not sure why you saw no value to go through the EXACT (!) procedure as per my post #18 but again tried something completely different.

May I ask you again to try the EXACT (!) scenario I outlined in post #18 in order to isolate the source of trouble in the most efficient way? Based on that scenario: please let me know
- what you observed differently
- what you believe does not work as expected

Thanks in advance.

[NinjaTrader_Dierk]

Based on chart B of the scenario in post #18 I can confirm your finding:
>> (close) Data is false for Dec 16 (1100.5 , correct value would be 1105.75)

This is a bug which will fixed with next update. I suggest holding off any further research beyond scenario in post #18. Thanks for reporting.

Please see also here: http://www.ninjatrader-support2.com/...718#post135718

[Harry]

Hi Dierk,

thank you for replying. Test 1 was the exact procedure as per #18. I did not want to offend you by adding Test 2, which reproduced the original problem that I had found.

[NinjaTrader_Dierk]

Thanks for clarification.

[Harry]

First findings with NT 7.0.0.7

Now tested the backadjusted merging of futures with ES 12-09 and ES 03-10. So these are the results:

It mostly works, which means that the data looks correct for the whole time series with the exception of the two days around the date chosen for backadjustment. So this is an improvement.

However, I would like to point to three problems that seem to be related to both backadustment and the session manager. I use a session template CME US Index Futures ETH, which exactly reflects the electronic session of ES, YM, NQ. So I have tested the data for this template, the CME US Index Futures RTH template and the standard session template 24/7.

These are the bugs that occured:

(1) Standard session template 24/7:

NT7 creates an artificial island, as can be seen on screenshot 1.

(2) Session template CME US Index Futures ETH:

Offset is applied at 0:00 CET and not at the end of the selected session, as can be seen on screenshot 2.

(3) When the session template CME US Index Futures ETH is applied to the old contract ES 12-09, NinjaTrader hides an entire day of data. December 11 is missing. This is shown on screenshot 3.

Please note that also for the session template 24/7, the first six hours of December 11 are missing, as can be seen on screenshot 4. This might be related to my local user time, which is CET.

[NinjaTrader_Dierk]

I replied on the other thread: http://www.ninjatrader-support2.com/...ad.php?t=24038

[Harry]

NT 7.0.0.6 unistalled. Then NT 7.0.0.7 reinstalled. Data base deleted. New data loaded via Historical Data Manager.

As I noted, there is some improvement, but still many bugs.

[NinjaTrader_Dierk]

Please see my post on your other thread. You needed to follow instructions in order to isolate the issues as the DB no longer holds any historical chart data.