IB & Historical Data in NT7B6

[Harry]

IB Forex charts with spikes on NT 7 (no spikes on NT 6.5.)

If I open intraday FOREX charts with NT 7, I will get spikes for all major currency pairs.

These spikes do neither occur on TWS nor on identical charts downloaded with NT 6.5.

Observed on simple default charts. Any ideas?

[trader65]

Ray ,
let me tell you that...what we asked from long time ago..for NT7 was a simple automatic rollover process for futures .....just change the f***n symbol name for the dataseries ,at volume rollover date with the front month symbol name.....so simple my friend..... and what we got is that stupid and completely unacceptable "merge policy" .
so what you telling me is that my 7 years historical backadjusted database in my TS 2000i ' s GlobalServer its completely useless for use in NT7 ...so there is no reason for me to buy your program.....

sorry for my bad english

kon

Since you have an already merged/back adjusted data file, you are telling NT NOT to merge data. Then as the contract expires, you have to export this data, self adjust the prices and import it back into the front month so that the data values are all correct historically.

If this does not sound like fun, then your best bet is to get the raw data (non adjusted) and import the data into each expired contract month. Leave the merge policy to "MergeBackAdjusted" and NT will always take care of grabbing from the DB, merging and back adjusting... The key is getting the raw unadjusted data --> Which you don't have right now.

[nutz505]

Harry,

I am not seeing any of the spikes you are seeing on NT7 with EURUSD. Attached is a screenshot I took a couple of mins ago on the same timeframe used. I am currently connected via IB TWS 900.6.

Rich

[NinjaTrader_Dierk]

Here is what I get on my setup:
- I nuked all content in <my documents>/NinjaTrader7/db
- Berlin timezone
- start TWS 900.X + NT
- connect IB
- 30 min chart for EURUSD, 24/7 session, no chart template, all default

-> works no problem

[NinjaTrader_Ray]

Ray ,

so what you telling me is that my 7 years historical backadjusted database in my TS 2000i ' s GlobalServer its completely useless for use in NT7 ...so there is no reason for me to buy your program.....

sorry for my bad english

kon

Not at all, import that data into any NT instrument, set the Merge policy to "DoNotMerge", or, just import it into for example, "ES ##-##" and you are good to go.

[Harry]

Spikes appear when using Historical Data Manager

Hi Rich and Dierk,

thank you for posting your observations. This helped me to identify a possibly cause for the problem. I had downloaded the data via the Historical Data Manager, while you possibly just downloaded the data by opening an intraday EURUSD chart. To reproduce, this is what I did:

(a) Delete DB, start with a new DB from scratch, select EURUSD via instrument manager.
(b) Opening a default chart for EURUSD with 30 days lookback: no spikes
(c) Downloading intraday data via Tools -> Historical Data Manager -> Download (Start Date 23.12.2009, End Date 30.12.2009, Minute, Last): spikes reappear

Can you reproduce this?

Sorry, did not mention it, I am also on TWS 900.6 and NT 7.0.0.6

[NinjaTrader_Dierk]

I see some spikes now. Will get back to you. Thanks

[NinjaTrader_Dierk]

The issue is caused by bogus data from IB (e.g. open < low of a bar). NT7 is more sensitive than NT6.5 in that regard. Next update will filter out the corrupted IB data.

Thanks for reporting that issue.

[trader65]

Not at all, import that data into any NT instrument, set the Merge policy to "DoNotMerge", or, just import it into for example, "ES ##-##" and you are good to go.

Ray,

If i set the merge policy to "DoNotMerge".. is that means the data series will lack the automatic rollover feature ?

[NinjaTrader_Ray]

Ray,

If i set the merge policy to "DoNotMerge".. is that means the data series will lack the automatic rollover feature ?

That is correct.

[jackyd]

Ray,

This is both a question and maybe a possible method for migrating NT6.5 historical data to NT7.

What I’ve been testing, for example, is say I’ve got 6 months of ES data in 6.5 that has been merged into the current 3/10 contract month. I import this data into the NT7 3/10 contract, and then update the instrument table for any overlapping expired contract months. So, for the 12/09 contract month, I would change the offset to zero, and change the rollover date to the beginning of my data (ie. June 1, 2009 = six months), then do the same for any other overlapping expired months. By doing this the merge policy (which I have set to MergeBackAdjusted) is not triggered in the past, but will be in effect going forward – which is what I want. And, based on my testing this is working good so far.

But, I understand there is a process that will run at some point to update the instrument table contract rollover dates/offsets.

Is there any information on how often this process will run, and what it will update? If it only updates current/future contract months, then I don’t think I’ll have a problem doing it the way I’ve described.

I know this is working the way I want now, but I’d appreciate feedback on whether I may run into problems during rollovers using this approach (assuming the issue with the MergeBackAdjusted policy is fixed).

[NinjaTrader_Ray]

Ray,

This is both a question and maybe a possible method for migrating NT6.5 historical data to NT7.

What I’ve been testing, for example, is say I’ve got 6 months of ES data in 6.5 that has been merged into the current 3/10 contract month. I import this data into the NT7 3/10 contract, and then update the instrument table for any overlapping expired contract months. So, for the 12/09 contract month, I would change the offset to zero, and change the expire date to the beginning of my data (ie. June 1, 2009 = six months), then do the same for any other overlapping expired months. By doing this the merge policy (which I have set to MergeBackAdjusted) is not triggered in the past, but will be in effect going forward – which is what I want. And, based on my testing this is working good so far.

But, I understand there is a process that will run at some point to update the instrument table contract rollover dates/offsets.

Is there any information on how often this process will run, and what it will update? If it only updates current/future contract months, then I don’t think I’ll have a problem doing it the way I’ve described.

I know this is working the way I want now, but I’d appreciate feedback on whether I may run into problems during rollovers using this approach (assuming the issue with the MergeBackAdjusted policy is fixed).

Very creative solution... I will need to get back to you on the questions about auto updating...I don't recall how exactly its triggered and there is a chance that we might make some changes in this area. I will know more within the upcoming week and be in a position to accurately spell out how it at least will work going forward.

[jackyd]

Thanks Ray, looking forward to the info.

[Harry]

Will NT 7.0.0.7 solve the spike issue for IB Forex data?

[NinjaTrader_Dierk]

Yes, it will.