Please outline steps to repairing historical data gaps for chart

[adamus]

If reloading historical data does trigger the limitation please let me know and we can use another source to load data however this process will only overwrite missing data.

Hi Christopher

I retrieved the data from another machine which also had the data on it, using export/import.

After doing the import in the Historical Data Manager, NT7 was connected to IB and the Daily chart did not automatically refresh - I wasn't expecting it to - so I right-clicked and choose "Reload historical data" to get the freshly imported data to show.

However on re-drawing, the chart did not show any of the fresh data, it was as if the gap in my database was still there. So I checked again in the HDM, and the data had been wiped.

I re-imported it, and did it again. Same result. I figured this was a glitch to do with caching or something, so I disconnected from IB, re-imported the data again and changed symbols in the chart and then changed back. The data was successfully displayed.

I figure you've got a bug there. I can't see that there's any rational explanation why the data should be wiped by the chart historical data reload

[NinjaTrader_ChristopherJ]

Thanks for reporting this. I will attempt to replicate on my end and forward to development.

[adamus]

Chris,
maybe I should have said: I have a large collection of daily, minute and tick data and I have experienced a ton of problems with data integrity with NT7.

Anyway, I checked my records to see what exactly had gone wrong with previous reloads, and it did seem that this was different.

So I tried it and it worked! Hallelulejah!

Well, I had to reduce the number of days I was loading to get past IB's data limitations, but it worked.

However I then scrolled back - I set it to load 200 days of data - and saw that in fact it had not loaded any daily data previous to Mar 2012. I then checked the daily bar chart and saw the same - there is no daily data previous to Mar 2012 on the chart despite having it set to load 1500 days.

I looked in the Historical Data Manager and saw that there is in fact a big gap in the data between Mar 2012 back to Feb 2009.

Hmmm. Where and when that gap happened, I do not know but I have a suspicion that it occured in the Historical Data Manager because I tried unsuccessfully to permanently edit a bad OHLC day from IB, but NT7 keeps overwriting my edit with the bad data. See the chart - there are in fact 2 bad days data. Sept 4th 2012 and Nov 5th 2012.

So what do I do to get the data for 2009 - 2012 back, permanently?



Thanks

Got this data error back again for Sept 04 last year as in image.

There is nothing that I did which I recognised as the trigger or the event which caused it.

I was convinced that it was bad daily data from IB, but when I call up the IB daily bar chart in the IB TWS, the data is good. See attached image.

When I asked around, someone said I shouldn't bother using NT7 daily data because it gets regularly messed up. Someone else then said that they thought this had been solved for NT7.

Do you have any outstanding issues for this? Could you give me an issue number?

Thanks

[NinjaTrader_ChristopherJ]

IB can use different servers that drive data within TWS than data that is provided to NinjaTrader therefore you may see differences when comparing charts such as bad ticks, less data, etc.

NinjaTrader will display all data that is available as it is sent from the broker/data provider. There are no open ongoing cases of data issues at this time with IB.

[adamus]

Hi Christopher,

it doesn't appear to be a data issue with IB - it appears to be a data issue in NT7 historical data management with all data providers.

I understand other people have the same issue using Zen Fire.

[NinjaTrader_ChristopherJ]

I have tested this on my end and prior to testing all historical data was deleted to be sure all new data was being pulled from IB servers.

I have included a screen shot of a Daily EURUSD chart showing the same data for the day in question (Sept. 4th) which indicates this data is stored on the IB servers with this Daily Low.

[adamus]

Hi Chris,

thanks for that screenshot. I can use it to approach IB as evidence.

Do you also see a massive bad tail for Nov 5th - as in my screenshot? This appears to be OK now.

[NinjaTrader_ChristopherJ]

Yes I also show a tail for 11/5 as well as referenced in the attached screen shot.

[adamus]

OK thank you that's great.

I do not show the tail for November 5th though now.

That begs the question: why on earth did NT7 re-request the data for Sept 4th, but not for Nov 5th, as you can see on the attached screenshot?

NT7 has wiped my edit for Sept 5th. See the second attachment - the HDM just about fits onto the screenshot.

[NinjaTrader_ChristopherJ]

Please edit the data for 9/4 within the HDM then be sure to click Save. Next please reload data within your chart and let me know if the issue persists.

[adamus]

IB can use different servers that drive data within TWS than data that is provided to NinjaTrader therefore you may see differences when comparing charts such as bad ticks, less data, etc.

NinjaTrader will display all data that is available as it is sent from the broker/data provider. There are no open ongoing cases of data issues at this time with IB.

Hi Christopher

I seem to be caught between a rock and a hard place here. This is the reply I got from IB:

Initial Description: Error in data provided over API
Response from angelp at 21-Feb-2013

Dear Adam Hardy,

Unfortunately that is not accurate the data sent through the API is identical to the data used in TWS.

Note: When using a third party vendor software many vendors are aware of IB's historical data limitations, because of this they will incorporate real time data into the historical data to provide you more robust charts, the problem is by doing this you are not getting an accurate reflection of the historical data being provided by IB which can cause the differences between NT Charts and IB's Charts when using the same data source.

In this particular case it would be advised to contact Ninja Trader directly for additional assistance. Ninja Trader would need to further investigate on there end why there platform does not accurately display the same charting data as in the TWS charts.

Regards,

[NinjaTrader_ChristopherJ]

Within TWS you have access to for example Second charts are available which will plot historical data however within NinjaTrader Hisotrical Tick data is not available when connected to IB therefore you cannot create Tick, Volume, Range or Second bars that will backfill and will only build bars real time.

It appears within TWS you are currently viewing a Daily chart based off of Midpoint. Within NinjaTrader Forex charts are built based off of the Bid.

When changing the chart within TWS to Bid/Ask as opposed to Midpoint within your chart parameters the low bid points will also be displayed as seen within NinjaTrader and referenced in the attached screen shot.

[adamus]

Hi Christopher,

this all sounds a lot more complicated than I thought. You are right that I displayed the chart in TWS using the midprice, which is the default option.

I changed the config to use Bid/Ask and now in TWS I see bad data too.

However I don't understand what I am looking at in TWS. I can't choose candlesticks, it will only show the chart in OHLC bars and the Open and the Close have the same level, i.e. in this format, every bar becomes a doji. Obviously something is different here, but what

Also, looking in the Historical Data Manager, I have only got data in the Last folder for EURUSD. I don't have anything in the Bid or the Ask folders for daily data. So how come NT7 is showing me bad data and putting it in 'Last' if it is in fact from the Bid?

And in fact in your TWS bid/ask chart, you don't show it but there is bad data on 2013-01-08, but it does not show up as bad data on the NT7 chart. What's going on there?

I will use your screenshot to try to get IB to do something.

[NinjaTrader_ChristopherJ]

NinjaTrader will store this data within the Last folder however is driven off the Bid price.

The spike within IB on 1/8 is based off the Max Ask price which is not used within NinjaTrader.

[adamus]

OK I see the problem.

However I would like to point out that it is incomprehensible why you would want to do that. If you get the bid price, then put it in the Bid folder - why store nothing there? And why store it in Last, when it's not Last?