yesterday's data all adjusted downwards!

[NinjaTrader_JoshP]

And to be sure we cover all bases, you are on B21 now correct?

[adamus]

I upgraded Sunday from beta 20 just before I became aware of this occurence.

[NinjaTrader_JoshP]

I will throw this on a chart on my computer here and run it for a day and see what I get tomorrow.

[adamus]

You might have to repeat it a few times. My estimate is that this happens approximately 1 in every 5 re-starts.

[adamus]

But then again it might occur straight away, as has happened here.

IB TWS closed itself down - something I normally prevent but things were a bit hectic and I missed my alarm.

On opening it, I have a different type of error - a bad bar. Exactly the same process as before. This time I then reconnected to IB but the bad bar remains - presumably until I redownload the data.

I'll have a look in the historical data to see what it is. Here are some images. It affected 7 out of 11 instruments in my default list.

Looking in the HDM, I can't see any bad ticks but I have a bad 1 minute bar for today 07:00 GMT+1 with the Open and the Low at 1.2871 on the $EURUSD for instance. The previous bar close was 1.2994 and the High is 1.2996'5 and Close is 1.2993.

The bad bar has the same timestamp on each chart where it appears.

Hope that helps,

Adam

[adamus]

update to the last message

I discovered that the bad bars in all the forex data originates from IQFeed.

I subscribe to IQFeed for FXCM data as a substitute for the slow, restricted IB historical data.

I can reproduce the bad bars.

However I cannot reproduce the other bad data and while investigating this issue, just came across another event. See images. Gaps in AUDUSD, NZDUSD and USDCAD.

The gaps appeared after I disconnected and restarted NT. My usual graph showed gaps on these three instruments - all 8 others were fine.

On reconnecting to IB with historical data disabled, the gaps remained. I then connected to FXCM and the gaps were filled but the bad bars appeared.

I downloaded the IB historical data to get rid of the bad bars. The gaps do not appear in the IB data either.

Hope that helps

[NinjaTrader_JoshP]

adamus,

Please go back to the step where you connected to FXCM. After getting the bad bars, please open up your Historical Data Manager and find that bar in the Edit tab. If the data is listed "wrong" in there then unfortunately that is simply the data being pushed down the pipe from your data provider. When you run into such a scenario you have the options to edit that bar in the Historical Data Manager. This is why this feature was added in NT7.

Also, you mention now that you connect to IQFeed as well. How does this play in for your previous reports? Did you connect to IQFeed at all? I recommend simplifying the scenario and just stick to one only for the flatline issue. (Note: My test yesterday was unsuccessful in reproducing.)

[adamus]

I discovered that the bad bars in all the forex data originates from IQFeed.

I subscribe to IQFeed for FXCM data as a substitute for the slow, restricted IB historical data.

I can reproduce the bad bars.

However I cannot reproduce the other bad data and while investigating this issue, just came across another event. See images. Gaps in AUDUSD, NZDUSD and USDCAD.

The gaps appeared after I disconnected and restarted NT. My usual graph showed gaps on these three instruments - all 8 others were fine.

On reconnecting to IB with historical data disabled, the gaps remained. I then connected to FXCM and the gaps were filled but the bad bars appeared.

I downloaded the IB historical data to get rid of the bad bars. The gaps do not appear in the IB data either.

Hope that helps

As I said above, the FXCM data was bad. That is not an issue here.

I cannot stop using IQFeed (as I said above). IB historical data is slow to download and restricted. Plus there are connectivity issues between NT and IB. If I stopped using IQFeed, I would stop using NT.

Anyway, it is probably part of the problem. I have my IB connection set up not to download historical data, so NT must get the history from IQFeed.

The gaps I mentioned above with the chart images were probably down to this.

[NinjaTrader_JoshP]

adamus,

Please then outline the complete exact interaction between both data feeds. This is very important. We will need to know exactly when which feed is connected first, connected second, etc. We need to know exactly which feed is disconnected first and second. We need to know if one feed was left on while the other was shut off.

Please provide exact listed steps. For example:

1. Connect IB
2. Connect IQFeed
3. Open up new 15min chart of $EURGBP on Forex session template
4. Leave running all day
...etc...

The relationship and order in which things are done is very important. Thank you for the information.

[adamus]

When it happens again, I'll note exactly what I did. I'm not 100% sure my recollection of the last event is correct.

[adamus]

Didn't see any anomalies in the data this weekend.

I'll keep checking.

I noted the way I shutdown and restart.

When shutting down on Friday night, I don't disconnect the live feeds manually. I just disable the strategies that are running and then exit NT.

On Sunday evening when I restart, I start two instances of IB - one live, one simulated account - and then I start NT.

Before connecting NT to IB at all, I look through the data from the last week to see if it's OK.

I hook up the live IB account first and then the simulated account. Both account connections are configured not to download historical data.

I then connect the IQFeed.

So that's the way I do it, for the record.

[NinjaTrader_JoshP]

Thanks for the info. I did not see anything out of whack on my end this weekend too.

[adamus]

I experienced an unusual occurrence with the chart updates yesterday.

I had run NT for roughly 18 hours already, and at 18:00 last night, for an hour or more I had not looked at the chart and I switched to it. It was a 5 min chart and when the next bar started as I watched, the price gapped upwards. I thought that was quite bizarre for mid-session and considering the relatively low volatility, and then I saw the times of the bars jumped 50 mins and the chart had just skipped about 10 bars. This was the same across all forex pairs in my default lists, which I cycled through to check.

I looked in the logs but couldn't see anything. There were no connectivity problems, no errors or warnings, nothing suspicious.

Here's the log: TracesAndLogs.zip

Normally I have the market analyzer running to collect the incoming tick data, but this week I forgot to start it or it got closed. Perhaps that would have prevented the issue.

Also suspicious is the secondary data supplier, IQFeed. I have IB connected first but with no historical data allowed, and then IQFeed. I sometimes see that the chart takes the IQFeed live stream, and sometimes IB. This time though I didn't check which was being used by the chart since it's difficult to tell (how could I tell, assuming I didn't know which provider was loaded first?).

[NinjaTrader_Kyle]

adamus,

Which instruments were effected, specifically?

Did reloading historical data clear the gap, or is it still in effect?

[adamus]

Hi Kyle,

reloading the data filled the gap.

It was AUDUSD, EURCHF, EURGBP, EURJPY, EURUSD, GBPJPY, GBPUSD, NZDUSD, USDCAD, USDCHF and USDJPY.