yesterday's data all adjusted downwards!

[NinjaTrader_Kyle]

adamus,

Thanks for the clarification.

1.) To confirm, you are using TWS 908.6, correct? (Help > About Trader Workstation)

2.) Please provide me with a copy of your workspace.
You will find workspaces in the Documents > NinjaTrader 7 > Workspaces folder.

[NinjaTrader_Brett]

Hello,

You lost connection in the trace at

2010-09-27 05:27:10:779 (IB) Cbi.Connection.ProcessEventArgs.ConnectionStatusEv entArgs Connected Connected NoError Market data farm connection is OK:cashfarm(2104)
2010-09-27 05:27:10:780 (IB) Cbi.ConnectionStatusEventArgs.Process: Connected Connected
2010-09-27 05:27:16:111 (IB) IB.Adapter.HandleMessage DataFarmConnectionOk


This is the reconnect time however this could explain the missing bars.

[adamus]

That's the wrong time. It was between 18:00 and 19:00.

But anyway then there should be a corresponding disconnected warning both in the bottom right corner of IB TWS and in the logs in NT, which there wasn't. (I'm searching on 'disconnect')

[adamus]

Another problem with the data last night

Last night as I was bringing up my strategies for the week and I ran into another data problem.

I lost the 29th September from my history for all symbols in my default list.

I'll try to detail the situation as it arose last night. Due to fighting other fires to get the strategies going, the whole process took a long time and I ended up too tired to post the message at that point. Hopefully I'll remember everything now.

I started up the PC, and installed NT beta22. I started up NT and connected to IB with the connection that allows historical data requests.

I decided to use IQFeed only in urgent situations or when IB is down, so I didn't connect to it then. So the only connection was the one connection to IB TWS live account.

I let the Market Analyzer window download its required historical data and get up and running on the live feed, which didn't take so long. It seemed that there was no disruption of the IB data feed detailed in another thread.

I switched to the chart window, which was also already opened by the NT startup. It was set to 30min bars on USDJPY with no indicators, only the Chart Trader active. I went through my default list of instruments (11 forex pairs), slowly waiting while each one loaded up historical data from the lame IB historical data server.

I didn't find any problems and I was about to post a msg here saying so, when I noticed suspicious slowness on the internet and I decided to reboot the broadband connection to stabilise it before the week began, hopefully. Since IB and NT should reconnect on their own, I left them running and went to the lan gateway and rebooted.

As the internet connection came back online, NT announced that the connection was lost, followed 5 seconds later by the reconnect.

I switched back to the trading workstation and both IB and NT were connected. However the chart showed that 29th Sept data was gone. On the 30 min bar chart, it went from 2010-09-29 00:00:00 straight to 2010-09-29 22:30:00 without a bar in between.

This was the same on all instruments in my default list. I had to reload all historical data for each one in turn.

Once that was finished, I was able to restart my strategies.

Here's a link to the traces & logs I uploaded: TracesAndLogs.zip

[NinjaTrader_Kyle]

adamus,

Thanks for this - I'll see if I'm able to replicate based on what you've provided here.

[adamus]

Got another one for you.

Here's the TracesAndLogs.zip

and here's the log from IB, which is probably worth looking at: Attachment.zip

and also here's what happened - see image:

Everything has been running 24 hours now almost.

During the day I added several symbols to my default list, including the one that isn't loading now $EURAUD. I loaded these extra symbols into the Market Analyzer. At that point, NT was able to obtain IB historical data, if slowly.

However just 10 minutes ago I changed the instrument on the chart to $EURAUD and NT decided to pull some historical data from IB, but it wouldn't download and repeated attempts failed and I even somehow ran into the "IB pacing violation" despite being unable to do a download.

The question is: is this affecting my strategies? They don't require historical data since they are running live off real-time data, correct? I am monitoring them for any reaction to bad data but so far I haven't detected any problem.

[NinjaTrader_Kyle]

adamus,

Clarify for me what type of strategies you're asking about - ATM or NinjaScript?

[adamus]

I'm asking about ninjascript strategies running in the strategies tab.

I have no idea what's under the hood as far as the data cache is concerned and whether this issue means that the whole data cache for all parts of NT has been foobar'd.

In my experience, if I try to call up any other instruments on this chart, the same bars will be affected and have bad data. Does this mean that the strategies running in the strategies tab will also be looking at bad data?

[adamus]

It might be relevant to your investigations to know that it sorts itself out after 23:00 GMT+1. Up until then, especially in the previous 2 or 3 hours, it's impossible to fetch any historical data.

[adamus]

They're here again

I have another data problem. This time it's about 5 30min bars all adjusted downwards and flat-lining.

I think it was set off by a disconnect with IB. I was trying to reload the historical data on a chart. Since it was after the close tonight I thought I'd reboot. I didn't check any of the charts before rebooting so it might have occurred before that. After rebooting, I saw this problem on about 9 out of 11 of the markets in my default list.

By the way, this reloading historical data means that I can't collect the live data. There is a measurable difference between InteractiveBrokers' live data and the historical data they provide. I've checked it out and tested this.

[NinjaTrader_JoshP]

At what time were you disconnected from IB? You should be able to see the time in the logs for when you were disconnected which would allow you to try and correlate with the chart.

[adamus]

These are the logs:

http://www.ninjatrader-support2.com/upload/20101006_0c2650aa90154ab8b7b8d7e8a42a7bac_TracesAndLogs.zip

Yesterday was a bad day for disconnects.

I'm not sure why you ask what time I was disconnected. As I said, I believe the charts were messed up by that disconnection while I tried to load the chart. I think it was $EURAUD which I had just selected and I saw what I thought was a gap so I did a history reload to check.

Then it disconnected and then I had an unmistakeable error on the chart, so I just restarted.

[NinjaTrader_Kyle]

adamus,

The reason we ask is to check against what is in the diagnostic files, and to determine if this was an expected or unexpected disconnection.

Please update us on the time as you're able.

[adamus]

If I am reading the log correctly, it was 22:50. On the 5th.

[NinjaTrader_Kyle]

Hello adamus,

The disconnection here looks to be unexpected.

In working on my end, I'm not able to replicate the behavior when simply dropping my internet connection to initiate the disconnection.

Would this suffice to replicate the issue on your end?

So that I may further test, please provide me with your workspaces and chart templates.
Workspaces will be found in the Documents > NinjaTrader 7 > Workspaces folder.
Templates will be found in the Documents > NinjaTrader 7 > Templates > Chart folder.