IQFeed errors on downloading historical data

[kdoren]

With NT7 B15 I still see the following error sometimes when downloading historical data from IQFeed:

Error on reloading Data: E,Could not connect to History socket. (Panic)

Here is the corresponding NT7 log entry:
4/29/2010 10:38:36 PM|0|4|Error on reloading data: E,Could not connect to History socket. (Panic)

I see it on both the 32-bit and 64-bit versions. No other applications are connected to IQFeed at the time. I tested this late at night, when markets are closed.

When downloading 3 weeks of 1-min & daily data for the instrument list "SP 500", the error popped up 10-20 times.

The error message is extremely unhelpful. I would like to know which symbol(s) failed to download, so I could download just the ones that failed.

[NinjaTrader_Bertrand]

Thanks for the report and feedback Kevin, is this still on B15 with IQFeed client 4.7.0.9?

Would you mind sending us full logs / traces for review via Help > Mail to Support?

Thanks in advance.

[kdoren]

Yes, B15 with IQFeed client 4.7.0.9. Same problem existed on B14 and previous builds. I'll email a log file.

[NinjaTrader_JoshP]

kdoren,

Is it only E that gives you this? Can you please try pulling up a single chart of it? What are your Instrument Manager settings for it? What are the symbol mappings? Thanks.

[kdoren]

There is no symbol "E" being downloaded, that is the exact same message I get when a download fails, no matter what symbol is being downloaded. I presume the "E" in the error message is a bug.

This particular error occurred in:

Tools -> Historical Data Manager -> Download -> Instrument List = SP 500 or NASDAQ 100, Start Date = 4/12/2010, End Date = 4/28/2010, Intervals = Minute + Day, Data Types = Last -> Download

I got the error 15 times when downloading SP 500, 2 times on NASDAQ 100.

[NinjaTrader_JoshP]

kdoren,

That is fine. I am interested in the exact instrument you are running into the issue on. Or are you saying the instrument changes?

Also, since Bertrand is out of the office, do you have the ticket # you sent in the log/trace files to? Thank you for the assist.

[kdoren]

Josh, I just sent an email {312323} to support with the NT7 & IQFeed log files.

The 4 issues detailed are:

1) IQFeed throws occasional errors when downloading data:
Error on reloading data: E,Could not connect to History socket. (Panic)
There is no symbol "E" being requested, this looks like a bug in IQFeed's error message which is passed to NT7.

2) NT7 requests data even though the data is in history. I do have the box checked for "Get Data From Server (If Available)", because if I leave it unchecked, NT7 never requests historical data.

3) NT7 requests the same data over and over again. I have a strategy which I run on the instrument list "NASDAQ 100", which adds minute & daily data series "QQQQ" in the strategy. NT7 appears to request historical backfill for each QQQQ series for each instrument in the list - 200 requests for QQQQ when I backtest the strategy over the list "NASDAQ 100".

4) Every time I connect to IQFeed, I get the error: IQFeed news feed is not enabled. But when I run the IQFeed test application, the news test works fine.

#1 could be an IQFeed issue, but 2 & 3 & probably 4 would seem to clearly be NT7 issues.

Thanks!

[NinjaTrader_JoshP]

kdoren,

Development talked to IQFeed about this error and this is what they had to say:

-Message is returned by IQFeed any time a connection is unable to be establish to the history server
-Anything that could cause a connection fail like firewall,anti-spyware, dropped internet packets, etc. could cause this
-IQFeed's own support on this usually goes down along the lines of oversaturated bandwidth usage on the user's network causing "drops". Sometimes it would be firewall related.

[kdoren]

That's a really bogus answer from IQFeed. Their protocols can't handle dropped packets? The rest of the world figured out how to handle dropped packets 40 years ago.

I'm seeing this problem even today, a Saturday, when loads are light. I disabled firewall/antivirus/antispyware/etc but it makes no difference.

In looking through the IQFeed log files, NT7 appears to make a flurry of requests once a minute for news headlines, even when NT7 is otherwise idle. Sometimes these requests trigger an error message:

STATUS (LookupError)=2010-05-01 13:35:20: News Connect Error SendRequest (66.112.156.211:60018). Error:10060.
TO CLIENT (LookupError)=2010-05-01 13:35:20: !ERROR! Could not connect to News socket.

I wonder if the news requests are somehow interfering with downloading historical data, but I can't find a way to disable them. Is there some way to shut them off?

Issues #2 and #3 (and possibly #4) are clearly NT7 issues, please look into them.

Also, when historical downloads fail, it would be very useful if the error message would say which symbol(s) failed. Then I could just re-download the ones that failed. Or maybe there is a better way for NT7 to handle the error, and re-download automatically.

Thanks.

[NinjaTrader_JoshP]

2. NT loads data when any of these are scenarios are applicable: http://www.ninjatrader-support.com/H...HistoricalData

3. Each backtest is ran independently from each other. When you run it and one of the scenarios in the above link is hit it will load data for the instruments even if it just loaded it for a prior test.

4. Will ask development about the news feed.

[kdoren]

2. NT loads data when any of these are scenarios are applicable: http://www.ninjatrader-support.com/H...HistoricalData

I just ran a few tests. One of my strategies always causes data to load from the server, even though the data is all in history. I have other strategies that work as you describe. I can't see (or imagine) anything in the strategy that could cause this. I'll see if I can generate a stripped down case which triggers the problem.

3. Each backtest is ran independently from each other. When you run it and one of the scenarios in the above link is hit it will load data for the instruments even if it just loaded it for a prior test.

You should put this your list of things that need improvement. I plan to run the same strategy in real time on a number of instruments, and each instance will have a reference bar series. It's pretty ugly if each instance makes a separate call to the feed for the same data which is already known to NT7.

[NinjaTrader_JoshP]

kdoren,

2. Thank you.

3. NT would not load data if you don't request a backtest with the current date. If you do request the current date then it will have to keep going off to the data provider. From backtest #1 you would have the data at that moment. When backtest #2 begins there would be new data for that date again and it would go off and try and retrieve it too. If your backtest date does not contain the current date, ensuring you do not run into the other two situations in terms of historical data by looking in the Historical Data Manager will prevent extra loading.

[NinjaTrader_JoshP]

4. Development wants to know if it worked for you in 6.5 before?
If so, has anything changed on your subscription?
Or has anything been changed in terms of your PC setup with firewalls, etc?

[kdoren]

2. There is definitely a bug in NT7 relating to requesting data from the feed.

If the Data Series is Daily bars or higher, NT7 always requests data from the feed, even if the data is all in history. This is true for both strategies and charts. Minute bars work as you described. You can reproduce this by opening a chart of daily bars, or with a null strategy:

namespace NinjaTrader.Strategy
{
    public class KDTestGetHistory2 : Strategy
    {
    }
}

4. I switched to IQFeed around the time I started using NT7, so I don't know if it worked differently in 6.5. It wouldn't be a subscription issue, the IQFeed diagnostic passes the news feed test, and NT7 appears to get news items successfully from IQFeed. However, if I am reading the IQFeed logs correctly, NT7 makes a burst of 54 "News Headline" requests to IQFeed once every minute. I'm wondering if this interferes with downloading historical data, but I don't know a way to disable it. NT7 does appear to retrieve news from IQFeed, but sometimes the IQFeed and NT7 logs show errors:

STATUS (LookupError)=2010-05-02 11:40:07: News Connect Error SendRequest (66.112.156.210:60018). Error:10060.
TO CLIENT (LookupError)=2010-05-02 11:40:07: !ERROR! Could not connect to News socket.

This could be an IQFeed client issue. I have tried disabling my firewall/antivirus/antispyware but it makes no difference.

I can reproduce this by opening NT7, connecting to IQFeed, selecting the "Log" tab, and letting it sit. Within a few minutes, the NT7 log will show the error "IQFeed news feed is not enabled." I usually get a few more of these messages within a few minutes, then they stop. But if I open a News window, news items from IQFeed appear to be there. And the IQFeed logs show continuing news requests every minute.

[NinjaTrader_JoshP]

kdoren,

2. We will look into it. Thanks.

4. Would it be possible if you sent in your login credentials so we can test it on our end? Thank you. Please send an email to support [at] ninjatrader [dot] com and put Attn: Josh with a link to this thread in it. Thank you.