MA data loading issues

[terenyi]

Ok, one more thing: I'm having some serious issues with the MA (using the Yahoo data feed):

1) I cannot seem to get the built-in SP 500 list to load (not even using the plain default template without indicators or anything). I have checked the symbol names (and removed the few instruments where they weren't correct or not set). When I load for example the Nasdaq 100 list (or any other) then all values (e.g. last price, last close) are shown almost instantly. Using the SP 500 list no values at all show up and nothing happens. Reloading doesn't do anything either. I don't understand why one list works (and really fast too) and the other one not at all.

If I put individual titles from the SP 500 list into another list, then they load.

I am trying to use the market analyzer as a stock screener. I have my custom indicators which can analyze each instrument but that won't help if I can't get the lists to load in the first place (not speaking of getting any indicators to load, see below).

2) Ok, this is really strange: I have a market analyzer with an indicator in one column. (The lookback period is one bar so this cannot be the reason for the following problem.) When I add instruments (or hit reload) the "loading data" window appears and shows only the the first instrument (which it is obviously processing) but gets stuck there. The displayed name doesn't change (sometimes it does advance I think but then very slowly) and even if I hit abort, it's still stuck and the window doesn't go away.

So far so bad, but now comes the strange part: If I switch to the main window (which I can since the loading window is not modal) and then press the X button (to close the application) then the window asking me if I want to save the workspace appears. But now suddenly the "loading data" window starts working and I see the displayed instruments changing as it is working through the list. If I close the "save workspace" window then the "loading data" window is stuck again.

So if I leave the "save workspace" window open (or if I reopen it by trying to close the whole application) then the "loading data" window quickly runs through all instruments but usually gets stuck at the last instrument. But while the "save workspace" window is open I can hit the cancel button and the "loading data" window will go away.

After this procedure the indicator column contains the correct values. (Well, at least insofar correct as it uses the values from the last bar which might however contain wrong data if it's this "phantom bar" as I described in http://www.ninjatrader-support.com/v...ad.php?p=50594).

So at least I can get all data to load and the indicator to be calculated, even if it's a bit complicated. This of course doesn't work with the SP 500 list because this doesn't load any data in the first place (see point 1).

3) Another weird thing is that after I get the data and my indicator to load as described in point 2, suddenly new "loading data" start popping up
this time saying that they cannot be aborted. One window per instrument pops up and closes after about half a second. Then a new window for the next instrument pops up and so on until it is at the end of the list. This is in constrast to point 2 where there is only one window which gets stuck.

4) In general the "loading data" window tends to get stuck pretty often. Sometimes it can't be cancelled in the first place and sometimes I can press the cancel button but then it's stuck aborting. Unfortunately I cannot say exactly when it happens. But I think it happens for example when loading/refreshing a list that contains indicators (see point 2) or when reloading the historical data of a chart after data has been loaded in a market analyzer (weird).

All of this behaviour seems very weird to me so I hope someone has any suggestions for me especially since I need the MA as a screener and so far this is not working at all.

[NinjaTrader_JoshP]

1) Loading the SP500 list into a Market Analyzer can be taxing on your system. I can't reproduce what you are seeing on my end though. I am able to load a majority of the list with the relevant prices. Please just be patient and let the feed load the data.

2) It is trying to load information. The Yahoo feed can be extremely slow.

The MA is not specifically designed as a stock screener. You will have difficulty finding a data provider that will provide you with a simultaneous stream to such a high number of instruments a screener would require.

[terenyi]

1) I monitored the network activity of NT and found the following: As soon as I add instruments to the MA NT connects to one of the mcsl2.finance.vip.*.yahoo.net and q1.finance.vip.*.yahoo.com servers each and downloads the data. While the data is being loaded the empty rows in the MA are gradually filled. For the Nasdaq 100 list this whole process takes a few seconds. If I try the S&P 500 list in contrast then NT starts communicating with these servers too and data is being exchanged for about 5-10min and then the transfer ends. The two differences are now that while loading the data empty rows are NOT being filled with data (i.e. the MA does not change at all) and even after NT has finished downloading data (i.e. no more network activity) still nothing changes and no data is being shown. So the question is what is different in the second case? Does NT maybe try to load too many things concurrently? Another question would be why it should take 10min for 500 titles if it takes a few seconds for 100 titles.

2) The question here is why the behaviour of the "loading data" window is linked to whether I try to close the application or not and why it is only not stuck when the "save workspace" window is open. This is definitely a NT issue.

3) What is NT doing now that makes all these windows pop up? I thought first no window is shown when the basic data is loaded and then when the data for the indicator is retrieved the "data loading" windows is shown (see point 2). But why do the "data loading" windows pop up again (and this time one window per instrument)?

It's obvious that the amout of data for a real-time screener with many instruments would be beyond the limits of any data provider. However, I only need daily data for my screener so that I can analyze the past trading day and place orders for the next day. So if NT stored historical data (which it does not if I understand the help file correctly) all it would need to do was to load a single bar per instrument per day (so the data needed for the screening would be for example far less than the data needed to open a sinlge chart). I know this is unfortunately not how NT works but this would definitely be technically feasible and maybe others might need this feature too.

[NinjaTrader_Dierk]

- YAHOO historical data requests are one after another (no concurrency) as regular HTTP requests
- historical data downloaded is save on your harddisk. not sure where you found any different info
- difference in runtime on HTTP requests could result from many different reasons. None of them is affected by NT

[terenyi]

Hallo Dierk (aber bleiben wir im Forum bei Englisch)!

Sorry for being so persistent but I'm really convinced that this is a NT issue (and I'm right if you see below ).

- YAHOO data requests are one after another (no concurrency) as regular HTTP requests

Yes, this is consistent with my analysis of the network traffic. But I just had a more detailed look at the network traffic at packet level and I found the problem (see below) .

- historical data downloaded is save on your harddisk. not sure where you found any different info

The help file under "When does NinjaTrader Load Historical Data" talks about a cache and says "The internal cache is managed by the .NET framework so it is unpredictable when the .NET runtime will clear it. In most cases, as long as you have at least one chart of your instrument open, the instrument’s data cache will persist." So this sounded to me like the data is only cached while the application is running.

But so that means that any historical data is saved and (assuming for this example that my indicator only needs 1 lookback bar) that the screener only needs to load one bar per day per instrument? So that should be no problem then?

- difference in runtime on HTTP requests could result from many different reasons. None of them is affected by NT

Yes but I did not say that I think the time difference it due to the HTTP traffic. Actually there is no reason to assume that the runtime of the HTTP requests is in any way related to the problem because the packets don't know if MA requests 100 or 500 instruments.

Anyway, I took the libterty of debugging your application and as I said, I found the problem:

The analysis of the network traffic showed that NT is using a single HTTP request to fetch the data of all instruments in the MA at once (which is a sleek solution), i.e. data is requested with a URL like http://download.finance.yahoo.com/d/bars.csv?s=FLEX+BIDU+EXPE&f=sl1k3 (note how the URL contains multiple ticker symbols).

Using the S&P 500 list this URL becomes quite long. However, it's not the problem that the URL is too long but that Yahoo rejects the reqest and returns the following:

Sorry, the Yahoo! Finance system limits quotes to 200 ticker symbols at a time, and your request included 490ticker symbols.  Please adjust your request to include 200 or less.

I wonder that this has not come up before or in any test.

Anyway, the solution is that the MA needs to use multiple requests to fetch the data requesting only up to 200 at once. That shouldn't be difficult to change and I think this is pretty hotfix-worthy!!!

Another thing I have noticed while looking at the network traffic: NT sends its request to finance.yahoo.com from which it however gets a HTTP/1.1 301 Moved Permanently response and is redirected to download.finance.yahoo.com. This is no problem per se as NT is smart enough to try again with the correct URL, however it generates unnecessary network traffic so you might consider updating the URL in the program.

Ok, now back to debugging my own programs.

[terenyi]

PS: This of course only answers problem 1) and still leaves questions 2) and 3) open.

[NinjaTrader_Dierk]

Thanks for your input and your analysis. We'll add it to the list of future considerations.

>> But so that means that any historical data is saved and (assuming for this example that my indicator only needs 1 lookback bar) that the screener only needs to load one bar per day per instrument?
Correct

[terenyi]

Correct

Cool. It'd be nice if this little fix could make it into the next release (the other things mentioned in this thread are just mere annoyances and nothing really critical) as this would turn the MA into a complete screener (at least with EOD data) even if it was never intended to be one.

[NinjaTrader_Dierk]

We'll rework the YAHOO adapter for NT7.

[billitin]

Last Price missing

I'll insert my two cents. I have problem with MA loading all data columns properly. I'm using IQFeed when the market is open and all the time I'm getting some rows with empty entry for Last Price (column marked as "Price" in the attachment). Sometimes they're a few, sometimes they're almost half of all rows. Note that all columns with indicator values always show up properly. Is that a bug or misconfiguration?

[NinjaTrader_Bertrand]

Hi billitin, do you get any error in the Log tab? Do you get all the data for a smaller list of symbols?

[billitin]

The log is blank, and if the list is under 100 symbols yes I do.

[NinjaTrader_Michael]

Hi Billitin,


Let me know what steps you're taking to set this up in addition to a full screenshot so I can attempt the reproduction of your situation.

[billitin]

Found workaround

1. Load Market Analyzer with lookback = 200 1-hour bars
2. Set one of columns to display last price, another - an indicator (e.g. EMA-50)
3. Connect to IQFeed and load large enough list of instruments (e.g. NASDAQ-100)

What I've noticed is that, strangely enough, indicators that I create on last price always acts right. That is, if I create indicator that says something like

OnBarUpdate(){
Indicator.Set(Close[0]);
}

it'll perfectly substitute the last price column with no glitches. So I've found a workaround.

[Stopped]

I have the same problem with the IQ Feed and MA in NT 7 Beta 18. My MA has only 20 or so instruments but I can't get the last quote and time stamp for FX even though other columns with indicators have values (although I am not sure they are the most recent ones). I have MA linked with charts. I see intraday charts are being populated, but data is not streaming. I need to keep doing reloads to get the latest data...