yahoo feed and charting

[hyena]

Hi, i'm having major problems with ninjatrader 6.5 candle charts (daily candles) and the std yahoo data feed IRO UK equities (LSE) - everything absolutely vanilla.

Basically when i chart any stock in the evening (a few hrs post market close) the last candle appears to be stuck sometime mid day latest pricewise (i have tried tens of stocks and its the same behaviour). Now if i run the stock on market analyser then it has the correct close, high,low, open etc prices so i'm puzzled as it appears that the correct data is in ninjatrader just the charting functionality wont recognise it or theres a setting i'm missing .... any ideas ?

rgds

ian

[NinjaTrader_ChristopherJ]

Hello Ian,

Thank you for your post.

I have tested this on my end by first opening a Daily chart for AAPL and wrote down the Open, High and Low values. Next opened the Market Analyzer window and added the Daily High, Daily Low and Opening columns. All values matched the values that were present within the chart window.

Please attempt the steps above and let me know if you receive different values within each window.

[hyena]

hi,

I think i mentioned in my earlier post i was talking about UK (LSE) stocks post close (so post around 5-30PM UK time which is currently BST) .. problems still there, i have attached a few screen prints which were taken about 30 minutes after close (but i have just checked now, an hour and 30 mins after close and as expected the ninja candle charting is still in error and will be all night and in to tomorrow). I did a long search on ninja forums and came across a post from 2 years ago describing exactly the same behaviour and i'd assumed that ninja had dealt with the bug as clearly the correct yahoo data is getting in to the ninja market alaysis app.

heres the link to the previous ninja article and ill attach the screen prints as example at the end of this post.

http://www.ninjatrader.com/support/forum/showthread.php?t=9625&highlight=Yahoo+close

Now if the bug is still there then i need to get rid of the erroneous current day close in ninja candle charting and replace it with a correct set of current day close data. Now its easy enough for me to write a vba sub in excel to pull in the required close data from yahoo and format it (assuming i have some help with the required file format) to get in to ninjatrader but is it possible to get rid of ninja charts currently erroneous current day close data and replace it with the good stuff ? alternatively if its the candle charting algorithm - do you intend to fix it soon ?

ps its entirely possible im missing the obvious as i am a ninjtrader newbie with burned paws ;-)

www.vousparlez.com/

www.vousparlez.com/

www.vousparlez.com/azn_advfn_1758BST.JPG

www.vousparlez.com/azn_charting_1756BST.JPG

www.vousparlez.com/azn_mkt_anal_1800BST.JPG

www.vousparlez.com/azn_yahoo_1800BST.JPG



rgds

ian

[NinjaTrader_Kyle]

Hi hyena,

Thanks for the additional information.

Unfortunately, these type of results can be expected, especially within the first few hours after a close.

The Yahoo Finance connection available in NinjaTrader is a delayed data connection so accurate close numbers will not be available immediately.

[hyena]

Hi hyena,

Thanks for the additional information.

Unfortunately, these type of results can be expected, especially within the first few hours after a close.

The Yahoo Finance connection available in NinjaTrader is a delayed data connection so accurate close numbers will not be available immediately.

hi,

thanks for the prompt reply

so we are absolutely crystal clear about this :

what you are saying (and what i have seen over the last few weeks ninjatrader testing) is that on the yahoo feed (your standard feed and the only one for uk equities) ninjatrader charting will NOT display anything like the close days stock data until well in to the next trading day ?

Note that many competing delayed feed applications do in fact show the last days trade/close data 10-30 minutes or so after close(incl yahoo finance itself) . Could you check with the ninjatrader programmers that this is really true as it makes ninjatrader pretty much useless on the yahoo feed and this is something you should make crystal clear on your website to everybody !

The reason i'm struggling to make sense of what you stated is that i can see in the ninja market analyser that you have the yahoo close data correctly in (see my screen print) but the ninja charting is charting some different (ninja bug corrupted ?) data for close .. its pretty clear from the screen prints i attached - perhaps show them to a ninja programmer

As i said its no trouble for me to run a vba excel macro to pull in to excel every day accurate close data from yahoo (and given a bit of ninjahelp) formatting a file to upload daily which will take me all of 5 minutes .. is this possible ??? and will this overcome the charting issue with the yahoo feed ?

As ive invested a considerable amount of time in ninja (becuase apart from this charting issue i like it) its in my own interest (as well as ninja's) to overcome what looks like a major ninja bug and if doing a one off historic data upload (for all uk LSE stocks for 60 days) and then daily manual uploads in to ninja each day from excel is the only way fwd then so be it

rgds

ian

[NinjaTrader_ChristopherJ]

NinjaTrader will display all information provided thru the data providers API when it is available.

In order to receive real time daily data I suggest connecting to a paid subscription to ensure you are receiving the most up to minute data possible.

I have provided a link below to the data providers supported by NinjaTrader. This page will go over the types of data they offer as well as the asset classes they support.

http://www.ninjatrader-support.com/HelpGuideV6/HistoricalData.html

[hyena]

wow, this is so painful

if you read this thread I'm not talking about real time data at all im talking about the standard 20-30 minute delayed yahoo data feed. I'm actually using yahoo API's in my excel vba apps quite happily and have no problem getting ACCURATE close data each evening.

Again what i am saying is that there is clearly a bug in your ninjatrader charting app as you have in your market analyser module perfectly correct close yahoo data each evening but run a chart and you have FALSE data for the current close day ... so you dont have to be a rocket scientist to see theres a problem in ninjacharting. So again to be perfectly clear in the evening ninjatrader has 100% correct yahoo close data when you run market analyser but 100% incorrect data when you chart a stock both in respect of close day - so theres clearly a major problem.

Now if you check out the 2008 ninja trader support query forum link along the same lines i attached, ninjatraders reply was to paraphrase "we ie ninjatrader make up the close data", i had assumed that as market analyser was showing 100% correct yahoo close data then you had sorted out this problem, but clearly not !

The issue is that many people are using ninjatrader with yahoo clearly unaware that they are relying on something fundamentally flawed and misleading, which is why i was trying to get to the bottom of whether its a bug that will be fixed or whether its something that should be pointed out to ALL ninjatrader users as it makes ninjatrader practically useless on yahoo feed.

Now in terms of feeds, as i am sure you know ninjatrader has very few datafeeds that cater for overseas equities which is why i'm using yahoo delayed which is fine for me as im not a daytrader.

Finally (and constructively) i was trying to suggest to you (programmers) a workaround in that its clearly not a problem to get (like you have in market analyser) 100% accurate close yahoo data 30 mins after close from the yahoo API, because I do it pretty easily in a few VBA apps i have written that i run both during the day and in the evening. So I was asking if one solution would be to format the evenings close data by instrument in to ninjatrader charting and overide the junk ninjatrader charting has in it for current close. As i said there is no issue in me downloading each evening using the yahoo API in to a simple excel spreadsheet OHLCV data by stock if i can upload them in to ninjatrader and overide ninjatrader chartings close day data?? is this possible if so could you elaborate ?

you might find this will help out many ninjatrader users who are happily trundling away on ninjatrader/yahoo not realising the close day figure is junk and like me would be horrified. Fortunately for me I was just testing out ninjatrader to potentially replace Sharescope which i currently use.

I'm going out of my way to try and be constructive as its a shame something as small as this would rule out ninjatrader on the yahoo feed as it has a lot of potential.

[NinjaTrader_ChristopherJ]

Can you please use the steps I have provided below to adjust your Market Data Request Interval:

• Disconnect
• Tools
• Account Connections
• Select Yahoo
• Change
• Next
• Next
• Market Data Request Interval
• Set to 1
• Next
• Finish
• Reconnect
• Open a new chart

Please let me know if you see any difference on your end with this setting during tomorrow session.

[hyena]

hi,

sorry i have been away for a couple of days, i will try this both in a few minutes (10PM UKTIME - hours after close) and again during tomorrows trading day.

Again I feel a bit guilty giving you a hard time but its because ninja has got the makings of a really great product and its a hairs width from 100% meeting my needs (and many other people using non US exchanges who only need EOD or delayed data for which feeds are not available or prohibitively expensive for EOD investors) ..

Incidently (before i logged on to the ninja forum) half an hour ago (after seeing as expected incorrect close data) did a quick experiment with the yahoo data feed switched off and simply setup a VOD.L.txt file loaded with 1 line of todays correct close data in ninjaformat (D;O;H;L;C;V) and it imported succesfully and when then running a chart showed perfect todays close data in charting (and the previous history as well). Now when i then connected to the yahoo feed and then again ran the same chart the OHLC remained perfectly charted BUT the volume figure had been overwritten with an erroneous volume figure ??

From this logically i take it a one off manual upload of yahoo data for the LSE FTSE/AIM stocks followed by a daily manual upload of the close days evening may be a goer (after all simply uploading a file is more or less like connecting a feed)

BUT in ninja6.5 you have to load every single EPIC code one by one :-(( .. i did look at the ninja access databases looking for the stored data but couldnt find it (if it stores in an access database this would make it pretty easy to update in 1 go from 1 file ??) .. is it possible in some way to aggregate the upload of the daily individual stock updates in to ninjatrader 6.5 ?

[hyena]

hi, just made the changes and reconnected to yahoo, no change same erroneous OHLC value/correct V for todays close in charting (all correct in mkt analyser)

[Stopped]

Hi hyena,

Although this is no substitute to the proper answer you should receive from NT customer support, I do share many of your frustrations when dealing with non-US equities and wanted to share my 2c and/or workarounds

• Firstly, bear in mind there is a difference between the EOD part of the connection (for daily bars and above) and the intraday part of it. Note that when you look at a UK stock on Yahoo! the last bar on their daily chart corresponds to yesterday's close (EOD) too, not today's, even though their ticker shows the latest close (intraday). A daily chart on NT loaded for the first time will take only the EOD data, in line with what is shown on Yahoo! (see VOD screenshot).
• Secondly, the Yahoo! connection on NT, as I understand it, is not streaming but doing snapshots at regular intervals, which you can customise to your needs following the instructions two posts below.
• Once you set up the Yahoo! connection to update every second and keep your charts open while the market is open the daily chart will create the most current bar with the data from the intraday part of the connection while the other bars will be based on the EOD history. While I am not 100% sure about this, I think that you can close and save your workspace after the market close and it will keep the most current values for the charts.

By the way, I am using the beta version of NT 7, which has better features than 6.5.

Hope I haven't added to the misunderstanding that seems to be going on in this thread!

Good luck,

Stopped

[Stopped]

This requires IMMEDIATE action!!!

Hi all,

This is as a follow up to hyena's leads and after further examination this morning.

NinjaTrader_Ray should read this too.


Yahoo! feed problem:

• Data during European hours is simply WRONG! See screenshot vod yahoo and compare it against vod lse. It is now midday in the UK, but the Yahoo! website is still using the close of two days ago as Previous Close, which produces wrong calculations of Net and Pct Change. Today's Volume data is also wrong.
• My guess (just a guess) is that Yahoo! EOD servers will update before US market hours... or perhaps even later!

NinjaTrader problem:

• Volume data on a Chart window does not match the info on the Yahoo! page (see screenshot vod yahoo). Incidentally, as hyena has already pointed out, the data is "correct" on Market Analyzer, so this is specifically a Chart bug.
• Specifically to NinjaTrader 6.5, and in addition to the above, the "Loading Data..." window froze after a refresh while connected to Yahoo! and I had to shut down the application (twice!)

I know the biggest problem is caused by the Yahoo! feed, not the NT platform. But tell this to a European user who decides to test the program using this free connection: They will ditch the software as soon as they see that the charts do not make sense.

Is NinjaTrader going to talk to Yahoo! to point them out to the problem - given the reputational damage this could cause YOU? Or should we do it ourselves individually?

Thanks,

Stopped

[NinjaTrader_Ray]

Thanks.

We will look into what you have reported.

[hyena]

hi,

ok during trading hours (UK LSE) ninjatrader charting shows numbers for OHLC which are consistent with yahoo, however volume is wrong ??. After trading ends (so about 6pm ish UK time onwards) ninja then shows incorrect figures for OHLC but correct figures for volume ??

now if i import a stock from good data (eg my sharescope) OHLCV (yahoo not connected) and run a chart - OHLCV all perfect (per yahoo) but if i then connect to yahoo (for mkt analyser) then OHLC remain perfect and Volume now becomes wrong.

Now i'm already using the Yahoo API in to my excel VBA sheets and i am used to living with the pain of yahoo segregating current day as a quotes system and previous days in a history system and the real pain of them only transfering the current close data to the history database at something like 7am (UK time) so i am sure whats happening with ninja is that for the current close days ninja cant get history in charting so it makes the data up (and the reason market analyser show correct is that its getting it from the yahoo quotes database) .. solution ninja does what i do and take T-1 from history and T=0 from quotes .. i'm sure its not as simple as this but for me its worked beautifully

Now to what i've tried in the hope that something similar will help many people on non us exchanges where there is virtually zero ninja compatible equity feeds other than yahoo

I already subscribe to a decent EOD LSE data feed (news feed used to be good but its crap now) as part of the sharescope application and it has a great export routine that exports amongst other things all stocks as individual files (EPIC filename) in pretty much ninja format (except its volume is to 2 d.p. which ninja doesnt like) and the filenames are Pure EPIC which youre probably gonna want to convert to yahoo just in case (usually just add.L to the end but some you need to get rid of other . in some of the EPIC names as well)

Now if you use something like ultraedit and do a batch search and replace (.00 for nothing) on the exported EPIC files, 10 seconds later all your files are good to go

next get a freeware application called "Bulk Rename Utility" and set it up to add a .L at the end of all files and remove all "." from the name, save the profile as default
(for one stock youre gonna need to do a manual file name edit BT.A i think in EPIC needs to go to BT-A.L in yahoo or be clever in how you set up the instrument in ninja ;-) )

now i did a one off upload of a years data for all stocks and imported it in to ninja last night with just a few issues (basically the sharescope data has a few corporate actions on older data that resulted in a d.p. column number and a ninja freakout and a few zero volume days which again caused a ninja freakout) to deal with.

now every night i'm gonna do the download and import and see how it goes .. timewise it takes me 5 seconds to do the export, 10 seconds for the ultraedit search and replace and 10 seconds for the batch rename. ninja then in its import window let me select all 1000 stock file uploads and swallowed them up perfectly

... result is i have perfect data :-)))) .. hope it carries on working

I do have the problem in that if i connect to yahoo it screws up my volume until i disconnect and unless ninja support can help me, i need to connect to yahoo to use market analyser wheras i would prefer to use the imported data (as its better than yahoo and doesnt corrupt my current close volume) and never connect to the yahoo feed

Now for those people that dont have access to another EOD feed that can be ninja imported its just as easy (though the yahoo data is only 99%) to knock up an excel workbook (stick yahoo version of EPICS vertically in a column) and either access the API directly though VBA or simply use a few formulae to concatenate together to do the same by creating the URL etc.

Or you could even download the freeware "yahoo and google historical download" and set it up to do the same as i'm doing with sharescope.

I did actually try contacting IB and kinetick to see if they had a LSE ninja feed .. never even got an answer from either of them ! at least with TDWaterhouse they got back to me within the hour with a polite "no we just cover the US"

It would be great if i could use ninja directly with IGINDEX (uk spreadbetting) or one of the many UK online stockbrokers but i can well understand there many not be enough volume to generate a return for ninja as in the US market, till the times when it does hopefully ninja will recognise that providing a workaround for EOD builds them up a potential future client base maybe one day. What they may not realise is that although "free ninja users" may not contribute financially today to ninja, come the day ninja exit, every "free ninja user" will add quite a few dollars to price they get ;-)



hope this helps

[hyena]

more on better LSE/AIM data in ninja

sorry to not include it all in 1 post, but just to echo "stopped", there are 2 issues :

1. integrity of underlying data (yahoo in this case).
2. ninjatraders problems with what looks like corrupting data feeds (yahoo in this case)

Now what i generally see with yahoo data is that its by is large ok for the larger and med caps but as you get to the smaller caps you get quite a lot of missed trades (which in a small cap is a big issue). At the end of the day there is no excuse for this EOD sloppiness, its not a cost issue its just pure sloppiness because yahoo have little competiton outside the US.

As i say the sharescope data is 99.99999999999% and seems to go in to ninja and chart ok, not thoroughly tested some of the instruments yet but i cant see why they wouldnt be 100% .. only issue i have is to use market analyser i need to connect to yahoo ... pity i cant just use my imported data :-(

heres an example of a LSE AIM small cap that i picked up to check as in ninja (thats my ninja with good imported data) it differed from one of my yahoo API vba excel sheets ... shows nicely the small cap issues yahoo has ... thats no excuse for the way ninja charts on a yahoo feed as ninja is taking the yahoo close data and making up something completely different.

LSE top, yahoo bottom, hours after close, getting on for midnight UK time

the thing that really gets to me is that both yahoo and ninja screw the HLOC but get the volume right ... i'm sure thats quite relevant though not too sure why