Catch series with no data

[pmorissette]

Hey,

I have been driving myself crazy trying to fix a problem here. Basically, I am adding a list of instruments in a strategy. However, some instruments seem to be invalid insofar that they don't have data (they get added without errors/warning/trace file entries..., but I don't have data for them - I found one of them by chance and the chart went from 0 to 0.01 with no data...)

I know that NT only calls BarsInProgress once the shortest series has a bar update, but how can we catch series that don't have any data at all? My strategy simply won't run because of this, and I can't seem to figure out a way of identifying the bad security (without going through all 500 one by one...). OnBarUpdate never gets called.

I've tried printing the name of the security if it had Closes[x].Count == 0, or null, within the Initialize method but that didn't work.

Any help would be appreciated.
Phil

[NinjaTrader_RyanM1]

Hello pmorissette,

This is unfortunately beyond our scope of support. If I had to solve this I would use market analyzer to populate the list with instruments and then ommit any that didn't have data.

[pmorissette]

Ok I see.

I'll try to do this and see if I can detect the ones that don't have data.

Thanks,
Phil

[pmorissette]

Hey,

I tried adding the list to the market analyzer and this did not yield any significant results. I found one security that had no data, but even after removing it from my list, the strategy still won't run.

I am not really doing anything "unsupported" as I am just trying to add multiple securities to a strategy and then use the barsInProgress index to trade individual securities.

Is there any way to find out which securities have no data from the code, or from anywhere else for that matter?

[NinjaTrader_RyanM1]

It's not that you're doing anything unsupported. We just can't offer a coding solution for you.

It's required to add series in the Initialize() method. Bar objects are not available here. Therefore any series that don't have data will not be discovered until runtime.

You'll have to discover through some other means which series don't have data and not attempt to add through code.

[pmorissette]

I see. What other methods are available? I have added the list to the market analyzer and each row seems to have data. I don't see why it is still not working.

Since my list is the SP500, it is more than tedious to go through every stock, especially since it is not in the default list (and adding it to the default list and subsequently removing it would be more than tedious).

Any clever solutions I am not thinking of?

[NinjaTrader_RyanM1]

Market Analyzer might not work for you if the bar type is offered on the MA and you are accesssing a different type through code. This is dependent on the indicator columns loaded on the MA - You can select an interval that matches what you're looking to add through code.

You might also want to look at the qualities of the instruments you're having trouble with, and use NinjaScript to generate a list that matches these.

What is the specific message you're getting when trying to access the instruments?

What are the price values used for these? Print (Closes[1][0]);

[pmorissette]

Hey RyanM,

So basically I am not getting any errors and I can't find anything in the log nor in the trace files. The strategy seems to run (doesn't stop right away) but then returns all zeros. I have some print statements in the OnBarUpdate so it never gets there. That's why I have a feeling that one (or more) series just doesn't have any data and since NT doesn't call OnBarUpdate unless there is data, then the strategy exits right away.

What do you mean by "qualities of the instruments"?

By the way I am using daily data obtained on Kinetick/Yahoo. I do get some errors when I try to download historical data in the historical data manager but none of these messages provide information on which security caused the error.

[NinjaTrader_RyanM1]

By qualities - I mean what can you check for. You'll probably have to look at one instrument that's causing issues and start poking around until you can identify something you could check for.

[pmorissette]

Ok well I think I know what the problem is - I have found one by chance. Basically, if that symbol is typed into yahoo finance, it has a data error on their chart, and I'm assuming this means something happened to the security (buyout, bankruptcy or something like that...). However, there is no way to check that from NT asides from pulling up a chart and seeing that there is absolutely no data.

However, creating 500 charts is more than inconvenient so I am trying to find a way to resolve this within the Initialize method (or some other method for that matter). Any ideas?

[NinjaTrader_RyanM1]

You'll probably have to check this in OnBarUpdate(). Use it to generate a report and then remove those instruments from the Initialize()

Share an example instrument as well as the exact add statement and I'll see what I can find out.

[pmorissette]

Here is my Initialize() method:

TraceOrders = false;
            CalculateOnBarClose = true;
            BarsRequired = lookBack + 1;
            EntryHandling = EntryHandling.UniqueEntries;
            ExitOnClose = false;

            if (bottomXShort != 0)
                capitalBot = capitalTop * (topXLong / bottomXShort);

            try
            {                
                InstrumentList il = InstrumentList.GetObject(list);
                DefaultOptimizationMethod bob = new DefaultOptimizationMethod();
                
                foreach (Cbi.Instrument i in il.Instruments)
                {                    
                    Add(i.FullName, BarsPeriod.Id, Math.Max(1, BarsPeriod.Value));
                    numberOfInstruments++;                    
                    tags.Add(numberOfInstruments, i.FullName);                    
                }  
            }
            catch (Exception)
            {               
                NinjaTrader.Gui.MessageWindow bob = new Gui.MessageWindow("Instrument List does not exist! Check your spelling you fool!", false);
                bob.ShowDialog();
            }

            if (vix)
            {
                Add("^VIX", PeriodType.Day, 1);
                vixIndex = numberOfInstruments + 1;
            }

My symbols are:

       MMM        ACE        AES        AFL        AKS        T        ABT        ANF        ADBE        AMD        AET        A        APD        ARG        AKAM        AA        AYE        ATI        AGN        ALL        ALTR        MO        AMZN        AEE        AEP        AXP        AIG        AMT        AMP        ABC        AMGN        APH        APC        ADI        AON        APA        AIV        APOL        AAPL        AMAT        ADM        AIZ        AN        AZO        ADSK        ADP        AVB        AVY        AVP        BBT        BMC        BHI        BLL        BAC        BCR        BAX        BDX        BBBY        BMS        BBY        BIG        BIIB        HRB        BA        BXP        BSX        BMY        BRCM        BF.B       CA        CBG        CBS        CF        CHRW        CI        CMS        CNX        CSX        CVS        COG        CAM        CPB        COF        CAH        CFN        KMX        CCL        CAT        CELG        CNP        CTL        CEPH        CERN        CHK        CVX        CME        CB        CINF        CTAS        CSCO        C        CTXS        CLF        CLX        COH        KO        CCE        CTSH        CL        CMCSA        CMA        CSC        CPWR        CAG        COP        ED        STZ        CEG        GLW        COST        CVH        CMI        DTV        DTE        DHR        DRI        DVA        DV        DF        DE        DELL        DNR        XRAY        DVN        DO        DFS        DISCA        D        RRD        DOV        DOW        DPS        DD        DUK        DNB        ETFC        EMC        EOG        EQT        EMN        EK        ETN        ECL        EIX        EP        ERTS        EMR        ETR        EFX        EQR        EL        EXC        EXPE        EXPD        ESRX        XOM        FLIR        FMC        FTI        FDO        FAST        FDX        FII        FIS        FITB        FHN        FSLR        FE        FISV        FLS        FLR        F        FRX        FO        BEN        FCX        FTR        GME        GCI        GPS        GD        GE        GIS        GPC        GNW        GENZ        GILD        GS        GR        GT        GOOG        GWW        HCP        HAL        HOG        HAR        HRS        HIG        HAS        HCN        HNZ        HP        HSY        HES        HPQ        HD        HON        HRL        DHI        HSP        HST        HCBK        HUM        HBAN        ITT        ITW        TEG        INTC        ICE        IPG        IBM        IFF        IGT        IP        INTU        ISRG        IVZ        IRM        JDSU        JPM        JBL        JEC        JNS        JNJ        JCI        JNPR        KLAC        K        KEY        KMB        KIM        KG        KSS        KFT        KR        LLL        LSI        LH        LM        LEG        LEN        LUK        LXK        LIFE        LLY        LTD        LNC        LLTC        LMT        L        LO        LOW        MTB        WFR        M        MRO        MAR        MMC        MI        MAS        MEE        MA        MAT        MFE        MKC        MCD        MHP        MCK        MJN        MWV        MHS        MDT        MRK        MDP        MET        PCS        MCHP        MU        MSFT        MOLX        TAP        MON        MWW        MCO        MS        MOT        MUR        MYL        GAS        NKE        NYX        NBR        NDAQ        NOV        NSM        NTAP        NYT        NWL        NEM        NWSA        NEE        NI        NBL        JWN        NSC        NU        NTRS        NOC        NOVL        NVLS        NUE        NVDA        ORLY        OKE        OXY        ODP        OMC        ORCL        OI        PCAR        PCG        PNC        PPG        PPL        PTV        PLL        PH        PDCO        PAYX        BTU        JCP        PBCT        POM        PEP        PKI        PFE        PM        PNW        PXD        PBI        PCL        RL        PX        PCP        PCLN        PFG        PG        PGN        PGR        PLD        PRU        PEG        PSA        PHM        QEP        QLGC        QCOM        PWR        DGX        Q        RSH        RRC        RTN        RHT        RF        RSG        RAI        RHI        ROK        COL        ROP        ROST        RDC        R        SAI        SCG        SLM        SWY        CRM        SNDK        SLE        SLB        SCHW        SNI        SEE        SHLD        SRE        SHW        SIAL        SPG        SJM        SNA        SO        LUV        SWN        SE        S        STJ        SWK        SPLS        SBUX        HOT        STT        SRCL        SYK        STI        SUN        SVU        SYMC        SYY        TROW        TE        TJX        TGT        TLAB        THC        TDC        TER        TSO        TXN        TXT        BK        TMO        TIF        TWC        TWX        TIE        TMK        TSS        TRV        TYC        TSN        USB        UNP        UPS        X        UTX        UNH        UNM        VFC        VLO        VAR        VTR        VRSN        VZ        VIA.B       V        VNO        VMC        WMT        WAG        DIS        WPO        WM        WAT        WPI        WLP        WFC        WDC        WU        WY        WHR        WFMI        WMB        WIN        WEC        WYN        WYNN        XL        XEL        XRX        XLNX        YHOO        YUM        ZMH        ZION        EBAY        JAVA       ACS       BNI

For example, I know that JAVA doesn't work. It is one of the instruments that is causing trouble.

I can't make a report in OnBarUpdate because it never gets called.

[NinjaTrader_RyanM1]

Here's the sequence I'm following, simplified from your code:
Add ("JAVA", PeriodType.Day, 1);

First run produces:
9/15/2010 2:25:13 PM Default The indicator 'MultiSeriesTester' has called the Add() method with an invalid instrument. Either 'JAVA' does not exist in the Instrument Manager or the specified exchange has not been configured.

Add Java on the fly by typing symbol into a chart.

Then, next time I get:
9/15/2010 2:25:25 PM Default Error on loading chart data for 'JAVA Default': Failed to retrieve data from YAHOO

I then used Print (Closes[1][0]);

This statement produces all zeroes, only when the instrument is defined in the instrument manager. Nothing is printed when the instrument is not defined. OnBarUpdate() is not called for the added series, but is called for primary, and you get zero when trying to acess the added series value. This might be a good check to look into.

You'll probably need to simplify your code and work with one instrument until you can verify what exactly is going on. Check the log tab of control center for any error messages.

[pmorissette]

Ok the weird thing is that this strategy works flawlessly with another list (which happens to be very similar to my current list - it's the composition of the SP is 2009 instead of 2010...)

The OnBarUpdate never gets called. There are no error I can catch, no errors in the log, no errors in the trace file.

To find the instruments that don't have data, I created a simple MA cross strategy and ran it on the list. I got errors for two of the securities, which I then removed.

I have no idea what to do now. Do you have any ideas?

[pmorissette]

Ok thanks for your help. It seems that I am having a problem with the VIX. This is what is causing my program to fail.

Thanks again for your help.

Phil