Strategy Unable to Access Prior Weekly Data in Market Replay

[fiddich]

Hello,

I'm trying to access previous weekly closes in a strategy I'm testing in market replay, however it keeps giving me the indexing error (cannot access a bar that doesn't exist), which confuses me because I have a couple months of historical data downloaded and loaded on the chart when running this strategy.

I've included a simple strategy below that just prints the close of the prior 4 weeks. It prints the first week in the output just fine but then errors on the second (the indexing error). Any thoughts as to why and how to fix this? I did this on ES 12-18 in late November, with historical data loaded back to the beginning of October (8 weeks). Thanks.

#region Using declarations
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.ComponentModel.DataAnnotations;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Windows;
using System.Windows.Input;
using System.Windows.Media;
using System.Xml.Serialization;
using NinjaTrader.Cbi;
using NinjaTrader.Gui;
using NinjaTrader.Gui.Chart;
using NinjaTrader.Gui.SuperDom;
using NinjaTrader.Gui.Tools;
using NinjaTrader.Data;
using NinjaTrader.NinjaScript;
using NinjaTrader.Core.FloatingPoint;
using NinjaTrader.NinjaScript.Indicators;
using NinjaTrader.NinjaScript.DrawingTools;
#endregion

//This namespace holds Strategies in this folder and is required. Do not change it. 
namespace NinjaTrader.NinjaScript.Strategies.Test_Strategies
{
    public class Test_Strat_030719 : Strategy
    {
        protected override void OnStateChange()
        {
            if (State == State.SetDefaults)
            {
                Description                                    = @"Enter the description for your new custom Strategy here.";
                Name                                        = "Test_Strat_030719";
                Calculate                                    = Calculate.OnPriceChange;
                EntriesPerDirection                            = 1;
                EntryHandling                                = EntryHandling.AllEntries;
                IsExitOnSessionCloseStrategy                = false;                
                IsFillLimitOnTouch                            = false;
                MaximumBarsLookBack                            = MaximumBarsLookBack.TwoHundredFiftySix;
                OrderFillResolution                            = OrderFillResolution.Standard;
                Slippage                                    = 0;
                StartBehavior                                = StartBehavior.WaitUntilFlat;
                TimeInForce                                    = TimeInForce.Gtc;
                TraceOrders                                    = false;
                RealtimeErrorHandling                        = RealtimeErrorHandling.StopCancelClose;
                StopTargetHandling                            = StopTargetHandling.PerEntryExecution;
                BarsRequiredToTrade                            = 20;
                // Disable this property for performance gains in Strategy Analyzer optimizations
                // See the Help Guide for additional information
                IsInstantiatedOnEachOptimizationIteration    = true;
            }
            else if (State == State.Configure)
            {
                AddDataSeries(BarsPeriodType.Week, 1);
            }
        }

        protected override void OnBarUpdate()
        {
            if (State == State.Historical)
                return;

            if (BarsInProgress == 0)
            {
                Print("Closes[1][1] = "+Closes[1][1]);
                Print("Closes[1][2] = "+Closes[1][2]);
                Print("Closes[1][3] = "+Closes[1][3]);
                Print("Closes[1][4] = "+Closes[1][4]);
            }
        }
    }
}

[fiddich]

Quick update - the ES historical data I have downloaded goes back to September for the 12-18 contract and then I have the 9-18 downloaded as well going back to June 2018.

[NinjaTrader_Jesse]

Hello fiddich,

Thank you for the post.

This is likely caused because there is currently no error checking to make sure enough data was loaded. Can you try adding the following line to the top of OnBarUpdate to see if this is related to the amount of data loaded?

        protected override void OnBarUpdate()
        {
            if(CurrentBars[0] < 4 || CurrentBars[1] < 4) return;

This just makes sure that before printing you see 4 bars processed on each series. This would ensure that the Closes[1][4] works.

Another item which can help is the Bar listed in the error as this may indicate when the problem is happening in processing. For example error on bar 0, or error on bar 150000 would give an idea of what point in processing this failed either beginning or end of the chart series.


I look forward to being of further assistance.

[fiddich]

Thanks for the response Jesse.

So do I have to download and then run market replay for 4 weeks to be able to access that data when using the line of code you provided? I can't use the loaded historical data to access the weekly closes right away in the above strategy?

There isn't that much market replay data available, so for testing anything longer term, it would not be possible then?

[NinjaTrader_Jesse]

Hello fiddich,

Thank you for your reply.

No, you do not need this much playback data, but you would need historical daily bars for this period of time to generate weekly bars. When your script starts processing it starts from bar 0. If the error you are seeing is listing a low bar number in the error, it is likely that you are just accessing data before it has been processed and is available

Here is an example print from playback:

Print(CurrentBar + " " + BarsInProgress);

44 0
45 0
0 1
46 0
47 0

In this test, I saw 45 primary bars close before the first close of a weekly bar. If you check Closes[1][4] before 4 weekly bars have closed, you'll get an error. Using the if check should allow the script to process enough data up to a point where you can start doing what you are trying to do.


I look forward to being of further assistance.

[fiddich]

Thanks Jesse.

Still a bit confused regarding the output of the print statement you provided.

I'm getting

260 0
1 1

So that means I've loaded 260 days but only 2 weeks? Shouldn't that be enough days to have many more than 2 weeks?

[NinjaTrader_Jesse]

Hello fiddich,

This would indicate to me that 260 primary bars were processed before reaching 1 processed weekly bar.

If you are using the same print:

Print(CurrentBar + " " + BarsInProgress);

The left number is the current bar, so your sample bar 260 was processed on the Primary series or BarsInProgress 0. Next CurrentBar 1 was processed for the secondary series, we see BarsInProgress 1 listed on the right.

What primary series is being used in this test? You asked So that means I've loaded 260 days but only 2 weeks?
Not necessarily, if you were using a Daily primary series then yes that means 260 day bars were processed. Otherwise, if you are using a less granular than 1-week primary we are likely just seeing the primary bars being less granular here.

I look forward to being of further assistance.

[fiddich]

Yes, using the same print statement as you and using daily bars.

So since I'm using daily bars, how come the strategy can't access the weekly close 3 weeks ago when it's loaded 260 daily bars?

Code below for reference. Thanks.

#region Using declarations
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.ComponentModel.DataAnnotations;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Windows;
using System.Windows.Input;
using System.Windows.Media;
using System.Xml.Serialization;
using NinjaTrader.Cbi;
using NinjaTrader.Gui;
using NinjaTrader.Gui.Chart;
using NinjaTrader.Gui.SuperDom;
using NinjaTrader.Gui.Tools;
using NinjaTrader.Data;
using NinjaTrader.NinjaScript;
using NinjaTrader.Core.FloatingPoint;
using NinjaTrader.NinjaScript.Indicators;
using NinjaTrader.NinjaScript.DrawingTools;
#endregion

//This namespace holds Strategies in this folder and is required. Do not change it. 
namespace NinjaTrader.NinjaScript.Strategies.Test_Strategies
{
    public class Test_Strat_030719 : Strategy
    {
        protected override void OnStateChange()
        {
            if (State == State.SetDefaults)
            {
                Description                                    = @"Enter the description for your new custom Strategy here.";
                Name                                        = "Test_Strat_030719";
                Calculate                                    = Calculate.OnPriceChange;
                EntriesPerDirection                            = 1;
                EntryHandling                                = EntryHandling.AllEntries;
                IsExitOnSessionCloseStrategy                = false;                
                IsFillLimitOnTouch                            = false;
                MaximumBarsLookBack                            = MaximumBarsLookBack.TwoHundredFiftySix;
                OrderFillResolution                            = OrderFillResolution.Standard;
                Slippage                                    = 0;
                StartBehavior                                = StartBehavior.WaitUntilFlat;
                TimeInForce                                    = TimeInForce.Gtc;
                TraceOrders                                    = false;
                RealtimeErrorHandling                        = RealtimeErrorHandling.StopCancelClose;
                StopTargetHandling                            = StopTargetHandling.PerEntryExecution;
                BarsRequiredToTrade                            = 20;
                // Disable this property for performance gains in Strategy Analyzer optimizations
                // See the Help Guide for additional information
                IsInstantiatedOnEachOptimizationIteration    = true;
            }
            else if (State == State.Configure)
            {
                AddDataSeries(BarsPeriodType.Week, 1);
            }
        }

        protected override void OnBarUpdate()
        {
            if (State == State.Historical)
                return;

            if (IsFirstTickOfBar)
                Print(CurrentBar + " " + BarsInProgress);

            if (BarsInProgress == 0)
            {
                if (IsFirstTickOfBar)
                {                    
                    if(CurrentBars[0] < 4 || CurrentBars[1] < 4) return;                

                    Print("Closes[1][1] = "+Closes[1][1]);
                    Print("Closes[1][2] = "+Closes[1][2]);
                    Print("Closes[1][3] = "+Closes[1][3]);
                    Print("Closes[1][4] = "+Closes[1][4]);
                }
            }
        }
    }
}

[NinjaTrader_Jesse]

Hello fiddich,

Thank you for your reply.

In contrast to your test, I am not sure without the full details of all options used. I am not seeing a problem with the strategy accessing data in this sample while testing from my end. I see that in realtime only after closing a full daily bar and having enough data loaded, it has 1 print like the following:

253 0
Closes[1][1] = 2777
Closes[1][2] = 2706.25
Closes[1][3] = 2704.25
Closes[1][4] = 2663.5

All prior weekly bars were accessed as there was enough data at this point.

When you ask "how come the strategy can't access the weekly close 3 weeks ago when it's loaded 260 daily bars?" are you waiting for a full daily bar to close in realtime and also have enough bars loaded? The print you provided indicates you did not have enough data loaded at the time of the print for your other prints to happen.

1 1 indicates that there was 1 bar at this point but the condition requires 4 bars so this would not be a time when your second print should happen.



I look forward to being of further assistance.

[fiddich]

Yes, I'm starting the strategy on 11/22/18 at 2:00PM CST and run it for a minimum of a few days and have done longer as well. On the open on 11/25, it prints 260 0 and then 1 1, but I'd expect the weekly data series to have more loaded at that time and be able to print those weekly close statements like yours did.

Are there any other settings that could be affecting this? I use the exact code I provided you, have daily data on ES going back many months on both the 12-18 and 9-18 contracts and applied the strategy to a daily ES 12-18 chart. I've also restarted NT multiple times. Can't think of anything else that would cause this - would appreciate your thoughts.

Thanks.

[NinjaTrader_Jesse]

Hello fiddich,

Thank you for your reply.

I would be unsure of the difference here, from my test the print happens almost immediately when running in realtime as long as I have enough data loaded beforehand. Do you currently have the chart configured to load enough bars? If you are using a small Days To Load you may need to wait up to 4 weeks of processing data in realtime for this to begin working. If you have enough historical data loaded, the script should be able to process that and continue sooner.

Have you at this point also tried to reload historical data for both the primary and weekly series? You may try opening a chart that has both series, the weekly and your primary and then reload all historical data while connected to the normal data provider. I would also suggest increasing the days to load for both series to account for the total span you are testing to make sure all data is downloaded.


I look forward to being of further assistance.

[fiddich]

Thanks Jesse.

I tried the Days to Load and opening both series in a chart while connected to Continuum, but neither did the trick.

What ended up working for me was loading the weekly data within the same chart the daily ES was loaded on. While I'm glad that worked, I still don't get why adding the data series within the strategy code doesn't seem to work - any ideas? I've never had it not work before until now.

[NinjaTrader_Jesse]

Hello fiddich,

Thank you for your reply.

I would be unsure why it wouldn't have downloaded the data in this case other than perhaps something to do with the DaysToLoad. Do you see this with any other instruments using this setup or was this kind of a unique situation? I was able to see this load on my end when I had tested this. This could potentially be related to the data cache however I don't want to get ahead of myself here and guess. Knowing if this happens with other combinations would help better identify what happened.

I look forward to being of further assistance.