Loading more days of data in Strategy Analyzer

[GregX999]

Hello,
So, my strategy is running on a 3 min chart. It uses AddDataSeries() to load a 1 tick series, as well as a 1 day series (as well as a few others, but not important here). In the strategy analyzer I'm running it with Tick Reply enabled.

When it runs, it prints out the number of bars loaded for each data series it loads. For the 1 day series it prints out a number equal to the number of days I've selected to run the strategy. So if I select to start on some Monday and end the following Friday, it prints "5". (By printing BarsArray[dailyIndex].Count inside State.DataLoaded).

But, my strategy needs "the previous day" in order to work correctly. So on that Monday, I get the "You are accessing an index with a value that is invalid since it is out-of-range" error because I'm trying to access Highs[dailyIndex][1] and Lows[dailyIndex][1], but there's only 1 daily bar available on Monday, Monday's. (I can verify that by printing BarsArray[dailyIndex].Count in the OnBarUpdate() method.)

So... my question is, how can I get the strategy analyzer (or my strategy in general) to load more bars when adding the daily series?

I don't run into this issue when running in Playback mode as I've manually downloaded enough data for my testing.

Thanks!

[NinjaTrader_Gaby]

Hello GregX999,

Thank you for your post.

If you are adding a daily series, but using a more granular primary data series, you'll need to add the amount of data needed to encompass the amount of daily bars you need.

In a backtest, make sure to select a past data that includes the amount of daily bars (keeping in mind the session dates). On a chart, use the number of primary bars to load that is equal to how many days back you need.

Additionally, does your script have a CurrentBars check for all your added data series?

CurrentBars - https://ninjatrader.com/support/help...urrentbars.htm

This way you can make sure your added 1 day series has enough bars loaded in order for you to access the index you are looking for.

Please let me know if you have any further questions.

[GregX999]

Hi Gaby,
Thanks for the reply! For further clarification...

If you are adding a daily series, but using a more granular primary data series, you'll need to add the amount of data needed to encompass the amount of daily bars you need.

I'm not sure how I would do this in the strategy analyzer. Is that the "Maximum Bars Lookback" setting?

In a backtest, make sure to select a past data that includes the amount of daily bars (keeping in mind the session dates). On a chart, use the number of primary bars to load that is equal to how many days back you need.

Not quite following you here... "make sure to select a past data that includes the amount of daily bars (keeping in mind the session dates)".

When running in a chart (in Playback mode) I don't get this issue as I've downloaded sufficient days of historical data for my tests, and the default "the number of primary bars to load" is sufficient.

Additionally, does your script have a CurrentBars check for all your added data series?

Yes, I usually use CurrentBars to check for such things. In this case I don't, because I want the script to fail if there's not enough data for it to run as expected (since I'm primarily in "testing mode" right now).

[NinjaTrader_Gaby]

Hello,


Sorry, there was a typo in my original response. In the Analyzer, make sure you select a past date that includes the amount of daily bars you need. For example, if in your script you need to load at least 3 daily bars, make sure you are loading at least 3 days in the analyzer.


Your script should have a CurrentBars check for all added data series to avoid errors. You can have the script return if the CurrentBars requirement is not met.

Removing the CurrentBar check is not a way ensure it has a far enough lookback period. You can make other type of conditions to make sure you have selected a far enough lookback period, but the script absolutely needs a currentbar check for each series when using BarsAgo on a series.

Please let me know if you have any other questions.

[GregX999]

Sorry, there was a typo in my original response. In the Analyzer, make sure you select a past date that includes the amount of daily bars you need. For example, if in your script you need to load at least 3 daily bars, make sure you are loading at least 3 days in the analyzer.

This was my issue in the original question... if I select a Start Date and End Date in the analyzer that covers, let's say, 5 days, then it loads 5 days of data, enough to cover that range. But... while running inside the first day it doesn't have access to a previous day - for that, it would need to load an extra day of data for the day previous to the Start Date.

How can I get the analyzer to load more than just the days to cover the selected range?

Your script should have a CurrentBars check for all added data series to avoid errors. You can have the script return if the CurrentBars requirement is not met.

Removing the CurrentBar check is not a way ensure it has a far enough lookback period. You can make other type of conditions to make sure you have selected a far enough lookback period, but the script absolutely needs a currentbar check for each series when using BarsAgo on a series.

I only have it removed temporarily while I'm working on this problem. Normally I would have it. (I use it in a bunch of other areas of my script already.)

But you did say "You can have the script return if the CurrentBars requirement is not met." --- is there a way to have the strategy stop immediately besides having it throw an error?

[NinjaTrader_Gaby]

Hello,

What do you mean by 'the first day doesn't have access to the previous day'? Is this when you are getting the index error?

If so, this is why the CurrentBars check is necessary. For example, if you needed 5 days to load across all your data series.

// Evaluates to make sure we have at least 5

// or more bars in both Bars objects before continuing.

if (CurrentBars[0] < 5 || CurrentBars[1] < 5 || CurrentBars[2] < 5)

return;

It is not possible to have the analyzer load additional data before the specified start date. A CurrentBars check ensures you have enough bars in across all data series for your script to process any barsAgo logic.

No, there is not another way to have the strategy 'stop' besides having error handling, like try/catch blocks.

https://ninjatrader.com/support/helpGuides/nt8/using_try-catch_blocks.htm

[GregX999]

Hi Gaby,
I really appreciate your patience with me!!! Thank you for that!

[QUOTE=NinjaTrader_Gaby;n1286553]
What do you mean by 'the first day doesn't have access to the previous day'? Is this when you are getting the index error?
[QUOTE]

The script runs through each day selected in the analyzer (from "start date" to "end date"). While it is running through the first day, CurrentBars[dayIndex] is equal to 0. There is no previous day's data for the script to access, so trying to access it throws the error. But my script relies on knowing the previous day's high and low.

Assuming I select a Start Date and End Date that encompass 5 days (ie: a Monday and the following Friday)...

If I print BarsArray[dayIndex].Count inside State.DataLoaded, it prints a number equal to the number of days I selected... so 5 in this case.

But printing BarsArray[dayIndex].Count inside OnBarUpdate() (when BarsInProgress == dayIndex), prints 1 - not 5.

And printing CurrentBars[dayIndex] prints 0.

And since the BarsArray only contains 1 bar, obviously it would throw an error if I try to access Highs[dayIndex][1].

If I comment out the code that tries to access Highs[dayIndex][1] and Lows[dayIndex][1], then the the script runs fine - and when it hits Tuesday's data BarsArray[dayIndex].Count is 2 and CurrentBars[dayIndex] is 1. And for Wednesday, 3 and 2, etc.

If, instead of just commenting that code out, I instead check CurrentBars[dayIndex] to be sure it's not 0, then the script runs all the way through, but Monday's results are not correct since previous day's High and Low weren't available, and my code uses that data to make its decisions.

In the example you gave (the code right below), if one of those CurrentBars objects was for a 1 day series, and you only select 5 days to run the analyzer (ie: from a Monday to the following Friday), then nothing would ever happen, it would just return for each of the five days, since CurrentBars[dayIndex] would be 0 when running Monday's data, and 4 when running Friday's data.

// Evaluates to make sure we have at least 5
// or more bars in both Bars objects before continuing.
if (CurrentBars[0] < 5 || CurrentBars[1] < 5 || CurrentBars[2] < 5) return;

I wonder if I'm just missing something obvious, as what I'm trying to do seems like it would be a common case. A strategy needing access to previous days of data seems like a totally natural thing (for moving averages, to get previous day's high and low, etc.).


An another option I did also experiment with...

BarsPeriod dayBarsPeriod = new BarsPeriod() { BarsPeriodType = BarsPeriodType.Day, Value = 1 };
AddDataSeries("ES 12-23", dayBarsPeriod, x, "CME US Index Futures ETH", null);

Which "force loads" x bars. It works if x is more than the number of days I've selected to test (ie: If I test M-F and set x=6).
But if x is less than the number of bars needed, it only loads data for the last x bars needed. (ie: If I try to test M-F, but x==3, it only loads data for Wed, Thurs and Fri)

This could be a viable solution if I could somehow set x to be equal to the number of days selected in the analyzer. Is that possible (Maybe with "if (IsInStrategyAnalyzer)" --- is there a way to access the start and end dates?)

But, to do this I'd also need a way to know what symbol is the current symbol. I know about "Instrument.FullName", but that doesn't seem to work in State.Configure, where I'm loading the data series.

Is there a way to make that option work?

[NinjaTrader_Gaby]

Hello,

That was just an example checking if there was at least 5 bars loaded. You will need to set the start date to 1 day before you want the strategy to start processing.

Use a CurrentBar check to make sure at least 1 daily bar was processed, if not return.

After 1 daily bar has been processed, the CurrentBar check will allow the strategy to start working and process the rest of the strategy's logic. At that point you can access the previous day's data on the actual start date you intended.

To illustrate how the CurrentBar check works, try adding this print statement to the top of OnBarUpdate():

Print(string.Format("{0} | BarsInProgress: {1}, CurrentBars[{1}]: {2}", Time[0], BarsInProgress, CurrentBars[BarsInProgress]));

Please note that adding data series dynamically is not supported. From the Help Guide:

"Arguments supplied to AddDataSeries() should be hardcoded and NOT dependent on run-time variables which cannot be reliably obtained during State.Configure (e.g., Instrument, Bars, or user input). Attempting to add a data series dynamically is NOT guaranteed and therefore should be avoided. "

https://ninjatrader.com/support/helpGuides/nt8/adddataseries.htm

[GregX999]

Use a CurrentBar check to make sure at least 1 daily bar was processed, if not return.

After 1 daily bar has been processed, the CurrentBar check will allow the strategy to start working and process the rest of the strategy's logic. At that point you can access the previous day's data on the actual start date you intended.

So, if I understand what you're saying (and it's very possible I'm still not getting something)...
If I want to get results for my strategy in the analyzer, lets say, from Monday Oct 2nd to Friday Oct 6th (so 5 days), then I would need to actually select Friday Sept 29 as the start date? (And just ignore the results for that day? Ideally, program my script to not take any trades when CurrentBars[dayIndex] == 0?)

And hypothetically, if my strategy needed 20 days of previous data (maybe for a 20-day moving average), then I would have to select Monday Sept 4th as the start date - selecting a range of 25 days in order to get 5 days of data? (Since the first 20 days would need to be "ignored" --- when CurrentBars[dayIndex] < 20.)

If this is the only way to get the data I need, ok, it works. It's not a horrible solution (in my case, since I only need one extra day). But is this really the only way? I can't believe this isn't an issue for people trying to develop strategies that need multiple previous days of data, like the 20 in my example above.

[NinjaTrader_Gaby]

Hello,

You would need to select October 1st as the start date, or whatever the last session date was. That way when the strategy processes the October 2nd bar, at least 1 bar has already been loaded so it can check the bar 1 day ago. You only need to add an additional day to be able to check 1 day ago.

Please note that your strategy would not take trades on that first date, it's just allowing the script to process. That is the purpose of the CurrentBar check.

Can you share the output from the print statement we provided?

Try adding the CurrentBars check as we've recommended. What results are you seeing? Are you still getting the index error?

[GregX999]

Yes, it works fine if I add the following at the very beginning of the OnBarUpdate() method:

if (CurrentBars[dayIndex] == 0) return;​

It basically skips the first day - doesn't take any trades or do any logic at all.
And I get results for the subsequent days.

I totally understand the whole CurrentBar thing. I use CurrentBar checks all over my code. My issue wasn't with how to check if I had enough data, it was with how to get MORE data.

Since it seems getting more data isn't possible, then yes, doing the CurrentBar check is the work-around.

But that was my hang-up... needing to select a start date that's X days earlier than you actually want to start feels like a janky work-around, and I was having a hard time believing that's actually what needs to be done.

Being able to somehow specify that you want x number of bars/days IN ADDITION (and previous to) the selected date range feels like something that should be available to do - either in code, or in the analyzer UI (like right under where you select your start and end dates for example).

Is there nothing like that in the works? An existing proposal? Something?

[NinjaTrader_Gaby]

Hello,

The CurrentBar check is used to ensure you have enough bars to process your strategy logic. You can increase the number of bars required for the CurrentBar check if necessary.

The Strategy Analyzer cannot load additional data prior to your selected start date. You need to select a start date that ensures you have enough data to execute your strategy logic.

[GregX999]

Ok, I guess it is what it is. Thanks for helping me further wrap my head around this strategy programming/testing stuff!