error/bug in add() function in initialise function

[PolarBear]

Hello all,

I've been messing about with the ADD() function for a few days observing some wierd / undocumented behaviour. I would appreciate if other users or NT developers could comment

For all cases all I am trying to do is add the default/current instrument but with differing time periods. These dataseries' are then used in the OnBarUpdate() function and (barsInProgress == xxx) to do different things at different times.

All comments relate to the Add function placed in the Initialize() function

(a) problems if Add is not in the lowest { } root.
Here I was trying to be tricky and dynamically add different data series based on user inputs, however I found that this would not work if the Add function was placed a few levels deep within an if statement or similar. I imagine this has to do with some kind of variable scope and allocation, however I can't see the behaviour documented.

For example, this kind of thing did not seem to work:
if (userVariable == 1440)
{
Add(PeriodType.Minute, 1440);
} else if (userVariable == 240)
{
Add(PeriodType.Minute, 240);
}
etc...

but this does:
Add(PeriodType.Minute, 1440);
Add(PeriodType.Minute, 240);
however you end up with 2 dataseries when I only wanted one...

(b) issues with anything other than PeriodType.Minute
None of the following seemed to work. I received no errors in the log or anywhere but my program would not backtest:
Add(PeriodType.Week, 1); // week
Add(PeriodType.Month, 1); // monthly
Add(PeriodType.Month, 3); // quarterly
Add(PeriodType.Year, 1); // yearly

I put this down to possibly only PeriodType.Minute supported by the add function, however then i ran into the next problem..

(c) issues with larger numbers of .minute:

I changed my code to the following for init to get my different timescaled bars:

Add(PeriodType.Minute, 240);
Add(PeriodType.Minute, 1440); // daily
Add(PeriodType.Minute, 1440*7); // weekly
Add(PeriodType.Minute, 1440*30); // monthly
Add(PeriodType.Minute, 1440*90); // quarterly
Add(PeriodType.Minute, 1440*365); // yearly

I noticed I was getting irregular values for ATR using the different data series. I debugged to investigate further and found the following:
- BarsArray {NinjaTrader.Data.Bars[8]} NinjaTrader.Data.Bars[]
+ [0] {instrument='$AUDUSD' from='2010-01-01' to='2011-10-04' period=240 Min splitAdjusted=False dividendAdjusted=False bars=2214} NinjaTrader.Data.Bars
+ [1] {instrument='$AUDUSD' from='2010-01-01' to='2011-10-04' period=1 Min splitAdjusted=False dividendAdjusted=False bars=530900} NinjaTrader.Data.Bars
+ [2] {instrument='$AUDUSD' from='2010-01-01' to='2011-10-04' period=240 Min splitAdjusted=False dividendAdjusted=False bars=2214} NinjaTrader.Data.Bars
+ [3] {instrument='$AUDUSD' from='2010-01-01' to='2011-10-04' period=1440 Min splitAdjusted=False dividendAdjusted=False bars=369} NinjaTrader.Data.Bars
+ [4] {instrument='$AUDUSD' from='2010-01-01' to='2011-10-04' period=10080 Min splitAdjusted=False dividendAdjusted=False bars=369} NinjaTrader.Data.Bars
+ [5] {instrument='$AUDUSD' from='2010-01-01' to='2011-10-04' period=43200 Min splitAdjusted=False dividendAdjusted=False bars=369} NinjaTrader.Data.Bars
+ [6] {instrument='$AUDUSD' from='2010-01-01' to='2011-10-04' period=129600 Min splitAdjusted=False dividendAdjusted=False bars=369} NinjaTrader.Data.Bars
+ [7] {instrument='$AUDUSD' from='2010-01-01' to='2011-10-04' period=525600 Min splitAdjusted=False dividendAdjusted=False bars=369} NinjaTrader.Data.Bars


Notice here how the "bars" property for the last 5 are all the same (369)
there is clearly an issue here.

Please help asap.

thanks
PB

[NinjaTrader_Bertrand]

PB, thanks for the post - dynamic adding would unfortunately not be supported with the Add().

What datafeed / broker are you using here in testing?

The Add() would be supported for non minute timeframes as well -

http://www.ninjatrader.com/support/helpGuides/nt7/add3.htm

The Period used itself seems to be picked up ok in your last example - what kind of bars back settings are you using to create the chart - this would be defined via the primary series from the chart ui for all added series as well. How much data can you access via your provider (minute that would be)?

[PolarBear]

hi thanks for the reply.
that link doesn't tell me anything I don't already know.

the 'backBars' setting is obviously what is the issue here. Can you please explain in detail where I set this, as I cannot see this parameter in the Strategy Tester, nor as one of the parameters for the Add() function.

--- edit
I found a setting for this I believe "Maximum bars look back". I set this to infinite, and now the "bars" property seems to increase when I test back from 2006, 2007, etc.
I'm still investigating whether my other issues are solved.
---


Reffering to the post below with the BarsArray information, I'm still a little puzzled why the first few data series have larger numbers for the 'bars' property (if this is the property you refer to). From your post, shouldn't they all have the same value as the first? In any case if you can please let me know how to set the backBars property.

RE: data... i'm back testing offline. I have excellent CQG 1 minute data going back from Jan 2006 to now.

thanks in advance

[PolarBear]

Also one other point after re-reading your post.
"The Add() would be supported for non minute timeframes as well -
http://www.ninjatrader.com/support/h...s/nt7/add3.htm"

I know the add function supports other timebases: eg ticks, volume, etc as per the documentation.

what is NOT clarified, is if the other time based periods are supported - day, week, month, year
as described I have experienced wierd issues that I cannot explain, and my solution was just not to use the month, week, etc, but to use multiples of minutes which is messy, and I would think not very good from a memory point of view for the underlying NT code.

I would ask that you comment on what you have experienced in real coding not just what "would be supported". I'm interested in real test code and examples that have been tested using these different time periods.

thanks

[PolarBear]

Summary of current issue

I really need help with this. I'm beating my head against the wall for days.

I have several data series (different time periods). I simply want the ATR done on these when there's a new bar of that time period.

In the Initialise function:
Add(PeriodType.Minute, 1); // data series 1 (data series 0 is the series we run the backtest on)
// other data series' USED FOR ATR UPDATE
Add(PeriodType.Minute, 240);
Add(PeriodType.Minute, 1440); // daily
Add(PeriodType.Minute, 1440*7); // weekly
Add(PeriodType.Minute, 1440*30); // monthly
Add(PeriodType.Minute, 1440*90); // quarterly
Add(PeriodType.Minute, 1440*365); // yearly


// also inits for variabel listed below (MAX_TIME_PERIODS = 5)
// atr calcs init
tmpATR = new double[MAX_TIME_PERIODS];

for (int i=0; i< MAX_TIME_PERIODS; i++)
{
tmpATR[i] = 0.0;
}


---------------------
In the bar update function:


if ((BarsInProgress >= 2) && (BarsInProgress < 7))
{
// ATR update
tmpATR[BarsInProgress-2] = ATR(BarsArray[BarsInProgress], aTRZonePeriod)[0];

}
---------------------
globals:
tmpATR is defined as :
public double[] tmpATR ;


--------------
when I run this and debug, I get the issue described below, where it appears there's not enough bars for the higher timeframe dataseries'.
The ultimate issue is that the ATR value of the first is correct, but then the ATR value of the other 4 are all the same (presumably, because there's not enough bars and they have the same number of bars - refer below).

Please help!
thanks in advance


- BarsArray {NinjaTrader.Data.Bars[8]} NinjaTrader.Data.Bars[]
+ [0] {instrument='$AUDUSD' from='2010-01-01' to='2011-10-04' period=240 Min splitAdjusted=False dividendAdjusted=False bars=2214} NinjaTrader.Data.Bars
+ [1] {instrument='$AUDUSD' from='2010-01-01' to='2011-10-04' period=1 Min splitAdjusted=False dividendAdjusted=False bars=530900} NinjaTrader.Data.Bars
+ [2] {instrument='$AUDUSD' from='2010-01-01' to='2011-10-04' period=240 Min splitAdjusted=False dividendAdjusted=False bars=2214} NinjaTrader.Data.Bars
+ [3] {instrument='$AUDUSD' from='2010-01-01' to='2011-10-04' period=1440 Min splitAdjusted=False dividendAdjusted=False bars=369} NinjaTrader.Data.Bars
+ [4] {instrument='$AUDUSD' from='2010-01-01' to='2011-10-04' period=10080 Min splitAdjusted=False dividendAdjusted=False bars=369} NinjaTrader.Data.Bars
+ [5] {instrument='$AUDUSD' from='2010-01-01' to='2011-10-04' period=43200 Min splitAdjusted=False dividendAdjusted=False bars=369} NinjaTrader.Data.Bars
+ [6] {instrument='$AUDUSD' from='2010-01-01' to='2011-10-04' period=129600 Min splitAdjusted=False dividendAdjusted=False bars=369} NinjaTrader.Data.Bars
+ [7] {instrument='$AUDUSD' from='2010-01-01' to='2011-10-04' period=525600 Min splitAdjusted=False dividendAdjusted=False bars=369} NinjaTrader.Data.Bars

[NinjaTrader_Bertrand]

PolarBear, for debugging your code I would strongly recommend to reduce the # of series you're working with here, it will make tracking and understanding of the issue much easier.

The MaximumBarsLookback refers to the # of data points accessible for your custom data series used in the code - http://www.ninjatrader.com/support/h...rslookback.htm

Yes, the Add() would work for monthly, weekly and daily frames as well.

Where do you then run your Init() logic? If this would be in the Initialize() I would recommend moving it to OnStartUp() instead as this would be run only once, Initialize() could be run multiple times.

For those synthetic higher timeframes bars using the minute bar multiples - what session template do you use and which feed / data-source?

For anything higher than 1440 min you would need to create a custom session template, for example spanning a full week without any session breaks.

[PolarBear]

this sounded good,
however I get an exception error that the "add" method can only be ran from the initialise method. I'm pretty sure the add method has to be in the init method referring to the documentation.

I'm not really sure what the suggestions here will help. I could try taking out some of the data series, but that doesn't really help as i need the data series to compute the ATR as advised.

I really wish that someone was able to help here. I've tried to provide as much info as possible including code, so it's very quick to quickly run and test to see if others are experiencing the same issues.

are there any NT support people here to do this?

thanks

[PolarBear]

could you please explain more (possibly how I would go about doing this) the comment about:
"For anything higher than 1440 min you would need to create a custom session template, for example spanning a full week without any session breaks."

thanks again

[NinjaTrader_Bertrand]

PolarBear, can you please email us the complete script you attempt to use to support at ninjatrader dot com?

The Add() method will definitely need to be in the Initialize(), however I was referring to your custom logic initing the ATR calcs, this should be moved to OnStartUp().

For the session template, simply create yourself a custom session template under Tools > Session Manager spanning from Sunday evening to Friday evening, the default Forex spot hours. This would be then done in one line so no session breaks are introduced. You can then create a chart with minute based data using this custom template, it would be set in the Format Data Series dialog box.

[PolarBear]

Hi,

thank you so much for the assistance. I have sent the code and a screenshot to the support email address advised.

thanks

[NinjaTrader_Austin]

Hi PolarBear, we've received your email and have directed it to Bertrand's attention.

[PolarBear]

Hi Bertrand,
Have you had a chance to work out what the issue is with this code?

thanks

[NinjaTrader_Bertrand]

I replied yesterday to the direct email you sent to me - the code you supplied would unfortunately not compile, could you please recheck what you sent us for testing?

Thanks,

[PolarBear]

hi, i've resent the code back again. please if you could reply as soon as possible that would be appreciated.

thanks

[PolarBear]

limitation with Add() int period maximum.

I have been following this up with NT support, and thought I'd post the response here, as I believe this to be useful for others to know. There is a inferred limitation with the maximum amount you can put in the int period parameter for the Add() function. The reason is that NT has session templates, and apparently, the maximum (in my case for forex) is one week, so NT will automatically split the bars up into 1 week segments if you go larger than that.
Very unhelpful if you are trying to calculate monthly , quarterly, values, etc...
The solution as below is to use daily bars they say, rather than minute bars

cheers
PB

With your minute emulation approach you run into the limitation that the longest continuous session you could define is actually one week long for FX (Sunday evening start going to Friday evening, created under Tools > Session Manager), so any bars you try to create being longer than a week > would not work since NT7 would force a session break here every week. Expanding those capabilities in on our suggestion list already for future consideration.

I would hence strongly suggest to work your strategy script with the native daily data here.