Using Local Variables for Permanent Storage of Exported DataSeries

[aviat72]

I want to run some calculations on DataSeries values exported by an indicator.
Can I use a local variable in my Indicator and assign it to the data-series once, or do I need to call the exporting indicator explicitly every time I need to access the exported data-series?

Will this work:

private DataSeries localDS;
...
Initialize(){
init = false;
...
}

OnBarUpdate()

if(!init){
   localDS = ExportingIndicator(BarsArray[0],....).ExportedDataSeries;
   init = true;
}

// Do Calc on localDS

OR
Do I need to lookup the Indicator explicitly in every call to OnBarUpdate

private DataSeries localDS;
...
OnBarUpdate(){
 localDS = ExportingIndicator(BarsArray[0],....).ExportedDataSeries;
//Do Calc on localDS

}

--------------------------

In the CallingIndicator which is calling the ExportingIndicator, the CalculateOnBarClose = false;

If I do not explicitly make the call to the indicator in every barUpdate (case #1), will the Update function on the ExportingIndicator still be called?

Is there something I can do explicitly in the CallingIndicator to ensure that the ExportingIndicator Updates without modifying the export property of the ExportingIndicator to call Update?

I am collecting Tick based stats in the ExportingIndicator, so I do not want the ExportingIndicator to update twice for the same tick.

[NinjaTrader_RyanM1]

Hello aviat72,

Have you seen this reference sample on exposing indicator values that are not plots.

This alows you to access indicator values in another strategy or indicator. Let us know if this isn't what you're looking for.

[aviat72]

Hi Ryan:

I guess a better way to frame my question would be.

Instead of the code below

if (CrossAbove(MACD(12, 26, 9), MACD(12, 26, 9).Avg, 1))
			{
				// Paint the current price bar lime to draw our attention to it
				B

I write

private DataSeries macdAvg;
private DataSeries macd;

OnBarUpdate(){

// Local variables initialized only once..
if(!init){
macdAvg = MACD(12,26,9);
macd = MACD(12,26,9).Avg;
init = true;
}


if (CrossAbove(macd, macdAvg, 1))....

will things still run fine even though the explicit call to MACD(....) are missing during the execution?

I think the answer is no but just to confirm.

[NinjaTrader_Bertrand]

aviat72, this will not work as you never populate the DataSeries then.

[aviat72]

aviat72, this will not work as you never populate the DataSeries then.

Thanks Bertrand. It seems that even though the local variable refers the exported data-series, the ExportingIndicator also needs to update properly. Is that right?

I wanted to get a better understanding of the sequencing of execution of calls to update indicator data-series.

I.
It which cases should the ExportingIndicator include an explicit call to Update() in the ExportingFunction?

Should I include a call to Update() by default in all ExportingFunction assuming NT will not call Update() twice on the same tick for the same indicator; i.e. NT will ensure that tick based statistics are not double counted because of an intrinsic call to Update and explicit user call(s) to Update().

I think this has to do something with the CalculateOnBarClose flag settings too.
----
II.
On a related note, I also want to clarify the index to use when accessing information about the just completed bar.

I have a CallingIndicator which is using an exported data-series from an ExportedIndicator. The CallingIndicator needs to sample the value at the bar-close. The ExportedIndicator needs tick data. I have set CalculateOnBarClose as false in the CallingIndicator and am using a FirstTickOfBar filter. The ExportedIndicator already has it as false by default.

In the CallingIndicator, I do not make the call to get the ExportedDataSeries until FirstTickOfBar is true. It seems this ensures that the ExportingIndicator gets updated one every tick, while the CallingIndicator processes the results only on when the bar closes. Please do confirm.

I also noticed that in the CallingIndicator I have to go back one bar to get the data generated in the bar just closed. ExportedDataSeries[0] is essentially empty because it now points to the new bar being created, while what I want is the data in bar just closed which is now in ExportedDataSeries[1]. Please do confirm.

Also does the situation change at all in the following scenarios:
(1) CallingIndicator also needs to update on every tick and hence the ExportedDataSeries is accessed on every OnBarUpdate method. I presume ExportedDataSeries[0] will have the relevant data pertaining to the current bar. What happens when the tick which results in a new bar (FirstTickOfBar) comes in? Will the CallingIndicator continue to use ExportedDataSeries[0] if it wants to some bar-aggregate based processing or should it now use ExportedDataSeries[1] to get the aggregate information from the the bar which just finished.

(2) CallingIndicator needs to update on every tick but the ExportingIndicator is by default set to update on bar close only (i.e. CalculateOnBarClose is true by default). I think in this case NT will update the ExportingIndicator on every tick ignoring the (local) default value of CalculateOnBarClose but use the value inherited from the CallingIndicator which needs updates on every tick. In this case, would we also have to go back to ExportedDataSeries[1] if we have to process aggregate statistics of the bar just closed (i.e. FirstTickOfBar is true in CallingIndicator).

[NinjaTrader_Bertrand]

The snippet that you provided would not compile, as you Set the dataseries with double values for each bar you cycle through the historical OnBarUpdate(). The double has no 'history' thus the values do not get as you would need them. Update() would only be used if you'd exposed internal calcs which needed an OnBarUpdate() prior to be accessed to be up to date.

When you're using CalculateOnBarClose = false and call calcs to be done on close (so on next FirstTickOfBar) you would need index / reference a bar further back, correct to get the last value representing the closed bar one.

Also there's a bug with overwriting CalculateOnBarClose in hosted indicators / methods, as such please do not set anything, just remove entries in the Initialize() related and set this from the 'mother' i.e. calling indicator or strategy only.

Would the above change anything in your findings?

Would have sample I code run to check into, which would clarify your questions?

Thanks

[aviat72]

Bertrand:

Thanks for your reply.

(I)Regarding the local DataSeries variable.
I think it boils down to what it means to assign a DataSeries to another DataSeries; i.e. LocalDataSeries = ExportingIndicator(...).ExportedDS;
Does it just copy the values as they exist now in the ExportedDS to LocalDataSeries or does it make LocalDataSeries a reference (perhaps readonly) to ExportedDS?

It would be great if it were a reference since then you could bypass the call to the cache lookup functions to identify the correct instance of the ExportingIndicator Object. Not only that, but if a DataSeries assignment actually involves a copy, it is going to be a performance hog, especially if it is not limited to 256 bars of data.

I would reconsider the exporting mechanisms to just export the relevant subset of the dataSeries, instead of the raw object itself, if the assign operator does a bar by bar copy.

(II) Behavior of Update
Right now I do not have any specific case to present, but I want to understand the architecture better so that I can write code with confidence.

(i) In indicators which collect aggregate statistics, will multiple calls to Update() mess up things by calling the OnBarUpdate() multiple times for the same tick? This can happen if you have two ExportingFunctions and each of them is called by a CallingIndicator. If both of them have an Update() call, and this results in two separate calls to OnBarUpdate(), it will be a problem when collecting aggregate statistics.

(ii)From what I understand of NT's core architecture, the ExportingIndicators inherit the CalculateOnBarClose property of the CallingIndicator. It would be great if this behavior was explicitly documented and tested under its different nuances (i.e. when the default behavior is different from the inherited behavior). The behavior when there are multiple layers of nested calls to Indicator adds another dimension to the issue. Another aspect to keep in mind when the ExportingIndicator is also running live in another chart with the same arguments but a different CalculateOnBarClose property value. I do not think that the cache check is checking for this property in the signature match logic.

(iii) I am currently accessing ExportedDataSeries[1] to get the statistics of the bar which just closed when FirstTickOfBar is true in the CallingIndicator. The behavior is perfectly understandable and consistent with good engineering design practices. What I would like is an official confirmation of this behavior under different combinations of CalculateOnBarClose on the CallingIndicator and the ExportingIndicator. Perhaps there is nothing to confirm and the behavior is the same, but I need to be sure.

[NinjaTrader_JoshP]

1. Let us take a step back and evaluate what your final goals here are. If you are simply trying to get values from an indicator's DataSeries then just access the indicator's DataSeries directly provided it has been exposed.

2a. Depends on how you have it coded. If you have things like counters aggregating, then yes, you would need to deal with that in the code.

2b. No, you should not have a CalculateOnBarClose line at all in the hosted indicator. This is critical that the hosted has no line at all. Not true or false.

2c. CalculateOnBarClose should only exist in the host script and never in the hosted script. That is the only way to ensure both are calculating "in sync" with each other. If you have been doing otherwise, please follow the suggestion in 2b and it should clear things up drastically. I would try this first to ensure that you weren't simply running into this issue before trying anything with Update().

[aviat72]

Josh:

0. I presume the host script is the CallingIndicator and the hosted script is the ExportingIndicator (i.e. the indicator being called from the CallingIndicator). Please confirm.

1. The objective here is to use values calculated in the ExportingIndicator without doing unnecessary data-series copies. Ideally, the CallingIndicator should have read only access to the ExportedDataSeries i.e. the LocalDataSeries is a read-only reference to the ExportedDataSeries.

However if every assignment calls results in the entire content of the data-series getting copied over into the LocalDataSeries object, then that objective is defeated. I would rather export the scalar values (or a small array) pertinent to the calculations being done in the Calling (hosting) indicator instead of the entire data-series; in many cases all you need is access to are the last few bars of data.

Originally I wanted to avoid the cache lookup by using a local reference to the exported DataSeries object. However the discussion has resulted in more questions. One directly relevant being the behavior of the assign operator with DataSeries object (reference or a bar by ar copy). The other of course is the general issue of how NT synchronizes execution of hosted and hosting indicators to not only ensure up to date values, but also to prevent multiple calls to hosted indicators for the same tick.

2. From what I understand, you want the CalculateOnBarClose property to be set only on the CallingIndicator (hosting) and not set on any of the CalledIndicators (hosted). Essentially what this means is that any indicator which can be used as a hosted indicator (which by default means all which export values) should not have the CalculateOnBarClose property set at all! Am I understanding this correctly because I feel this is an unusually restrictive requirement.

You may often want tick by tick updates when the hosted indicator is being used independently; so locally the hosted indicator will have CalculateOnBarClose = false. However if the hosted indicator is included in another hosting (calling) indicator the exported data-values may be sampled at bar close only.

For indicators which do not do aggregation this is not an big issue since even if the hosted indicator does not get called on every tick, the calculation done on bar close would still be valid and all that is needed.

However if the hosted indicator does aggregate computation based on tick data, then it has to run in the tick by tick mode, regardless of whether the hosting indicator is doing. The hosting indicator may need the values at bar close only, but the hosted indicator needs to do tick by tick processing.

3. Update() function:
It would be great to get a more precise description of how the NT handles multi-indicator synchronization. Does NT guarantee that on any indicator object instance, the OnBarUpdate() function is called once and only once for each tick, independent of the number of places the Update() function call is placed? Or should the user start adding explicit code to ignore multiple calls for the same tick?

A typical example would be a case where an indicator exports multiple data-values and each of the get function associated with the exported values has an explicit call to Update(). If the hosted function accesses more than one exported value or if there are more than one hosted indicator instances, there will be potentially multiple calls to Update() for the same tick. Does NT ensure that the OnBarUpdate() is not called multiple times? This is critical for indicators which collect aggregate values.

I do realize that if this indicator has multiple input series (say a 1 minute and 5 minute input series on the same instrument) that the OnBarUpdate will be called multiple times for the same incoming tick but with different BarsInProgress varlues.

[NinjaTrader_JoshP]

0. Yes. The host calls the hosted.

1. If this is your objective all you need to do is expose what you want in the hosted script and then access that from the host. Please see here: http://www.ninjatrader-support2.com/...ead.php?t=4991 Same concept applies to anything you want to expose whether that be BoolSeries (as in the example) or DataSeries objects.

2. Yes you are correct. The CalculateOnBarClose should only exist in the host and not the hosted. If you want to use the indicator separately you will have to set the setting separately.

3. Each time you call Update() NT will see if it needs to update OnBarUpdate() of the hosted script or not. Because NT is multi-threaded it can be the case that the host receives the event before the hosted, leaving the hosted calculations one step behind the host when calling something like [0]. When you have Update() in the property getter of whatever you are trying to access that is exposed in the hosted script it will get the hosted script up to speed if the host is ahead of the hosted. Development has mentioned that this does not introduce any double OnBarUpdate() event for the same tick.

[aviat72]

Josh

(0) First of all thanks for confirming that the NT update logic ensures that OnBarUpdate gets called only once regardless of the number of Update calls. This is critical especially for indicators which aggregate.

(1) I understand the export mechanism etc. quite well. What I wanted to get is a handle on is the performance cost of exporting a data-series. If every assignment to an exported data-series results in a copy of the data-series, then perhaps I need to export less than the entire data-series; i.e. create a smaller array of the size needed and export that instead of the entire data-series. Data-series without the 256 size limitations can be very long and a copy of all the data, every time the exported value is accessed can result in a lot of unneeded copy operators.

(2) I am not sure I understood the requirement of the CaluclateOnBarClose properly. NT will propagate the flag value of the host series, down to all hosted indicators, regardless of the local setting in the hosted script, right? Please confirm. Ideally they should not exist in the hosted indicators but most default indicators have some value set so it is hard not have have something in there. As long as those values are ignored it is not a problem.

When you are using the indicator explicitly in a chart, then you have independent control so these issues are not relevant.

They do become relevant is host is sampled on bar close (and hence CalculateOnBarClose = true is the natural setting), but by default the hosted need tick data (and hence default values are CalculateOnBarClose = false). In this case we should use the host with the CaluclateOnBarClose = false and then use the FirstTickOfBar filter. This will allow all the hosted indicators to run in the tick-mode.

We should however be conscious that the hosted indicators would already have moved on to the next bar (assuming Update has been called) and hence index offset by 1.

[NinjaTrader_JoshP]

1. When you call the indicator you are not exporting or copying anything. You are simply running an instance of the indicator and accessing those calculations directly. There is no penalty other than that of accessing the indicator.

2. Incorrect. You simply cannot have any CalculateOnBarClose line in hosted indicators. The line can only exist in the host and nowhere else. This is absolutely critical. In 6.5, some system indicators may have the line, but that is why they have been removed in NT7. You can consider this as a limitation if you want to think of it in that regard, but it is necessary you be mindful of this requirement.

[aviat72]

Josh:

(1) Regarding access to exported values: When you export a data-series and assign it to a local variable, does it create a new copy of the data-series or creates a reference to the data-structure existing inside the hosted indicator. This depends on the internal implementation of the DataSeries class.

I think it is a copy of the data-series since otherwise the hosted indicator could actually change the values of by setting values.
-If not then it is great for performance but I will have to be very careful about not overwriting anything in the local variable (LocalDataSeries).
-If it is indeed a copy I will rethink what to export to prevent unneeded data-series copies.

(2) Thank you for clarifying the difference in 6.5 and 7 regarding CalculateOnBarClose. I will make sure that my custom indicators too are not setting it.

[NinjaTrader_JoshP]

1. If you actually assign/set it to something, then you would have a duplicate. What I am saying is there should be no reason you would need to assign it to something though. Just call the series from the indicator directly whenever you need it.

2. Not a problem at all. Glad we are on the same page.

[aviat72]

Jason

In my hosting indicator I use multiple nearby indices to sample the ExportedDataSeries. That is why I was using a local variable. Writing the full name of the indicator with all the arguments is much more cumbersome than just writing the name of the local data series.

(localDS[1] - localDS[2]) is much easier to write and maintain than:
(HostedIndicator( input,param1,param2, ....).ExportedVariable[1] -
HostedIndicator(input, param1, param2, ....).ExportedVariable[2])

I need to rethink my architecture to reduce the performance penalty.