Multi-timeframe strategy and indicator OnBarUpdate firing out of sync

[kiffgj]

I have re-created an issue I'm seeing with a MTF strategy and indicator combination in a very simple test example attached.
The strategy is added to a 5 min chart (I'm testing on FDAX), the strategy adds a 20 tick data series, so does the indicator.
If I don't call Update() on the indicator I never see an OnBarUpdate fired on the indicator, ONLY on the strategy.
If I call Update() on the indicator I see a very strange sequence of OnBarUpdate, to the point where the indicator ends up firing OnBarUpdate 5 mins in ADVANCE of the strategy.
If I call Update(idx, bip) on the indicator I seen an even stranger scenario where after the first Strategy OnBarUpdate where BIP=1, it calls Update on the indicator with BIP=1, the indicator proceeds to fire OnBarUpdate until the end of the chart data

Attached is the very simple strategy and indicator (Test.cs) along with the example output of the 2 different Update calls

[NinjaTrader_Jesse]

Hello kiffgj,

What you are seeing is correct in terms of calling Update, you need to call the indicator for it to be used. If the indicator does not have a plot to be accessed then you need to use its Update method. It otherwise will only work in realtime and may affect how it works based on it not having historical calculations. When hosting an indicator the strategy needs to delegate that the indicator calculates.

You can read about how multi series bars are processed in the following link: https://ninjatrader.com/support/help...lightsub=multi

The sample you provided also seems to be in a strange format, I would also suggest to generate a new empty indicator so that the indicator generated code is correct and present at the bottom inside of the region. That will be regeneated as you create the indicator so you should not be editing that part of the code. Your strategy class won't have any generated code at the bottom, it looks like you may have moved that from the indicator code into the strategy code. I would suggest to delete this test and start over as two separate files to ensure everything is correct.

As a side note you don't need to call base members on any of the overrides except OnRender which controls if plots are visible or not, you can remove base.OnBarUpdate();.



Please let me know if I may be of further assistance.

[kiffgj]

I don't use any code generation in NT, I'm a professional developer with over 30 years experience working in the banking industry.
I purposefully only included the bare minimum code that is required to demonstrate the problem, so as to not complicate the issue.
I understand the requirement to call Update() and am fine with that, what I'm stating is the order OnBarUpdate is fired in relation to the strategy is clearly wrong.
In terms of not calling base, as a developer I would normally look at the code in the base class to determine if it should be called, but given the NT code is obfuscated, it's impossible to determine, so I added it just in case.

[NinjaTrader_Jesse]

Hello kiffgj,

If you are working outside of how the platform files are normally structured then our support will be less effective. When reporting issues and attaching files I would suggest to use the standard formats so that we can accurately assist. If you include files that are structured odd or missing parts we will generally always make note of that as it falls outside the norm.

I would suggest to generate a new empty indicator with a Plot and then call that from a new empty strategy in its OnBarUpdate. Please don't add the base.OnBarUpdate call to ensure that is not part of the problem. If you are still seeing a problem with standard format files, please attach that instead and we can review that.

Please let me know if I may be of further assistance.

[kiffgj]

Hi Jesse

As you asked I have generated the code, using this code I get EXACTLY the same output and result, ie. the order of OnBarUpdate fired on the indicator is definitely wrong.

As a small exmaple I get this:
Strategy BIP:1 Idx:0 09/08/2021 08:00:01
Strategy BIP:1 Idx:1 09/08/2021 08:00:02
Strategy BIP:1 Idx:2 09/08/2021 08:00:11
Strategy BIP:1 Idx:3 09/08/2021 08:00:29
Strategy BIP:1 Idx:4 09/08/2021 08:01:02
Strategy BIP:1 Idx:5 09/08/2021 08:01:46
Strategy BIP:1 Idx:6 09/08/2021 08:02:06
Strategy BIP:1 Idx:7 09/08/2021 08:02:55
Strategy BIP:1 Idx:8 09/08/2021 08:03:24
Strategy BIP:1 Idx:9 09/08/2021 08:04:00
Strategy BIP:0 Idx:0 09/08/2021 08:05:00
Indicator BIP:1 Idx:0 09/08/2021 08:00:01
Indicator BIP:1 Idx:1 09/08/2021 08:00:02
Indicator BIP:1 Idx:2 09/08/2021 08:00:11
Indicator BIP:1 Idx:3 09/08/2021 08:00:29
Indicator BIP:1 Idx:4 09/08/2021 08:01:02
Indicator BIP:1 Idx:5 09/08/2021 08:01:46
Indicator BIP:1 Idx:6 09/08/2021 08:02:06
Indicator BIP:1 Idx:7 09/08/2021 08:02:55
Indicator BIP:1 Idx:8 09/08/2021 08:03:24
Indicator BIP:1 Idx:9 09/08/2021 08:04:00
Indicator BIP:0 Idx:0 09/08/2021 08:05:00

But it SHOULD be this:
Strategy BIP:1 Idx:0 09/08/2021 08:00:01
Indicator BIP:1 Idx:0 09/08/2021 08:00:01
Strategy BIP:1 Idx:1 09/08/2021 08:00:02
Indicator BIP:1 Idx:1 09/08/2021 08:00:02
Strategy BIP:1 Idx:2 09/08/2021 08:00:11
Indicator BIP:1 Idx:2 09/08/2021 08:00:11
Strategy BIP:1 Idx:3 09/08/2021 08:00:29
Indicator BIP:1 Idx:3 09/08/2021 08:00:29
Strategy BIP:1 Idx:4 09/08/2021 08:01:02
Indicator BIP:1 Idx:4 09/08/2021 08:01:02
Strategy BIP:1 Idx:5 09/08/2021 08:01:46
Indicator BIP:1 Idx:5 09/08/2021 08:01:46
Strategy BIP:1 Idx:6 09/08/2021 08:02:06
Indicator BIP:1 Idx:6 09/08/2021 08:02:06
Strategy BIP:1 Idx:7 09/08/2021 08:02:55
Indicator BIP:1 Idx:7 09/08/2021 08:02:55
Strategy BIP:1 Idx:8 09/08/2021 08:03:24
Indicator BIP:1 Idx:8 09/08/2021 08:03:24
Strategy BIP:1 Idx:9 09/08/2021 08:04:00
Indicator BIP:1 Idx:9 09/08/2021 08:04:00
Strategy BIP:0 Idx:0 09/08/2021 08:05:00
Indicator BIP:0 Idx:0 09/08/2021 08:05:00

Given the indicator has the same data series as the strategy, surely the strategy AND indicator should be processing each data series in the same order and in sync.
If you then look at the data beyond 08:05, you'll see the indicator tick data is processed 5 mins in advance of the strategy, clearly NOT correct.

The above is using the Update() method called from the strategy. If you instead call the overload Update(idx, bip), you will see even stranger results with ALL ticks for the entire data series being processed on the indicator after the very FIRST Update call made from the strategy.

This only appears to occur when using a secondary data series with a SMALLER time frequency of the main data series.

[kiffgj]

Digging into this a bit further and looking at the realtime output, I believe the answer is to use Update(idx, bip), as for REALTIME ONLY it appears to be working correctly. So I would guess there's a bug with Update(idx, bip) when used to process historic data that causes the inidicator to fire ALL ticks for the entire historic data on the first call.

REALTIME output using Update():
Strategy BIP:1 Idx:4359 12/08/2021 10:40:36
Strategy BIP:1 Idx:4360 12/08/2021 10:41:56
Strategy BIP:1 Idx:4361 12/08/2021 10:42:36
Strategy BIP:1 Idx:4362 12/08/2021 10:42:48
Strategy BIP:1 Idx:4363 12/08/2021 10:43:20
Strategy BIP:1 Idx:4364 12/08/2021 10:43:58
Strategy BIP:0 Idx:536 12/08/2021 10:45:00
Indicator BIP:0 Idx:536 12/08/2021 10:45:00
Indicator BIP:1 Idx:4359 12/08/2021 10:40:36
Indicator BIP:1 Idx:4360 12/08/2021 10:41:56
Indicator BIP:1 Idx:4361 12/08/2021 10:42:36
Indicator BIP:1 Idx:4362 12/08/2021 10:42:48
Indicator BIP:1 Idx:4363 12/08/2021 10:43:20
Indicator BIP:1 Idx:4364 12/08/2021 10:43:58

REALTIME output using Update(idx, bip):
Strategy BIP:1 Idx:4366 12/08/2021 10:46:02
Indicator BIP:1 Idx:4366 12/08/2021 10:46:02
Strategy BIP:1 Idx:4367 12/08/2021 10:46:20
Indicator BIP:1 Idx:4367 12/08/2021 10:46:20
Strategy BIP:1 Idx:4368 12/08/2021 10:46:34
Indicator BIP:1 Idx:4368 12/08/2021 10:46:34
Strategy BIP:1 Idx:4369 12/08/2021 10:47:29
Indicator BIP:1 Idx:4369 12/08/2021 10:47:29
Strategy BIP:1 Idx:4370 12/08/2021 10:47:55
Indicator BIP:1 Idx:4370 12/08/2021 10:47:55
Strategy BIP:1 Idx:4371 12/08/2021 10:48:47
Indicator BIP:1 Idx:4371 12/08/2021 10:48:47
Strategy BIP:1 Idx:4372 12/08/2021 10:49:42
Indicator BIP:1 Idx:4372 12/08/2021 10:49:42
Strategy BIP:0 Idx:537 12/08/2021 10:50:00
Indicator BIP:0 Idx:537 12/08/2021 10:50:00
Strategy BIP:1 Idx:4373 12/08/2021 10:50:56
Indicator BIP:1 Idx:4373 12/08/2021 10:50:56

[NinjaTrader_Jesse]

Hello kiffgj,

Thank you for the reply.

From the output you provided nothing seems incorrect. Historical data is processed like you have shown. You would get the same output if you were accessing the plot you added instead of Update as well.

If you wanted the output to be like realtime in historical you could enable TickReplay. That would let the bars build in a more granular way and would reproduce the output for realtime you are showing.

Please let me know if I may be of additional assistance.

[kiffgj]

Hi Jesse

Thanks for you reply but I'm a little confused to what is more granular than tick data. My example strategy adds tick data with a interval of 20, the strategy is run on a 5 min chart and has Calculate set to OnBarClose. So I should get an OnBarUpdate fired every 5 mintues on the primary data series and an OnBarUpdate fired every 20 ticks on the 2nd data series.

According to the documentation for Update():
"Forces the OnBarUpdate() method to be called so that indicator values are updated to the current bar."

If I call Update() [no parameter overload] - when invoked when BIP=1 on the strategy, it has NO effect on the indicator, however when BIP=0 on the strategy it fires OnBarUpdate on the indicator for the missing ticks with BIP=1, then it fires the BIP=0 that corresponds to the strategy, AND then continues to fire OnBarUpdate for FUTURE ticks on the indicator for which the strategy HAS NOT yet seen.

And example of the above scenario:
Strategy BIP:1 Idx:0 12/08/2021 08:00:00
Strategy BIP:1 Idx:1 12/08/2021 08:00:06
Strategy BIP:1 Idx:2 12/08/2021 08:00:13
Strategy BIP:1 Idx:3 12/08/2021 08:00:37
Strategy BIP:1 Idx:4 12/08/2021 08:01:02
Strategy BIP:1 Idx:5 12/08/2021 08:02:11
Strategy BIP:1 Idx:6 12/08/2021 08:03:23
Strategy BIP:1 Idx:7 12/08/2021 08:04:53
Strategy BIP:0 Idx:0 12/08/2021 08:05:00
Indicator BIP:1 Idx:0 12/08/2021 08:00:00
Indicator BIP:1 Idx:1 12/08/2021 08:00:06
Indicator BIP:1 Idx:2 12/08/2021 08:00:13
Indicator BIP:1 Idx:3 12/08/2021 08:00:37
Indicator BIP:1 Idx:4 12/08/2021 08:01:02
Indicator BIP:1 Idx:5 12/08/2021 08:02:11
Indicator BIP:1 Idx:6 12/08/2021 08:03:23
Indicator BIP:1 Idx:7 12/08/2021 08:04:53
Indicator BIP:0 Idx:0 12/08/2021 08:05:00
Indicator BIP:1 Idx:8 12/08/2021 08:05:37
Indicator BIP:1 Idx:9 12/08/2021 08:06:24
Indicator BIP:1 Idx:10 12/08/2021 08:07:20
Indicator BIP:1 Idx:11 12/08/2021 08:08:26
Indicator BIP:1 Idx:12 12/08/2021 08:09:27 <-- At this point in time the indicator has actually processed 5 mins of tick data IN ADVANCE of the strategy!!!!
Strategy BIP:1 Idx:8 12/08/2021 08:05:37

This could be put down to a side affect of not calling Update for BIP=1 I guess....

HOWEVER if I instead call the overloaded Update() method supplying idx and bip, it still doesn't work.
As I mentioned previously as soon as I invoke Update(idx, bip) from the strategy for the first time, it fires OnBarUpdate on the indicator for ALL data in the series until the both data series have been processed, at this point it reverts to processing the data series on the strategy, for example:

Strategy BIP:1 Idx:0 12/08/2021 08:00:00
Indicator BIP:1 Idx:0 12/08/2021 08:00:00
Indicator BIP:1 Idx:1 12/08/2021 08:00:06
Indicator BIP:1 Idx:2 12/08/2021 08:00:13
Indicator BIP:1 Idx:3 12/08/2021 08:00:37
Indicator BIP:1 Idx:4 12/08/2021 08:01:02
Indicator BIP:1 Idx:5 12/08/2021 08:02:11
Indicator BIP:1 Idx:6 12/08/2021 08:03:23
Indicator BIP:1 Idx:7 12/08/2021 08:04:53
Indicator BIP:0 Idx:0 12/08/2021 08:05:00
Indicator BIP:1 Idx:8 12/08/2021 08:05:37
Indicator BIP:1 Idx:9 12/08/2021 08:06:24
Indicator BIP:1 Idx:10 12/08/2021 08:07:20
Indicator BIP:1 Idx:11 12/08/2021 08:08:26
Indicator BIP:1 Idx:12 12/08/2021 08:09:27
Indicator BIP:0 Idx:1 12/08/2021 08:10:00
Indicator BIP:1 Idx:13 12/08/2021 08:12:14
Indicator BIP:1 Idx:14 12/08/2021 08:14:31
Indicator BIP:0 Idx:2 12/08/2021 08:15:00
...continues to the end of the data series without processing any further strategy data until the end


I can't see how you can state that this is correct, it obviously is NOT processing indicator data series correctly when invoked from a strategy.
As your docuementation states, calling Update() should force OnBarUpdate to be called so that indicator values are updated to the current bar, the fact there is an overload to specify the BarsInProgress clearly suggests it should consistently work reglardless of what BarsInProgress is passed as a parameter (assuming it's a valid BIP).

At the very least if you choose not to report this to the development team, the documentation should be updated to reflect the fact it only works as expected when TickReplay is enabled.
I personally cannot believe this is by DESIGN and firmly believe there is an underlying issue here that needs further investigation.

[NinjaTrader_Jesse]

Hello kiffgj,

The Update method is a utility method for scripts with no plots, it does not force OnBarUpdate to be called in a certain order, it makes sure the script is up to date with processing at that point so you can access NinjaScript properties in the file. TickReplay would be what you are looking for if you wanted to process/print like you had shown in realtime. Update would have nothing to do with changing how OnBarUpdate is called or the order it is being called.

Please let me know if I may be of further assistance.

[kiffgj]

I am completely dumbfounded by your response - you state:
"[update] does not force OnBarUpdate to be called in a certain order, it makes sure the script is up to date with processing at that point"
and
"Update would have nothing to do with changing how OnBarUpdate is called or the order it is being called"

Yet the documentation for the Update method CLEARLY states this:
Definition
Forces the OnBarUpdate() method to be called so that indicator values are updated to the current bar. If the values are already up to date, the Update() method will not be run.

https://ninjatrader.com/support/helpGuides/nt8/?update.htm

I'm not going to waste any more of my time trying to convince you there is a problem here, there clearly is - your failure to see it, or lack of desire to invesitage amazes me.

[NinjaTrader_Jesse]

Hello kiffgj,

I believe you are getting stuck on this description and how it is worded, what you are describing is not how Update is used. The description states the same thing I had said, I just worded it differently.

Forces the OnBarUpdate() method to be called so that indicator values are updated to the current bar

Here is what I said:

it does not force OnBarUpdate to be called in a certain order, it makes sure the script is up to date with processing at that point

It updates the script being called to the current bar, or the other overload which you specify a bar. That's all it does. There is no mention of changing the order of calls to OnBarUpdate or how the script processes in general or how it should be in contrast to the primary script. This is just a utility function to quickly Update only the indicator instance you called it on.

The other description on the page you had linked also outlines what use case Update is used for:

When indicators are embedded (called) within a NinjaScript strategy, they are optimized to calculate only when they are called upon in a historical backtest. Since the NinjaTrader indicator model is very flexible, it is possible to create public properties on a custom indicator that return values of internal user defined variables. If these properties require that the OnBarUpdate() method is called before returning a value, include a call to this Update() method in the property getter.

Here is a simple example of a use case for Update:
If we assume you made an indicator that has no plots but has a public string property which is being called by a strategy. That property won't get updated if you just call the property by its self. It won't have any context when its being called. Update() is used to bring the script up to date with now so you can get the correct value for right now when you called the property.

The method works as expected from your prints, if you want the prints like you had shown for realtime you can use tick replay and not use Update at all unless the indicator in question required its use.

Please let me know if I may be of further assistance.