RemoveLastBar() question

[Harry]

Example WMA

The current code for the WMA looks like this:

if (BarsArray[0].BarsType.IsRemoveLastBarSupported)
{
    if (CurrentBar == 0)
        Value[0] = Input[0];
    else
    {
        int back = Math.Min(Period - 1, CurrentBar);
        double val = 0;
        int weight = 0;
        for (int idx = back; idx >= 0; idx--)
        {
            val += (idx + 1) * Input[back - idx];
            weight += (idx + 1);
        }
        Value[0] = val / weight;
    }
}
else
{
    if (IsFirstTickOfBar)
    {
        priorWsum = wsum;
        priorSum = sum;
        myPeriod = Math.Min(CurrentBar + 1, Period);
    }
    wsum = priorWsum - (CurrentBar >= Period ? priorSum : 0) + myPeriod * Input[0];
    sum = priorSum + Input[0] - (CurrentBar >= Period ? Input[Period] : 0);
    Value[0] = wsum / (0.5 * myPeriod * (myPeriod + 1));
}

Unless I am mistaken, you could replace that code with

if(CurrentBar == 0)
     Value[0] = Input[0];
else
{     
     if (IsFirstTickOfBar)
     {
        if(CurrentBar == 0)
        {
             priorWsum = 0.0;         
             priorSum = 0.0;
        }
        else
        {
             priorWsum = wsum[1];
             priorSum = sum[1];
        }
        myPeriod = Math.Min(CurrentBar + 1, Period);
     }
     wsum[0]  = priorWsum - (CurrentBar >= Period ? priorSum : 0) + myPeriod * Input[0];
     sum[0] = priorSum + Input[0] - (CurrentBar >= Period ? Input[Period] : 0);
     Value[0] = wsum[0] / (0.5 * myPeriod * (myPeriod + 1));
}

This code would run much faster on Renko bars compared to the original code which loops through all values of the lookback period. For standard bar types it would consume a little more memory which can be neglected as there are only two additional series with 256 fields. The impact on speed for standard bar types can also be neglected.

[Harry]

Hello Harry,

Thank you for your patience.

Upon further research we have found that this behavior is very deep into the core logic of NinjaTrader. As this is the case, it is not something that we are going to re-visit in the foreseeable future. It is expected behavior for bar types that remote that last bar in order to rebuild the current bar.

Please let me know if I may be of further assistance.

Thank you again for your detailed explanations.

Following our discussion I have reviewed my own indicators and also found 12 NinjaTrader system indicators that return false values on Renko bars.

http://ninjatrader.com/support/forum/showthread.php?t=102713

[NinjaTrader_PatrickH]

Hello Harry,

Thank you for your information on the system indicators.

My colleague will update the other thread with the id for the request.

[Harry]

Renko Bar Processing with Secondary Bar Series

I would like to come back on the Renko Bars processing.

I have understood that the prior bar is processed again in Calculate.OnEachTick, such that the bars are processed in the ordercurrent bar (last tick) - prior bar - current bar - next bar (first tick). I believe that this way of bar processing is too complicated, but let it be.


Secondary bars added to Renko bars not processed properly

A real problem comes up, when secondary bars are added to Renko bars via an indicator. In this case the next tick of the secondary bar series is processed in the wrong place. Usually NinjaTrader processes the secondary tick after the primary tick. However in this case the new secondary tick is processed between the two preceding bars instead of where it belongs (after the new primary tick).

Expected: current bar (last tick) - secondary bar (last tick) - prior bar (revisited) - current bar (revisited) - next bar (new tick) - secondary bar (new tick)

Observed: current bar (last tick) - secondary bar (last tick) - prior bar (revisited) - secondary bar (new tick) - current bar (revisited) - next bar (new tick)

Basically this means that the new tick of the secondary bars is processed between the two prior bars or bar series 1. It does not belong there!


Problem with a new session

Let us look how this plays out at the session break!

This gets really ugly. The first tick of the first secondary bar of the new trading day is processed between the second but the last and the last primary bar of the preceding trading day.

Here is a real case from today (BIP=1: 3-Renko, BIP=2: 1-minute, BIP=3: daily)

historical bar processing on prior day as expected ….
BIP = 0 CB = 839 24.08.2017 22:14:59 // last Renko bar closed
BIP = 1 CB = 2844 24.08.2017 22:15:00 // last minute bar closed
BIP = 2 CB = 7 24.08.2017 22:15:00 // daily bar closed

Session Break (all bars prior to session break historical, all bars after session break real-time)

BIP = 0 CB = 838 24.08.2017 22:01:30 // Renko bar 838 reinserted from yesterday
BIP = 1 CB = 2845 25.08.2017 15:31:00 // today’s first minute bar 2845 processed
BIP = 2 CB = 8 25.08.2017 22:15:00 // today’s daily bar processed
BIP = 0 CB = 839 24.08.2017 22:14:59 // Renko bar 839 reinserted from yesterday
BIP = 0 CB = 840 25.08.2017 15:30:01 // first Renko bar 840 processed for today
BIP = 0 CB = 840 25.08.2017 15:30:01 // first Renko bar 840 processed again
BIP = 1 CB = 2845 25.08.2017 15:31:00 // today’s first minute bar 2845 processed
BIP = 2 CB = 8 25.08.2017 22:15:00

As you will notice the first secondary ticks of the new trading day are processed between the last two primary bars of the prior trading day.

I would consider this a bug.

[NinjaTrader_PatrickH]

Hello Harry,

Thank you for your post.

We will test this further with your latest detail. I will follow up when we have any details on this matter.

[Harry]

In fact there is a workaround for this problem.

As each Renko bar is processed three times

-> regular processing (when CurrentBars is increased by one)
-> second processing (intermediate call after the prior bar has been reprocessed)
-> third processing (revisiting the bar after the last regular tick of the consecutive bar)

you would simply pass values calculated from the secondary bar series, when the Renko bar is processed the first time. The second and third processing are both ignored, as the values written to any primary DataSeries might be false, as they are calculated from future ticks.

It is a bit tricky to deal with this issue, but feasible.

[HalTech]

Hello Harry,

Thank you for your patience.

IsRemoveLastBarSupported == true Bars Types, like Renko, can update data between bars (and thus can't be used in Tick Replay). In the SMA indicator, when IsRemoveLastBarSupported == false and we have a strong guarantee that data between bars can not change, once Period bars have elapsed, we see this code:

double last = Value[1] * Math.Min(CurrentBar, Period);
Value[0] = ((last + Input[0]) / (Math.Min(CurrentBar, Period) + 1));

When IsRemoveLastBarSupported is false, and we might be able to modify building or previous bars, we find the following:

sum = priorSum + Input[0] - (CurrentBar >= Period ? Input[Period] : 0);
Value[0] = sum / (CurrentBar < Period ? CurrentBar + 1 : Period);

In other words, when our old bars can be trusted never to change, we calculate everything on the spot with bar data that remains reliable intra-bar, and no information needs to be saved between bars inside the sum or priorSum variables. When bars themselves can change as we're building a new bar however, we store data externally in the SMA indicator itself between bars in sum and priorSum. We then reset this data every new bar with the following:

if (IsFirstTickOfBar)
priorSum = sum;

To sum up everything simply :

• When IsRemoveLastBarSupported == true, intra-bar data can be modified
• When it's false, it can't be, and is instead "write once, read forever" like a burned CD

Please let me know if you have any questions.

------------------------------
I tested this with a print statement and it seems this is the opposite. I tested this with the NinjaTrader 8 default time based chart and BarsArray[0].BarsType.IsRemoveLastBarSupported is false.
So the else code is executed:
sum = priorSum + Input[0] - (CurrentBar >= Period ? Input[Period] : 0);
Value[0] = sum / (CurrentBar < Period ? CurrentBar + 1 : Period);[/CODE]

Am I correct? I just want to understand this and know what is correct.
Thank you.

[Harry]

------------------------------
I tested this with a print statement and it seems this is the opposite. I tested this with the NinjaTrader 8 default time based chart and BarsArray[0].BarsType.IsRemoveLastBarSupported is false.
So the else code is executed:
sum = priorSum + Input[0] - (CurrentBar >= Period ? Input[Period] : 0);
Value[0] = sum / (CurrentBar < Period ? CurrentBar + 1 : Period);[/CODE]

Am I correct? I just want to understand this and know what is correct.
Thank you.

NinjaTrader comes with some bar types - for instance Renko or Linebreak bars - that have a property called IsRemoveLastBarSupported which is set to true. Let us call them irregular bars as opposed to regular bars such as minute, tick and volume bars. The difference between the two bar types is the way they are processed by OnBarUpdate() when a NinjaScript is set to Calculate.OnEachTick or Calculate.OnPrice change.

(1) Regular bars: The ticks are orderly processed one by one. A new bar starts when the prior bar is complete.
(2) Irregular bars: The ticks are not orderly processed one by one. When the first tick of a new bar has been processed, OnBarUpdate() jumps back to the prior bar to remove it and reprocess it.

Therefore when coding an indicator that shall also run on irregular bars such as renko bars, you need to pay attention to two difficulties

(A) recursive formula using prior values
(B) correct synchronization of multi-bar series scripts

(A) Example: SMA (NinjaTrader inbuilt indicator)

The code shown in the lower bracket

 if (IsFirstTickOfBar)
priorSum = sum;

sum = priorSum + Input[0] - (CurrentBar >= Period ? Input[Period];
Value[0]= sum / (CurrentBar < Period ? CurrentBar + 1 : Period);​

is simpler and slightly faster to execute. However, this code will not work with irregular bar types.

Why?

The reaseon is that when the first tick of the bar N+1 is processed, the final value of sum for the bar N is assigned to priorsum (line 76 of the indicator).
Right afterwards the bar N is processed again. The newly assigned value for priorsum is now used to calculate the value for sum (line 78 of the indicator).
This is bad, because for calculating sum for the bar N priorsum derived from bar N-1 and not priorsum derived from bar N should be used.

This results in false calculations.

To adapt the code for irregular bars, priorsum has to be replaced with a variable referring to the bar index, such as Value[1] which is used in the alternative code.

(B) Synchronization of multi-bar series scripts

In my experience it is not possible to perform a correct synchronization of a multi-bar series script that uses Renko or LineBreak bars as a secondary bar series, In case you really need such an exotic setup, you should use alternative Renko bars that have the property RemoveLastBarSupported() set to false.

[HalTech]

Thank you Harry for the explanation. I appreciate your time.

I am not trying to use alternative bar types (Renko or LineBreak, etc) I am just trying to understand the logic in the SMA code.

So, to be sure that I understand what is happening.

The IsRemoveLastBarSupported is read only and the default value is false. See - https://ninjatrader.com/support/help...rsupported.htm

The following is in reference to the built-in SMA indicator.
So in the SMA OnBarUpdate(), the else logic (sum = priorSum + Input[0]....) will be executed unless the IsRemoveLastBarSupported value is changed to true.

You stated "In case you really need such an exotic setup, you should use alternative Renko bars that have the property RemoveLastBarSupported() set to false."
I don't understand. IsRemoveLastBarSupported is already false so the default SMA code will be executed.
Is this what should happen?

[Harry]

RemoveLastBarSupported() depends on the bar type. It can be set to true or false. For instance, the property is set to false for minute, tick and volume bars. However, it is set to true for the in-built Renko and LineBreak bars.

The code of the SMA will be processed accordingly.

[HalTech]

Okay that makes sense.
Thank you for the information.
I appreciate your time and help.