You need to recode many system indicators

[Harry]

SMA flawed

Many of the system indicators of NinjaTrader are flawed, because they use recursive formulae. For example the SMA cannot be correctly used, because it introduces huge rounding errors. This occurs, when applying it to an indicator, as the values to which it is applied are not multiples of a ticksize.

Best way to study this is to put the SMA on a chart with a small lookback period and then increase the lookback period to increase the size of the rounding errors.

Example

To show the problem I have applied a 3-period-SMA to smooth the CurrentDayVWAP. The VWAP starts with the begin of the session. For the prior period to the session the values of the VWAP are reset. The smoothing is applied starting with the 4th bar to make sure that it works correctly.

Blue: Original VWAP
Red: Smoothed with Default System Indicator SMA
Yellow: Smoothed with Recoded SMA (standard formula)

The left chart below is a 1 min chart with a lookback period of 65 days. The right chart is the identical, but has a reduced lookback period of 5 days. You will easily notice that the larger lookback period introduces an error to the SMA, which makes it unusable.

Conclusion

I have had similar experiences with different indicators, including the Stochastics. They all display false results, when applied to other indicators in combination with smaller timeframes and longer lookback periods. My recommendation is - as far as possible - to scrap all system indicators that use recursive formulae and replace them with the standard versions.

Otherwise I will have to recode them all for my use, as I did with the SMA.

[Harry]

Possible Solution for the Problem

Have tested now various recursive formulae for SMA and other indicators. The solution to the problem is relatively easy:

The problem did not occur, when I added the line

 
MaximumBarsLookBack = MaximumBarsLookBack.Infinite

So all indicators that use recursive formulae should have this line added to their Initialize() section.

[NinjaTrader_Bertrand]

Hi Harry, thanks for reporting in your findings - correct, due the recursive nature there's a 'catch up' period involved. I'll forward your analysis so it can be added to our feedback list.

[NinjaTrader_JoshP]

Harry,

The issue is highly likely due to how you are resetting your VWAP. Can you please attach your exact VWAP implementation and then we can take a look at the exact interaction of all the potential factors together for what you are running into?

Please take a look at my attached screenshot using the VWAP indicator I have on my own end here. Red line is a 3 period SMA. Orange line is VWAP. VWAP starts freshly calculating on the first bar of a new session as can be seen by the VWAP line being equal to the close price of the first bar of the session. If you count off the first 3 bars of the session and check VWAP prices, we end up with this:

1142.75
1142.51
1142.51

A 3 period SMA on those values = 1142.59 which is exactly what the SMA value is shsown on the bar marked by the green arrow. Note the attached chart is a 65 days lookback period chart.

Remember that when you are running another indicator on an indicator that resets values, the second indicator does not reset values along side the first indicator. That means all of a sudden you end up stuffing in values that are not necessarily reflective of what is being charted anymore.

[Harry]

Hi Josh, thank your for answering.

I had coded a VWAP that only displays data during the RTH session, During the Globex session I had just reset it via CurrentSessionVWAP.Reset().

I am aware that a SMA(3) will plot nonsense during the first three bars of the session. So I only used it starting with the fourt bar, basically following the logic

 
if (SessionBar <=3)
  SmothedCurrentSessionVWAP.Set (CurrentSessionVWAP[0]);
else
  SmoothedCurrentSessionVWAP.Set (SMA(CurrentSessionVWAP,3)[0]);

Now the problem is that due to its recurrent formula the SMA digs further into the past, and the chart being large, it comes along with a carryover of rounding errors, which it happily unloads on the chart. Of course this is due to

- some prior values of the SMA series not being assigned
- the setting MaximumBarsLookBack.TwoHundredFiftySix

But in the end I am left with the problem that - although I have four bars on my chart, the indicator can not produce a correct SMA(3), which is not acceptable. Using a recursive formula for the SMA is counterintuitive and produces this unnecessary carry over.

Also I have encountered at least 20 complaints from other people, where false values have been produced by indicators due to this MaximumBarsLookBack.TwoHundredFiftySix settings, that I will probably switch all indicators to MaximumBarsLookBack.Infinite. There are so many problems induced by this that it is hundred times cheaper to buy some RAM or new CPU.

Again this problem has 2 solutions: Not to use recursive formulae where not necessary and/or changing the default settings of the indicators by adding that single line.

Harry,

The issue is highly likely due to how you are resetting your VWAP. Can you please attach your exact VWAP implementation and then we can take a look at the exact interaction of all the potential factors together for what you are running into?

Please take a look at my attached screenshot using the VWAP indicator I have on my own end here. Red line is a 3 period SMA. Orange line is VWAP. VWAP starts freshly calculating on the first bar of a new session as can be seen by the VWAP line being equal to the close price of the first bar of the session. If you count off the first 3 bars of the session and check VWAP prices, we end up with this:

1142.75
1142.51
1142.51

A 3 period SMA on those values = 1142.59 which is exactly what the SMA value is shsown on the bar marked by the green arrow. Note the attached chart is a 65 days lookback period chart.

Remember that when you are running another indicator on an indicator that resets values, the second indicator does not reset values along side the first indicator. That means all of a sudden you end up stuffing in values that are not necessarily reflective of what is being charted anymore.

[NinjaTrader_JoshP]

Harry,

256 vs infinite is a conscious choice the user needs to make. Making everything as infinite defeats all memory savings the concept was intended to create (these savings are quite significant with 256). We recognize there are for sure certain needs that will arise that requires infinite which is why it is provided as an option, but the vast majority of cases do not require it and as such 256 will remain the default for the considerable benefits to most users.

The way the SMA indicator is programmed is for, again, memory and CPU savings which are considerable. This concept of course would not make sense when data points are wiped out of the DataSeries the SMA is suppose to be running on. In my prior post please find that the SMA indicator has no issues with lookback 256 when all data points are present.

The issue here is simply trying to mix two different concept techniques which are "incompatible" together. If you want to remove data points from a series you would not want to use lookback infinite, or no recursive logic.

Why does lookback infinite work in this scenario?
-This works because of the impact of removing old bars on a lookback of 256. A lookback of 256 is a rolling lookback essentially, which means you are pretty much only running the indicator on the latest 256 bars. When you wipe out data in bars, every time the indicator rolls forward a bar, it all of a sudden is running calculations on empty or placement holder values which results in the SMA values you see. When you use infinite you are not rolling your calculations and as such despite wipping out old values the calculations are persisted and maintained for every single bar thus leaving you with accurate values to base your latest calculations on at the expense of considerable memory being used to store infinite indicator history.

Why does no recursive logic work in this scenario?
-This works because instead of relying on old values you simply take the last 3 bars and run an average on them. This means you are using a loop on every single bar. This works, but at a considerable CPU hit when you start doing things like SMA periods of 20 or 50. That is 20 or 50 nearly redundant calculations being done every single bar. You switch to CalculateOnBarClose = false and all of a sudden you have hundreds of thousands of redundant calculations taking considerable CPU hit for something as simple as a Moving Average. This is why the recursive logic is used as these redundant calculations will result in a huge performance hit. The recursive logic effectively reduces those hundreds of thousands of calculations down to just one per incoming tick. That is to say, if a bar had 2000 ticks, running a 50 period SMA on a recursive logic = 2000 calculations while running it on a straight loop average logic = 100,000 calculations.

So to sum things up, basically you are running into performance concepts that unfortunately cannot be used together with your particular scenario. The performance concepts will remain intact because the pros vastly outweigh the con. What that means for your particular case is you have 3 options.

1. Do not remove old data points so lookback 256 can work properly (or at a minimum, do not wipe data within that latest 256 range)
2. Remove old data points, use lookback infinite and take the memory hit
3. Remove old data points, use non-recursive logic and take the CPU hit

[Harry]

Thank you Josh for your detailed answer. Absolutely agree with you.

Just a note here:

NT users - not only me - are being trapped by the concept, because it is too complicated and counterintuitive. I have lots of indicators, which depending on supplementary criteria do not affect a precise value to every single bar of the DataSeries. In this case I use the Reset() instruction.

Your answer means that within all strategies or indicators that leave gaps in the DataSeries, I cannot call other indicators without verifying whether they use a recursive formula. NinjaTrader will not alert me to this, but simply produce false values. It is not easy to always pay attention to comply with the three options you mentioned at the end of your recommendations.

[NinjaTrader_JoshP]

Harry,

To check for placeholder values you would want to use .ContainsValue() on the DataSeries object. That will tell you whether or not the value has been set or not.