Strategies - Backtesting versus Live - Origins of Indicator Plotting Difference

[Sc1entia]

Hi folks,
This is a rather long winded enquiry, so please bear with me.

I am trying gain a fundamental understanding of why some Indicators do not plot correctly (or at all) when performing Strategy backtesting using the Strategy Analyser (e.g. the "StrategyPlot()" custom indicator which is known not to work for charts in the Strategy Analyser).

To this end, I am specifically looking to understand how the Strategy Analyser performs backtests for any given strategy for any given set of data.

I am led to understand that when performing becktests, optimisations are put in place so that indicators "OnBarUpdate" method is not called unless specifically forced to (using "Update()" to make sure that a correctly updated value is returned back to the strategy from the indicator).

To this end I set up a very simple custom strategy that calls a very simple custom indicator. The indicator just plots "Close[0]" on a line graph. It also has one exposed public variable which is updated (via "update()") each time the strategy "OnBarUpdate" method is executed. A series of print statements allow me to trace through the strategy and indicator as it is executed within the Strategy Analyser. Printed outputs consist of "Initialize Reached" and "OnBarUpdate Reached" for both Strategy and Indicator, as well as "CurrentBar", "ExposedVariable" and "Close[0]" values for the Indicator. The backtest data contents is consists of a custom dataset, whose values for this test are irrelevant, except to say that there is 1798 data bars present.

When I run the backtest, I get the following generic output (assuming that the Output Window show the Print output in the correct sequence).

Initialize Strategy - Expected
Initialize Indicator - Expected
For Each of the subsequent 1798 Data Bars:
OnBarUpdate - SATestIndicator - Expected
OnBarUpdate - SATestStrategy - Expected
Correct and consistent CurrentBar Numbers - Expected
ExposedVariable and Close[0] are showing correct values - Expected

I would have thought that would be it, but here's where it gets interesting... next is

Initialize SATestStrategy
Initialize SATestIndicator (three times)
OnBarUpdate - SATestIndicator (for CurrentBar 0 through to CurrentBar 1487 - Strategy is not includeed this time)
ExposedVariable is NOT updated.
Close[0] has correct value.

Thats It... End of output...

This last part (second loop through the data) is the part I am trying to understand, and can't seem to find it in the documentation. Could someone please let me know whether my observation below is in any way correct?

The example is a representive example., so you will see the same kind of behaviour for any dataset during a backtest. I'm just trying to understand the methodology (so that I can avoid any related pitfalls along the way for myself)

I know the initialize gets called a few times. I don't care about this and am told that it is necessary for "other" reasons. Ok.

Backtest Observation: It's almost like The first run through is where each bar is passed to the Strategy itself (which will only update/interact with any Indicators if explicitly told to via "update()". No plots are generated from this run, just strategy entries/exits etc. The second run through is where the Plot for the indicator is generated and therefore only runs through the indicator only, and it is only passed the price data (ExposedVariable is not updated in the example). So anything that is going to be plotted correctly, during the backtes needs to be generated from the price feed itself (no reliance on variables being set from the strategy). Is this observation correct? Anyone have anything further to add or any other questions relating to this topic?

Question: Why stop at Bar number 1487 out of 1798?

Thanks and regards,

Sc1entia

[NinjaTrader_Bertrand]

Sc1entia, thanks for the post and your observations on this, correct the strategy plot sample should be used on the regular charts only.

Which NinjaTrader version are you working on?

Could you please post your sample script used so we can check into here?

You're correct, the Initialize() can be called multiple times - for a method that is called once at starting up we have introduced OnStartUp() in our NT 7 release.

[Sc1entia]

Hi Bertrand,

Thanks for the prompt response.

I am using NinjaTrader version 7.00.17.

I have attached the strategy, indicator and test data that I have used. Data for the test data is 05/01/2000 (used a date that would not likely be overwritten). I have used 10sec bars in the test.

The main thing that I am after with this thread is to discover how things are designed to work within the Strategy Analyser during back-testing.

The observation I have included in my first post makes sense to me, but it does not explain why the second loop around only uses 1487 out of the 1798 data bars that should apparently be there (from the first loop). Any insights would be great.

Thanks,

Sc1entia

#region Using declarations
using System;
using System.ComponentModel;
using System.Diagnostics;
using System.Drawing;
using System.Drawing.Drawing2D;
using System.Xml.Serialization;
using NinjaTrader.Cbi;
using NinjaTrader.Data;
using NinjaTrader.Indicator;
using NinjaTrader.Gui.Chart;
using NinjaTrader.Strategy;
#endregion

#region NameSpace - Strategy
namespace NinjaTrader.Strategy { //Start of NinjaTrader.Strategy namespace 
[Description("SATestStrategy")]
public class SATestStrategy : Strategy { // Start of public class PS
#endregion
    
#region Variables
private int Temp = 0;
#endregion

#region Method - OnStartUp()
protected override void OnStartUp() { 
ClearOutputWindow(); //Clear down at the start of each run
} 
#endregion

#region Method - Initialize()    
protected override void Initialize() {
Print("SATestStrategy Initialize Method Called");
Add(SATestIndicator());
SATestIndicator().Name = "SATest";
CalculateOnBarClose = true;
}
#endregion

#region Method - OnBarUpdate()
protected override void OnBarUpdate() {
SATestIndicator().iExposedVariable = Temp;
Print("SATestStrategy OnBarUpdate Method Called - CurrentBar: " + CurrentBar + "  :    ExposedVariable Passing Value: " + Temp);
Temp = Temp + 1;
}
#endregion

#region Properties
[Description("Temp")] [Category("Parameters")]
public int iTemp {
get { return Temp; }
set { Temp = value; } }
#endregion

} // End of public class SATestStrategy
} // End of NinjaTrader.Strategy namespace 

#region Using declarations
using System;
using System.ComponentModel;
using System.Diagnostics;
using System.Drawing;
using System.Drawing.Drawing2D;
using System.Xml.Serialization;
using NinjaTrader.Cbi;
using NinjaTrader.Data;
using NinjaTrader.Gui.Chart;
#endregion

#region Indicator
namespace NinjaTrader.Indicator { //Start of Indicator NameSpace
[Description("The SATestIndicator is a 'Non-Plot' Indicator. It's exposed Value is updated from the calling Strategy Method. This indicator is of no use as a stand-alone indicator")]
public class SATestIndicator : Indicator { //Start of Public Class SATestIndicator
        
#region Variables
private double ExposedVariable = 0; //The Calling Method supplies the already calculated value which is to be plotted.
#endregion Variables

#region Initialize
/// <Initialise Summary
/// This method is used to configure the indicator and is called once before any bar data is loaded.
protected override void Initialize() {//Start of Initialize
Print("SATestIndicator Initialize Method Called ");
Add(new Plot(Color.Black, PlotStyle.Line, "SATestIndicator_Plot"));
Plots[0].Pen.Width = 3;
Overlay = true; 
}//End of Initialize
#endregion Initialize

#region OnBarUpdate
protected override void OnBarUpdate() {//Start of OnBarUpdate
Value.Set(Close[0]);
if (Close[0] > 100.00)PlotColors[0][0] = Color.Green;
if (Close[0] < 100.00)PlotColors[0][0] = Color.Red;
Print("SATestIndicator OnBarUpdate Method Called - CurrentBar: " + CurrentBar + " - ExposedVariable: " + ExposedVariable + " : Close[0]: " + Close[0]);
}// End of OnBarUpdate 
#endregion OnBarUpdate

public double iExposedVariable
{ set { ExposedVariable = value; Update();}}

}//End of Public Class SATestIndicator
}//End of Indicator NameSpace

#endregion Indicator

#region NinjaScript generated code. Neither change nor remove.
// This namespace holds all indicators and is required. Do not change it.
namespace NinjaTrader.Indicator
{
    public partial class Indicator : IndicatorBase
    {
        private SATestIndicator[] cacheSATestIndicator = null;

        private static SATestIndicator checkSATestIndicator = new SATestIndicator();

        /// <summary>
        /// The SATestIndicator is a 'Non-Plot' Indicator. It's exposed Value is updated from the calling Strategy Method. This indicator is of no use as a stand-alone indicator
        /// </summary>
        /// <returns></returns>
        public SATestIndicator SATestIndicator()
        {
            return SATestIndicator(Input);
        }

        /// <summary>
        /// The SATestIndicator is a 'Non-Plot' Indicator. It's exposed Value is updated from the calling Strategy Method. This indicator is of no use as a stand-alone indicator
        /// </summary>
        /// <returns></returns>
        public SATestIndicator SATestIndicator(Data.IDataSeries input)
        {
            if (cacheSATestIndicator != null)
                for (int idx = 0; idx < cacheSATestIndicator.Length; idx++)
                    if (cacheSATestIndicator[idx].EqualsInput(input))
                        return cacheSATestIndicator[idx];

            lock (checkSATestIndicator)
            {
                if (cacheSATestIndicator != null)
                    for (int idx = 0; idx < cacheSATestIndicator.Length; idx++)
                        if (cacheSATestIndicator[idx].EqualsInput(input))
                            return cacheSATestIndicator[idx];

                SATestIndicator indicator = new SATestIndicator();
                indicator.BarsRequired = BarsRequired;
                indicator.CalculateOnBarClose = CalculateOnBarClose;
#if NT7
                indicator.ForceMaximumBarsLookBack256 = ForceMaximumBarsLookBack256;
                indicator.MaximumBarsLookBack = MaximumBarsLookBack;
#endif
                indicator.Input = input;
                Indicators.Add(indicator);
                indicator.SetUp();

                SATestIndicator[] tmp = new SATestIndicator[cacheSATestIndicator == null ? 1 : cacheSATestIndicator.Length + 1];
                if (cacheSATestIndicator != null)
                    cacheSATestIndicator.CopyTo(tmp, 0);
                tmp[tmp.Length - 1] = indicator;
                cacheSATestIndicator = tmp;
                return indicator;
            }
        }
    }
}

// This namespace holds all market analyzer column definitions and is required. Do not change it.
namespace NinjaTrader.MarketAnalyzer
{
    public partial class Column : ColumnBase
    {
        /// <summary>
        /// The SATestIndicator is a 'Non-Plot' Indicator. It's exposed Value is updated from the calling Strategy Method. This indicator is of no use as a stand-alone indicator
        /// </summary>
        /// <returns></returns>
        [Gui.Design.WizardCondition("Indicator")]
        public Indicator.SATestIndicator SATestIndicator()
        {
            return _indicator.SATestIndicator(Input);
        }

        /// <summary>
        /// The SATestIndicator is a 'Non-Plot' Indicator. It's exposed Value is updated from the calling Strategy Method. This indicator is of no use as a stand-alone indicator
        /// </summary>
        /// <returns></returns>
        public Indicator.SATestIndicator SATestIndicator(Data.IDataSeries input)
        {
            return _indicator.SATestIndicator(input);
        }
    }
}

// This namespace holds all strategies and is required. Do not change it.
namespace NinjaTrader.Strategy
{
    public partial class Strategy : StrategyBase
    {
        /// <summary>
        /// The SATestIndicator is a 'Non-Plot' Indicator. It's exposed Value is updated from the calling Strategy Method. This indicator is of no use as a stand-alone indicator
        /// </summary>
        /// <returns></returns>
        [Gui.Design.WizardCondition("Indicator")]
        public Indicator.SATestIndicator SATestIndicator()
        {
            return _indicator.SATestIndicator(Input);
        }

        /// <summary>
        /// The SATestIndicator is a 'Non-Plot' Indicator. It's exposed Value is updated from the calling Strategy Method. This indicator is of no use as a stand-alone indicator
        /// </summary>
        /// <returns></returns>
        public Indicator.SATestIndicator SATestIndicator(Data.IDataSeries input)
        {
            if (InInitialize && input == null)
                throw new ArgumentException("You only can access an indicator with the default input/bar series from within the 'Initialize()' method");

            return _indicator.SATestIndicator(input);
        }
    }
}
#endregion 

[NinjaTrader_Austin]

Sc1entia, we will take a look into this and get back to you.

[NinjaTrader_RyanM1]

Hello Sc1entia,

Thanks for the files. In all my tests, the numbers match up except when Minimum Bars Required is set and then the difference is that value.

SATestIndicator OnBarUpdate Method Called - CurrentBar: 1798 - ExposedVariable: 1798 : Close[0]: 100


I duplicated your test on the same instrument, series and day. Let us know your results when Minimum Bars required is set to 0.

[Sc1entia]

Hi RyanM,

Both loops are iterating correctly now (I reinstalled NinjaTrader - cleanly). Thats one part of the original post cleared up...

Question: Could ye guys let me know whether the Observation in the original post is correct? Is it completely correct or are there any mistakes? I am asking because I can't seem to find any relevant information in the documentation or anywhere on the forums. This question focuses on what is supposed to happen every time a back-test runs...

[NinjaTrader_RyanM1]

Sc1entia,

OnBarUpdate() is the same event for indicators and strategies. If you call an indicator method in a strategy, then that logic is processed alongside strategy logic. You can see this in your test output where alternate lines of strategy and indicator are processed.

[Sc1entia]

Sc1entia,

OnBarUpdate() is the same event for indicators and strategies. If you call an indicator method in a strategy, then that logic is processed alongside strategy logic. You can see this in your test output where alternate lines of strategy and indicator are processed.

RyanM,

Ok, I understand that. What about the second "loop" in the back-test where only the OnBarUpdate method for the Indicator only is iterated through for each bar. What is going on there or why does that happen?

Thanks,

Sc1entia

[NinjaTrader_RyanM1]

I'm not familiar with this second loop or where to see it. What do I need to do in NinjaTrader to duplicate this test? So far I've been running the attached strategy in the strategy analyzer and then viewing the output.

[Sc1entia]

If you look at the output window, you will see that there are two output loops. In one of the loops OnBarUpdate for BOTH the strategy and the indicator is called (for 1798 data bars), while the second loop OnBarUpdate for ONLY the indicator is called (for 1798 data bars). Do you see this in your output? If so, then why are there two "loops", and why is there a second loop where ONLY the indicator gets called? (Call them what you want - to me they are loops)...

[NinjaTrader_RyanM1]

I see what you're looking at now. Your output for the second loop is from looking at a chart within the strategy analyzer. The StrategyPlot is a placeholder for the logic that is processed through the indicator. Neither your custom indicator method or your strategy are called during the plot.