Indicator Value Discrepancies in Back Test for Secondary Series

[RideMe]

Hello,

When I back test my strategy and obtain prints of indicator values, the values don't always align with values obtained from previous back tests. It does align for the primary series (renko bars with the smallest number of ticks), but in many instances for the second and/or third series (renko bars with more ticks), the indicator values are different. For example, the same trade in a second back test that occurs at the exact time, on the exact same current bar for the primary series, and with all the same indicator values on the primary series, will reflect different indicator values for one or more of the other higher series. When I review these discrepancies on the chart for the higher series, sometimes the data on the chart is consistent with the prints, and other times it is not. I can not figure out why this is occurring or how to fix it. After reading help guides I thought it may have to do with the higher series data needing to be synced, but the trades in the strategy are all supposed to occur OnBarUpdate. I understand renko bars aren't great for backtesting, but thought this had more to do with understanding fill prices rather than causing any discrepancies with indicator values.

Here is a sample of actual prints for the same trade but for two separate back tests. Per the data, the second series indicator values have changed, but the first and third series are exactly the same:

CurrentBar		Time	Close		Series1 Values					Series2 Values					Series3 Values
46296	Entry	5/3/2023 12:03:25 PM	1760.6		0.164609	0.046032	0.047407	0.028891		-0.3786	-0.49493	-0.62231	-0.62651		-0.00754	-0.01003	-0.00249	-0.0238
46296	Entry	5/3/2023 12:03:25 PM	1760.6		0.164609	0.046032	0.047407	0.028891		-0.37374	-0.44668	-0.57237	-0.56882		-0.00754	-0.01003	-0.00249	-0.0238

Any help understanding why the values might be changing and how to fix this would be appreciated.

[NinjaTrader_BrandonH]

Hello RideMe,

Thanks for your post.

Standard Renko bars can be challenging for backtesting as the barstypes use functionality to remove the last bar in the data series and replace the bar with a new Open price.This is done with reversals and cannot be simulated historically, because the bars are already built. Testing may be done with the Playback Connection to fully simulate standard Renko bars with accurate reversals and order fills.

This information could be found on the Disrepencies: Realtime vs Backtest help guide page here:

https://ninjatrader.com/support/helpGuides/nt8/?discrepancies_real-time_vs_bac.htm

Playback: https://ninjatrader.com/support/help...ion.htm

[RideMe]

Hi Brandon,

Thank you for the response. I've been looking into this a bit further, but still have some concerns so I took a look at the specific tick data to see if I could shed some light on the issue. One thing I noticed was that when I exported the tick data to a txt. file, the data did not match the data in the historical data window in ninjatrader. In fact, it was significantly different at many times. Is there a reason the exported data differs? Could this cause the backtests to be inconsistent?

I appreciate that renko bars can have there issues, but I have run the same back test multiple times with consistent results, then I'll run it one more time and absolutely everything changes. I have deleted my cache, reloaded all historical data etc., and the original results from the first 10+ times I ran the strategy are never to be seen again. And the new results are not even close; all indicator values are different, most of the trades taken are completely different. It's as though all of the data has changed. I'm hoping you can help shed some light.

[razeus]

Hi Brandon,

Thank you for the response. I've been looking into this a bit further, but still have some concerns so I took a look at the specific tick data to see if I could shed some light on the issue. One thing I noticed was that when I exported the tick data to a txt. file, the data did not match the data in the historical data window in ninjatrader. In fact, it was significantly different at many times. Is there a reason the exported data differs? Could this cause the backtests to be inconsistent?

I appreciate that renko bars can have there issues, but I have run the same back test multiple times with consistent results, then I'll run it one more time and absolutely everything changes. I have deleted my cache, reloaded all historical data etc., and the original results from the first 10+ times I ran the strategy are never to be seen again. And the new results are not even close; all indicator values are different, most of the trades taken are completely different. It's as though all of the data has changed. I'm hoping you can help shed some light.

Hi, Do you export export data through NinjaTrader or other? I will try replicate this export... How can you compare this data mismatch after exporting?

[RideMe]

Hi. Yes, I export it through Ninjatrader. It is exported into a txt file. I then compared the txt file data to the data that appears in the historical data window in the tools menu. I simply looked at different days in the historical data window (randomly picking different days and times) and then located the same date and time in the txt file and compared the prices at those times. The prices were often off by quite a bit.

[NinjaTrader_BrandonH]

Hello RideMe,

Thanks for your notes.

When testing this on my end I see that the exported Historical Tick data matches the Historical Data window.

See the attached screenshot demonstrating this.

Keep in mind that the exported Historical Tick data is in UTC time, not your local timezone.

For example, in the screenshot we compare the 3:59:59:932 PM tick data in the Historical Data window with the same bar (195959 9320000 UTC time) in the export file and see that the data matches.

More information about Tick Data Formats could be found on this help guide page:
Hello RideMe,

Thanks for your notes.

When testing this on my end I see that the exported Historical Tick data matches the Historical Data window.

See the attached screenshot demonstrating this.

Keep in mind that the exported Historical Tick data is in UTC time, not your local timezone.

For example, in the screenshot we compare the 3:59:59:932 PM tick data in the Historical Data window with the same bar (195959 9320000 UTC time) in the export file and see that the data matches.

More information about Tick Data Formats could be found on this help guide page:

https://ninjatrader.com/support/helpGuides/nt8/index.html?importing.htm#UnderstandingImportFileAndDataFormats

This help guide page details that the exported data will be in the UTC time zone:

https://ninjatrader.com/support/helpGuides/nt8/exporting.htm

[RideMe]

Hi Brandon,

Thank you. This clarifies the issue regarding the tick data, but I'm still having issues with indicator values varying for each backtest. To try to resolve, I replaced the renko bars with a different type of bar that accurately reflects the OHLC. However, when I ran the strategy multiple times with these bars, the indicator values for higher time frames continued to change. On further review, I saw the following posts, which seem to discuss the issue that I am having:

https://forum.ninjatrader.com/forum/ninjatrader-8/strategy-development/1120483-mcclellan-indicator-discrepancy-between-chart-and-strategy
https://forum.ninjatrader.com/forum/ninjatrader-8/indicator-development/105828-indicator-accessing-multiple-time-frame?_gl=1*xzt0iq*_gcl_au*MTkzMzc2MTQzLjE3MTM4MDc 1MjQ.#post820196

I did test the AddChartIndicator method in my strategy, but the values printed still did not match the values reflected on the higher time frame chart.

Do I need to use the plot method (per the "HigherTimeFrameIndicatorPlotExample")? I did test this as well, and it seemed to produce the correct values reflected on the higher time frame, but I ran into troubles when trying to plot more than one indicator from the higher time frames. Is this the only solution? Is there a simpler solution? To be clear, I do not actually need to plot the higher time frame indicators - this would be too messy. I simply need the correct indicator values from the higher time frames for the strategy. The indicator values I am looking for are those that existed at the time a prior bar closed on the higher time frames (as it would appear on the higher time frames chart).

Thank you,

[NinjaTrader_BrandonH]

Hello RideMe,

Thanks for your notes.

To clarify, are you testing the strategy using time-based bars, such as Minute bars?

When using AddChartIndicator() in a NinjaScript strategy you cannot use any secondary added data series for the indicator, only the strategy's primary data series can be used.

From the AddChartIndicator() help guide page:

"An indicator being added via AddChartIndicator() cannot use any additional data series hosted by the calling strategy, but can only use the strategy's primary data series. If you wish to use a different data series for the indicator's input, you can add the series in the indicator itself and explicitly reference it in the indicator code (please make sure though the hosting strategy has the same AddDataSeries() call included as well)

o If a secondary or null Bars series is specified by the calling strategy (not the indicator itself), the strategy's primary series will be substituted instead."

AddChartIndicator(): https://ninjatrader.com/support/help...tindicator.htm

Yes, the AddPlot() method could be used to add plots to the strategy and assign the values of the indicators calculating from a secondary series to the plots so they appear on the chart.

When you mention "I ran into troubles when trying to plot more than one indicator" what exactly is the issue you encountered?

How exactly are you defining and accessing the indicators using an added secondary series in the custom Strategy?

If possible, please provide us with an exported reduced test script that demonstrates the behavior you are reporting and the steps you are using to reproduce the behavior. Note that a reduced copy refers to a copy of the script that contains the minimum amount of code needed to reproduce the issue. All other code is commented out or removed.

To create a copy of your script to modify, open a New > NinjaScript Editor, select your script, right-click in the Editor, select 'Save as', name the script, and click OK.

We look forward to assisting further.

[RideMe]

Hi Brandon,

Thank you for the follow up. I think I've got this working now so that prints of indicator values for higher time frames are accurate. However, it only seems to be working for the most recent bar. For example, if I print the values for Values[3][6], shouldn't this provide me the higher time frame indicator's value 6 bars ago, based on a 6 bar count of the higher time frame? When I print this value, it seems to provide me the value for the higher time frame indicator, but using a 6 bar count of the primary bars. Is there a way to ensure the bar count is based on the higher time frame when referring to the higher time frame indicator value?

Thank you.

[NinjaTrader_BrandonH]

Hello RideMe,

Thanks for your notes.

Printing Values[3][6] would print out the 4th plot in the script for the value 6 bars prior to the current bar on the chart.

The first plot added to a script would have an index of 0, such as Values[0][6]. The second plot added to the script would have an index of 1, such as Values[1][6]. The third plot added to the script would have an index of 2, such as Values[2][6]. The fourth plot added to the script would have an index of 3, such as Values[3][6], and so on.

The "6" in Values[3][6] would be the barsAgo value being passed in indicating you are accessing the indicator plot 6 bars prior to the current bar on the chart.

Values: https://ninjatrader.com/support/help...nt8/values.htm
AddPlot(): https://ninjatrader.com/support/help...t8/addplot.htm

How exactly are you defining the indicator in the script and assigning it to the plot?

Are you calculating the indicator from the secondary series by passing in the BarsArray index into the indicator when instantiating it?

An example of this could be seen on this help guide page linked below.

Multi-Instrument / Multi-TimeFrame: https://ninjatrader.com/support/helpGuides/nt8/index.html?multi-time_frame__instruments.htm#UsingBarsObjectsAsInpu tToIndicatorMethods