Strategy Analyzer Innaccuracy

[samish18]

The difference between running a strategy live and backtesting over the same period is astounding. I am not talking about getting a different entry/exit price or a little bit of slippage, but completely different trades. The strategy running live might takes trades at 10 am, 11 am and 1pm while backtesting of the same period it would trade at 11:30 am only. How can I backtest more accurately?

[NinjaTrader_BrandonH]

Hello samish18,

Thanks for your post.

Please review the help guide document on the differences on real-time vs backtest (historical).

https://ninjatrader.com/support/helpGuides/nt8/discrepancies_real-time_vs_bac.htm

A strategy running real-time (live brokerage account, live market simulation, Playback connection etc...) will produce different results than the performance results generated during a backtest.

When in historical data, only the Open, High, Low, and Close will be available and there will be no intra-bar data. This means actions cannot happen intra-bar, fills cannot happen intra-bar. All prices and actions come from and occur when the bar closes as this is all the information that is known.

Because of this, OnBarUpdate will only update 'On bar close' as it does not have the intra-bar information necessary for 'On price change' or 'On each tick' and the script will not have the intra-bar information to accurately fill an order at the exact price and time.

Below is a link to the help guide on Calculate.

https://ninjatrader.com/support/help.../calculate.htm

To improve the accuracy of a backtest, you may use Tick Replay along with an added 1-tick series to have logic processed intra-bar and have orders filled intrabar.

Tick Replay would be used to have the logic process OnEachTick or OnPriceChange with historical data, but this does not allow for intra-bar order fills. You would need to add a single tick data series and submit orders to that single tick data series for a strategy that uses Tick Replay.

High Order Fill Resolution allows for intra-bar order fills with historical processing, but is not compatible with Tick Replay.

Please reference the SampleIntrabarBacktest example and the following Help Guide links for more information.

SampleIntrabarBacktest 'Backtesting NinjaScript Strategies with an intrabar granularity' - https://ninjatrader.com/support/help...ipt_strate.htm

TickReplay — https://ninjatrader.com/support/help...ick_replay.htm

Developing for Tick Replay - https://ninjatrader.com/support/helpGuides/nt8/developing_for__tick_replay.htm?zoom_highlightsub= developing+for+tick+replay

Additional information may be found in this NinjaTrader Forum post — https://ninjatrader.com/support/foru...mance?t=102504

[samish18]

If the strategy is intended to execute on bar close, why would intrabar granularity differentiate backtests and live?

[NinjaTrader_BrandonH]

Hello samish18,

Thanks for your notes.

If your strategy is set to use Calculate.OnBarClose, you would have to make sure that the same exact data is being used between the real-time results being compared and the backtest results being compared.

This means the starting bar and the ending bar of the backtest must match the starting and ending bar you are comparing when running the strategy live for the backtest results to be similar to the realtime results.

If there is a difference in data between the tests then the strategy results could be different.

From the Discrepancies: Realtime vs Backtest help guide page:

"Differences in chart data

•If you run a strategy in real-time on DAY1 and then DAY2, you are now backtesting your strategy on DAY1 data instead of processing like it did in real-time so there could be differences. You should understand how chart bars are built.

•If using tick based charts, all it takes is a single tick difference between real-time and historical data to generate completely different looking charts. This in turn would impact the calculations of your strategy should the data sets be different."

Getting Filled on an Order

"Fills are determined based on 4 data points, OHLC of a bar since that is the only information that is known during a backtest.

•During simulation using real-time live market data or Playback, the fill algorithm is dynamic in that it uses incoming market data (both price and volume) to determine if an order should be filled or not.

•During real-time live brokerage trading, orders are filled according to market dynamics."


Further, I would like to recommend using high order fill resolution with a one-tick data series to place trades on when running the backtest to improve results. You can learn more about fill resolution on the help guide link below.

Historical Fill Processing https://ninjatrader.com/support/help...ical_fill_.htm

[samish18]

For my understanding, in a situation where the bar size is individual ticks and the strategy is only considering the past 4 ticks (maybe the strategy goes long if 4 ticks were greater than the prior), after the first couple of ticks, wouldn't the backtester and live produce the same results?

[NinjaTrader_BrandonH]

Hello samish18,

Thanks for your notes.

If you are testing a strategy on a 4-Tick data series then only the Open, High, Low, and Close values of that 4-Tick bar would be available for the Strategy Analyzer to process. No intrabar information would be available. Orders are filled based on 4 data points, OHLC of a bar since that is the only information that is known during a backtest.

Whereas when the strategy is running real-time, intrabar data is available to process and orders are filled according to market dynamics.

Differences in data will also cause differences to occur when comparing real-time performance to backtest (historical) performance.

See this forum thread for more information about comparing real-time results to backtest results and using prints to find differences in performance for more information: https://ninjatrader.com/support/forum/forum/ninjatrader-8/strategy-development/100192-comparing-real-time-historical-and-replay-performance?t=102504

[samish18]

What if you were testing a strategy on a 1-Tick data series? Then the open, high, low and close would be equal. Additionally with regards to using the same data, what if this strategy (on 1-tick data series) only entered if the previous 4 1-tick data entries were successive increases or decreases - would the difference in data matter past the first 5 1-tick data points?

[NinjaTrader_BrandonH]

Hello samish18,

Thanks for your notes.

Yes, any difference in data could cause the results to differ when comparing backtest (historical) results to real-time results.

A 1-Tick data series would be the most granular series you could test the strategy on. The OHLC of each 1-Tick bar would be processed. You could use High order fill resolution with a 1-Tick series to improve backtest results when running the strategy with Calculate.OnBarClose.

From post # 4: "Further, I would like to recommend using high order fill resolution with a one-tick data series to place trades on when running the backtest to improve results. You can learn more about fill resolution on the help guide link below."

See this help guide page for more information about Historical Fill Processing and Order Fill Resolution: https://ninjatrader.com/support/help...ical_fill_.htm

See this forum thread for more information about comparing real-time results to backtest results and using prints to find differences in performance for more information: https://ninjatrader.com/support/forum/forum/ninjatrader-8/strategy-development/100192-comparing-real-time-historical-and-replay-performance?t=102504

[samish18]

I reran the backtest with higher order resolution and am still getting vastly different results from those running live. I will reiterate, the changes are not minor price or entry point differences, but taking a different number of total trades at very different times

[NinjaTrader_BrandonH]

Hello samish18,

Thanks for your notes.

To understand what might be causing the differences, you would need to make sure you are backtesting the script using the same exact data that you are using when running the strategy realtime and add prints to the script to understand exactly what is different between the two tests.

If you have a large number of prints occurring in the Output window, you could write the information to a file using a StreamWriter.

StreamWriter: https://ninjatrader.com/support/help...b=streamwriter

Review this forum thread detailing how to compare realtime performance to historical performance to identify differences between the two tests: https://forum.ninjatrader.com/forum/...nce#post100192

The forum thread linked above also contains a publicly available video and the test script used.

[samish18]

How can I make sure the data is exactly the same? Also where would the print statements be the most insightful (every order placed?).

[NinjaTrader_BrandonH]

Hello samish18,

Thanks for your notes.

You could ensure that the Start Date and the End Date in the Strategy Analyzer are set to the same data the strategy is being run on in the Chart window. Ensure the first bar and the last bar are the same between the Chart Display of the Strategy Analyzer and the Chart.

The Chart Display of the Strategy Analyzer window could be used to see what bars are being compared in the backtest. You could compare those bars to the bars on the Chart window to see if there are differences.

Prints should be added one line above conditions to place trades that prints out all the values used in the conditions to place trades along with a label for each value and the comparison operators being used in the conditions. You could also add a print within the conditions to place trades to see if the condition is triggered.

Below is a link to a forum post that demonstrates how to use prints to understand behavior.

https://ninjatrader.com/support/forum/forum/ninjatrader-7/strategy-development-aa/99209-date-conditions-not-working?p=791121#post791121

I recommend reviewing the publicly available demonstration video in the forum thread linked in post # 10.

[samish18]

I will run them with both with prints and get back to you. The issues I have been seeing have been with the data set, I've been using RTH on the same days for both the live and backtest. This seems to be an issue with all the strategies I've tested live: In live conditions, the strategy takes many more trades. Is this a common issue you've seen before with a piece of code missing?

[NinjaTrader_BrandonH]

Hello samish18,

Thanks for your notes.

This would have to do with how your logic is evaluating in the strategy when comparing the backtest results to the strategy results running live.

Debugging prints would need to be used to determine exactly how your logic is evaluating between backtest and realtime results.

To increase the accuracy of a backtest so that it is more accurate (within one tick) you could enable tick replay and add intrabar granularity to your strategy.

See more about intra-bar granularity on this forum thread.
https://ninjatrader.com/support/forum/forum/ninjatrader-8/strategy-development/94098-isfirsttickofbar-vs-onbarclose-for-backtest-live?p=773377#post773377

Further, live orders are filled on an exchange with a trading partner on an agreed upon price based on market dynamics. Backtest orders are not using these market dynamics. Instead, these are filled based on logical rules from processing historical data.

Please note, as a script is enabled live on a chart or on the Strategies tab of the Control Center, the historical data will be processed first before the strategy transitions to processing real-time data. As the historical data is processed, this is the same as performing a backtest and will provide the same historical results as backtesting in the Strategy Analyzer.
https://ninjatrader.com/support/foru...442#post782442

For the tests to be even similar, it is necessary to test the exact same sets of data, without mixing and matching the data types. (Meaning it would not be wise for a comparison of real-time or playback to include historical data and the days tested must start and end on the same session)

Once the basics of getting similar test environments (meaning same data set, and intra-bar granularity) have been satisfied, it becomes clearer how to spot exactly what has caused the differences in the order fills.

Switching to view the chart may show that the charts appear very very similar but there are still differences in the final performance.

Prints can be added to the script (or this information can be written to file if its a large amount of information) to further understand the behavior.

See the post below about using prints for debugging.
http://ninjatrader.com/support/forum...979#post510979