How reliable in the strategy analyzer?

[samish18]

How reliable is the strategy analyzer? If using slippage and backtesting over the same period as running live, what is the maximum difference in results?

[NinjaTrader_Jesse]

Hello samish18,

The Slippage setting is used to set the amount of slippage in ticks per execution used in performance calculations during backtests. You can use 0 slippage to calculate trades base on the price data or you can include a positive amount to simulate having slippage on executions. The value you use would be a number you need to come up with based on your actual slippage that you see when trading.

The maximum difference in results would be impossible to calculate because two tests can vary in a lot of ways. Depending on how the strategy is programmed and if it uses performance information you may see differences ranging in different execution prices to having the strategy work completely differently based on its logic.

[samish18]

I have a strategy that I tested in the strategy analyzer and like, however when I run it live, it takes a different number of trades and see about 1-2 points of slippage on ES

[NinjaTrader_Jesse]

Hello samish18,

You can see differences between historical and realtime tests for a number of reasons. You can compare the results between the tests to get a very broad overview, if you can spot the specific differences that were happening you may be able to determine what is happening different in each test. More commonly you would need to add debugging into the script so you can output how it worked for each test. That lets you compare the results and see specifically what was different. You can find more information on this topic in the following links.

https://ninjatrader.com/support/helpGuides/nt8/discrepancies_real-time_vs_bac.htm?zoom_highlightsub=discre

https://support.ninjatrader.com/s/article/Developer-Guide-Comparing-Strategy-Results?language=en_US

[samish18]

This is a strategy that runs on bars of ticks (not renko or anything else). I have loaded the same data set for both, yet am seeing it take completely different trades - rarely making the same trade. I want to get the results I saw in backtesting.

[samish18]

The strategy only uses current price action and should not deviate nearly as much as it does

[NinjaTrader_Jesse]

Hello samish18,

Tick bars are one area where you can see differences between both backtests and realtime tests. Tick bars are built based on a number of ticks, if the start date changes at all between tests the bars will build differently so you can expect different results in that case. It would be fairly difficult to get an exact match between a historical and realtime test when using tick data.

In order to get matching results you would need to have the exact same data loaded starting with the same tick as the historical test. That would let the script do its initial bactest and reach the same conclusion and then include the now realtime data. You would otherwise need to take the results from the bactest as more of a suggestion of what's possible because the realtime test could look fairly different depending on how your logic has equated on the new bars.

[samish18]

When comparing the real-time and backtest, the real time appears to make many more trades (including most of the trades shown in the backtest) as well as experiencing 2 to 4 points in slippage on NQ.

[samish18]

Would bars of size 1-tick see the same issue?

[NinjaTrader_Jesse]

Hello samish18,

Any two tests are going to differ when different data is being used when you run your conditions over each dataset. using 1 tick bars could also differ depending on your trading conditions and what tick you start on.

The only way to have an exactly identical test between two modes is to have the strategy process the same. Lets say you did backtest and want to replicate those results and test in realtime now. To do that you need to apply a strategy to a realtime chart where the exact same historical dataset that was used in the backest is loaded.If the strategy is able to once again start on the same data point it can reproduce the same results as the backtest and then enter realtime.

[samish18]

Could you elaborate on how 1 tick bars would differ? I understand how starting position would affect 100 or 300 tick bars.

[NinjaTrader_Jesse]

Hello samish18,

Here is a very simple way to think about it.

For 1 tick bars if you load 100 bars (ticks) and then do an moving average you will get a different result if you change any of the 100 ticks that you loaded or also started at a different point in time which would shift the ticks being used.

Lets say you are using a 500 Tick chart with 1 Days to load. If you select 1 Days to load we are only going to get the same tick 500 chart as long as the very first tick in the dataset is always the same as the first time we viewed it. If you try to compare this on another PC or data provider or if you started the chart at a different time previously you may get a different chart. You could also get a slightly different chart at a later time when the data provider has finalized historical data for this day and the historical data has been reloaded.

[samish18]

If a strategy were to only look at the past 6 bars in the onBarUpdate (for example: go long if the past 5 ticks were increases) wouldn't any difference in data not matter.

[NinjaTrader_Jesse]

Hello samish18,

I am not sure what you are asking, if you have a question please provide a clear question in your reply.

[samish18]

If on 1 tick bars, you were not using a moving average but simply looking at 5 consecutive increase in ticks, would the data matter past the initial 10 ticks?