Different backtest results after NinjaTrader upgrade (Trailing Stop bug)

[John_Paodo]

I recently upgraded from 8.0.21.1 64-bit to 8.0.24.3 64-bit

I have a Strategy build from the Strategy Builder (no manual coding).

The backtest Results of this Strategy before and after the upgrade are different:

The entry are matching 100% of the time (Both Time & Price execution)

I am using Trailing Stop (in %) to Exit.

The exits are sometimes the same, sometimes they are different.

In the environment after upgrade, sometimes the Trade exits directly after the entry. Whereas in the environment before upgrade, the trade keeps open.

EVERYTHING is the same between both environments except the NT upgrade (I am using Virtual Machines -> I made a copy of the non upgraded environment on which I ONLY applied the NT upgrade to validate my hypothesis).

My hypothesis, after hours of research, is the following (99,9% sure): One of the upgrades between 8.0.21.1 and 8.0.24.3 introduced the following bug:

The trailing stop value is calculated from the High of the bar EVEN on the entry bar in High Order Fill Resolution. This is not correct since I can enter during the bar, before the High occurs (I use limit orders placed the bar before the entry and Order Fill Resolution is High). Therefore, if the Entry Price < High * (1 - % Trailing Stop) the trade is immediately closed after it is entered (because my Entry < Stop Loss value from the start of the Trade).

In attachment a Simple Strategy to reproduce the bug. You can run it on whatever instrument. When Entry Time = Exit Time, you have found an occurence of the bug (highlighted in my results).

[NinjaTrader_JustinD]

Hello John_Paodo,

Thank you for your message.


Let's ensure that the Historical Data on both platforms are the same by doing the following steps (on both computers/platforms):

• Disconnect from your Data Provider
• Go to the Control Center > Tools > Historical Data > Edit window
• Right-click on and 'Remove' all Historical Data for any instruments/expiries you are backtesting these strategies with
• Restart NinjaTrader 8

When you restart NinjaTrader, reconnect to your Data Provider and run the backtest again on both platforms. You may wish to manually download Historical Data from the Control Center > Tools > Historical Data > Load window while connected to your Data Provider before doing this on both platforms as well.


Please let me know if the results do not remain the same after completing the steps above.

[John_Paodo]

Hello Justin,

I did what you advice but the behavior is exactly the same.

I aslo checked under Control Center > Tools > Historical Data and the data is exactly the same (See printscreens).

Can you please investigate the bug with the Trail Stop in backtesting mode described in my post ? You will be able to reproduce it on whatever data you want.

I don’t think the change in results before/after the upgrade is coming from the historical data. I am 99% sure is coming from the bug around Trailing Stop, probably introduced in one of the last upgrades.

Thanks

[NinjaTrader_ChrisL]

Hi John, thanks for posting.

There are no reported issues with the SetTrailStop method. I do not see what this test script would do to prove there is something wrong with SetTrailStop. The strategy posted submits an entry order far above the market and the trailing stop is not targeting this entry with a signal name. I attached a script that uses SetTrailStop and works fine on my end, note the use of the signal name and From Entry Signal. Note: the trailing behavior is based on the Calculate property of the script. So when the script is Calculate.OnBarClose, the stop will only be updated at the close of every bar.

Kind regards,
-ChrisL

[John_Paodo]

Hello Chris,

Thank for your reply.

The following post in the forum is spotting the same issue with Trailing Stop: https://ninjatrader.com/support/foru...g-settrailstop

« One odd behavior I have seen is that although the trailing stop adjusts tick-by-tick, as you would expect, it will sometimes immediately adjust relative to the extreme of the bar even though that is in the past. So following your example, you enter at 100 but the high of the current bar is 104; the stop immediately adjusts to 100 and you are right back out. »

I adapted your strategy (Trailing Stop = 0.001 instead of 0) and could reproduce the issue.

Please note I am using High Order Fill Resolution.

I will take a concrete example to highlight the issue (Printscreen 2021.09.09 - 02 - Trades)

On 13/08/2019 11:00:01 the strategy enters @1.11921
The initial trailing stop should be 1.11921 * (1 - 0.001) = 1.11809
But as you can see, the trade is immediately exited on 13/08/2019 11:00:01 @1.11921

According to me, this is because the initial trailing stop is calculated based on the high of the entry bar: 1.12098 * (1 - 0.001) = 1.11986 (See printscreen 2021.09.09 - 03 - Stop Calculation).

This does not make sense since the high of the bar is yet to come when the Strategy enters.

You can see on the orders that the initial value of the Trailing Stop is 1.11986 instead of 1.11809 (See printscreen 2021.09.09 - 04 - Orders).

Since the initial value of the Trailing Stop (1.11986) is above the Entry Price (1.11921), the trade is immediately exited.

I hope this clarifies.

Best regards

[NinjaTrader_ChrisL]

Hi John, thanks for your reply.

Can you confirm that you have 1 tick data available for the whole month of August 2019? The high-resolution series of 1 second must be built with tick data and most data providers will not give tick data that far back on a historical data request. You can check for the tick data through Tools>Historical data>Edit tab.

Kind regards.

[John_Paodo]

Hello Chris,

I confirm: I have tick data for August 2019 (See Printscreen 2021.09.09 - 05 - Tick data).

I checked the ticks around the Entry in my example (13/08/2019 11:00:01)

The entry is @1.11921 and none of the ticks following the entry is going as low as 1.11809 to trigger the stop loss (Printscreen 2021.09.09 - 06 - Entry Price Tick Data)

The issue is not coming from the data but from the way the initial stop is calculated (it is calculated based on the high of the entry bar instead of the entry price).

On the printscreen 2021.09.09 - 04 - Orders you can see that my theory seems correct: the price of the original stop is 1.11986 (High of the entry bar: 1.12098 * (1 - 0.001) = 1.11986)

Best regards

[NinjaTrader_ChrisL]

Hi John, thanks for your reply..

The strategy analyzer is filling these orders on the average entry because in real time it would have been a rejected order due to placing the stop too close to the market price. It is the same scenario that Paul pointed out in the link you referenced. Increasing the trail distance would alleviate this. If you needed the stop to be close to the market price the strategy must handle the rejection scenario with OnExecutionUpdate/OnOrderUpdate to capture the order rejection and re-submit the order at a valid price level. We have an example use case here:

https://ninjatrader.com/support/help...and_onexec.htm - OnExecutionUpdate/OnOrderUpdate examlpe
https://ninjatrader.com/support/help...orhandling.htm - RealTimeErrorHandling documentation.

Kind regards,
-ChrisL

[John_Paodo]

Hello Chris,

I updated the strategy: Calculate = On each Tick

I also changed: Oder Fill Resolution = 1 Tick

The issue remains the same, see Printscreen 2021.09.10 - 02 - Trades - Calculate on each Tick

My point is not about the update of the Trailing Stop but about the very first value of it.

Can we go back to the real trading life for a minute (Without using NinjaTrader).

Let’s imagine we enter EURUSD @1,11921 with a Trailing Stop 0,1%

Do you agree that the value of the very first value of Trailing Stop at the Entry is 1,11809 (= 1,11921 * (1 - 0,001)) ?

[NinjaTrader_ChrisL]

Hi John, thanks for your reply.

The strategy analyzer will only run scripts OnBarClose, even when Calculate is set to OnEachTick. The stop is simulated being adjusted and the order, in real time, would have been rejected because there is a drastic price jump between the bars. If you increase the stop distance from 0.001 to something line 0.01 it will reduce this issue in historical mode. You can see it happening in real time if you run your strategy through the Playback connection.

Kind regards,
-ChrisL

[John_Paodo]

Chris,

How do you explain that running the exact same Strategy on the exact same Data gives different results between NinjaTrader 8.0.21.1 64-bit and NinjaTrader 8.0.24.3 64-bit.

Let's take another example: 13/08/2019 15:00:00

NT 8.0.21.1 64-bit: The trade is working fine and exits 36 Minutes later (Printscreen 2021.09.10 - 03 - Trades - NinjaTrader 8.0.21.1 64-bit)
NT 8.0.24.3 64-bit: The trade is kicked out directly (Printscreen 2021.09.10 - 04 - Trades - NinjaTrader 8.0.24.3 64-bit)

[NinjaTrader_ChrisL]

Hi John,

I did some more research into the previous version and there was a change to SetTrailStop where it was being influenced by an added data series. See 14577 here Using High Order Fill Resolution falls under this category so this would explain what we are seeing here. In the previous version SetTrailStop was using added data series to change price which is not how it will work in real time practice:

Help Guide Note:
Should you have multiple Bars objects of the same instrument while using SetTrailStop() in your strategy, you should only submit orders for this instrument to the first Bars context of that instrument. This is to ensure your order logic is processed correctly and any necessary order amendments are done properly.

What you were seeing is a lucky side effect of the actual bug where SetTrailStop was being influenced by the high resolution series. The fix we applied ensures the trailing stop is only using the primary data series on the chart/analyzer.

[John_Paodo]

Thanks Chris,

Can you give me access to the installer of 8.0.23.0 and 8.0.22.2 so we can make sure the behavior changed in 8.0.23.0 ?

Thanks

[NinjaTrader_ChrisL]

Hi John,

I tested R22, 23, and the continuous version and the High Order Fill Resolution stops affecting SetTrailStop in R23. I am not able to post the installer publicly, I will private message it to you if the forum will allow me.

Kind regards.

[John_Paodo]

Following Chris’s advice, I increased the Trailing Stop to 1% and searched for an extreme trade with the same issue.

(Note: I hardcoded in the Strategy the timestamp of the trade to make it more visible on the chart (by removing the Trades before and after it)).

On 11/07/2013 00:00:00 a Trade is entered @1.30223 but it is directly exited.

On the 60 Minutes & 1 Minute chart I plotted the Entry Price (1.30223) and what a 1% trailing stop represents.

You will notice that there is no reason for the Trade to be exited at the same moment it enters (the trailing stop is way below the entry price).

Can you please have a closer look at the example and maybe share it with your colleagues. I am convinced there is something wrong around Trailing Stop in Backtesting.

Thanks in advance