Huge discrepancies in backtests

[samish18]

Investigating the indicator, it appears the in transferring the double value calculated leads to a rounding error - when calculating it directly in the script vs in the indicator I get slightly different values, what appears to be an error in store the long decimal - does this make sense to you?

[NinjaTrader_Jesse]

Hello samish18,

You could try rounding the number to a ticksize.

https://ninjatrader.com/support/helpGuides/nt8/roundtoticksize.htm

[samish18]

Would rounding explain the discrepancy between live and backtest though - its clear that backtest isn't the "correct value" as the double is incorrectly stored, but neither is the live run

[samish18]

I did some more investigating and added prints to the indicator as well as the strategy. For reference, in the attached file, lines with the word "Break" are printed from the strategy and other lines are printed from the indicator. It appears the discrepancy between the indicator and strategy is that the strategy only calculates the value when BarsInProgress == 0, however the indicator calculates it for each data series. How might I avoid this problem?

[NinjaTrader_Jesse]

Hello samish18

I wouldn't be able to say if that will resolve it without trying it, did you try to round the value?

Regarding the strategy, If the indicator calculates for all series there would not be a way to prevent that from the strategy, you would have to modify the indicator in that situation.

[samish18]

SInce the indicator values from later times are printing before the current bar and that value is being relayed in backtests, does this mean the strategy is viewing "future" data?

[NinjaTrader_Jesse]

Hello samish18,

Strategies or other scripts cannot look into the future, the bar data has not been processed. If your indicator is using other instruments those will generally have different bars and timestamps so there can be variation from the primary series. The indicator will need to be processed before the strategy so it can return a value as well.

[samish18]

What is the meaning of the prints such as:
2:45 PM Indicator Print
2:45 PM Indicator Print
2:48 PM Indicator Print
2:45 PM Strategy Print?

[NinjaTrader_Jesse]

Hello samish18,

That looks correct, the indicator processed first so the strategy can read its values. The indicator you previously had shown is using two different instruments so those likely will have differences in data which could result in different times being printed depending on which instruments have had new data and which have not. The code also does not limit the indicator to process on single bars in progress, that would mean it would have 3 total updated before returning a value to the strategy, one for the primary and one for each secondary series.

[samish18]

So what is the implication of this - the value returned by indicator called by the strategy is equal to the most recent indicator value which occurs at a time that has not occurred yet

[samish18]

2:45 PM Indicator Print
2:45 PM Indicator Print
2:48 PM Indicator Print
2:45 PM Strategy Print?

Does the above prints not indicate that the 2:48 bar of the indicator is processing before the current strategy bar and thus the strategy is able to get insight into the future when backtesting?

[NinjaTrader_Jesse]

Hello samish18,

The analyzer cannot look into the future, you can use a Print to output the CurrentBar for each series to see how the data is processed from start to finish in order.

[samish18]

I believe it is using "future" values. In the above file, the prints show the indicator value used by the strategy is the most recent one from the indicator, which is timestamped later into the future. When running live and using the same prints, I do not see this phenomenon. When running live, the value used by the strategy is also the most recent one printed from the indicator, however, running live, the timestamps happen in chronological order

[NinjaTrader_Jesse]

Hello samish18,

I would suggest printing the CurrentBar index so you can see that the bars are being processed in order. Again the platform has no ability to look into the future, it is simply processing the bars in the order that they were loaded. You will see some differences in processing when comparing a multi instrument script historical and in realtime, those will have two different orders of processing, there is no specific order that the events should happen in realtime.

[samish18]

Below is an excerpt from printing the current bar. Line with "STRAT" were printed in the strategy while other lines were printed from the indicator. "STRAT" lines contain: The time, the indicator value (which you can see is the most recently printed value from the indicator even if it is in the future), the current bar and the close.

The other lines, from the indicator, contain: The time, the indicator value, the current bar, the BarsInProgress index, Closes[0][0], Closes[1][0] and Closes[2][0].

Closes[1][0] and Closes[2][0] are used to calculate the indicator. Closes[0][0] and Closes[1][0] are from the same instrument.
As you can see, Closes[0][0] remains constant across BarsInProgress. Closes[1][0] and Closes[2][0] do not. These change and in fact DO see future values that are used to calculate the indicator. That calculated value (from future values) is then passed to the strategy.



12/29/2023 3:25:00 PM STRAT 0.005354294154885 42396 17770.5
12/29/2023 3:35:00 PM 0.005354294154885 42397 0 17767.5 17771.25 95.1525
12/29/2023 3:36:00 PM 0.005355047485051 235478 1 17767.5 17768.75 95.1525
12/29/2023 3:39:00 PM 0.00535391756927838 235479 1 17767.5 17772.5 95.1525
12/29/2023 3:39:00 PM 0.00535391756927838 45435 2 17767.5 17772.5 95.1525
12/29/2023 3:42:00 PM 0.00535278813022994 235480 1 17767.5 17776.25 95.1525
12/29/2023 3:35:00 PM STRAT 0.00535278813022994 42397 17771.25
12/29/2023 3:45:00 PM 0.00535670555782303 42398 0 17763.25 17763.25 95.1525
12/29/2023 3:45:00 PM 0.00535670555782303 235481 1 17763.25 17763.25 95.1525
12/29/2023 3:48:00 PM 0.00535761039399783 235482 1 17763.25 17760.25 95.1525
12/29/2023 3:45:00 PM STRAT 0.00535761039399783 42398 17763.25
12/29/2023 3:50:00 PM 0.00535761039399783 42399 0 17761.5 17760.25 95.1525
12/29/2023 3:51:00 PM 0.00535783665080661 235483 1 17761.5 17759.5 95.1525
12/29/2023 3:51:00 PM 0.00535783665080661 45436 2 17761.5 17759.5 95.1525
12/29/2023 3:54:00 PM 0.00535708253575048 235484 1 17761.5 17762 95.1525
12/29/2023 3:50:00 PM STRAT 0.00535708253575048 42399 17760.25
12/29/2023 3:55:00 PM 0.00535708253575048 42400 0 17741.5 17762 95.1525
12/29/2023 3:57:00 PM 0.00535934551803768 235485 1 17741.5 17754.5 95.1525
12/29/2023 4:00:00 PM 0.00535700713592028 235486 1 17741.5 17762.25 95.1525
12/29/2023 4:03:00 PM 0.00535806292672626 235487 1 17741.5 17758.75 95.1525
12/29/2023 4:06:00 PM 0.00535753497930802 235488 1 17741.5 17760.5 95.1525
12/29/2023 4:09:00 PM 0.00535896822156203 235489 1 17741.5 17755.75 95.1525
12/29/2023 4:12:00 PM 0.00535896822156203 235490 1 17741.5 17755.75 95.1525
12/29/2023 4:15:00 PM 0.00535979834394187 235491 1 17741.5 17753 95.1525
12/29/2023 4:18:00 PM 0.00535987382236555 235492 1 17741.5 17752.75 95.1525
12/29/2023 4:21:00 PM 0.00536062872353911 235493 1 17741.5 17750.25 95.1525
12/29/2023 4:24:00 PM 0.00536010027039207 235494 1 17741.5 17752 95.1525
12/29/2023 4:27:00 PM 0.00536040223086023 235495 1 17741.5 17751 95.1525
12/29/2023 4:27:00 PM 0.00536040223086023 45437 2 17741.5 17751 95.1525
12/29/2023 4:30:00 PM 0.00536025124637355 235496 1 17741.5 17751.5 95.1525
12/29/2023 4:33:00 PM 0.00535949645150389 235497 1 17741.5 17754 95.1525
12/29/2023 4:33:00 PM 0.00535949645150389 45438 2 17741.5 17754 95.1525
12/29/2023 4:36:00 PM 0.00536010027039207 235498 1 17741.5 17752 95.1525
12/29/2023 4:39:00 PM 0.00536010027039207 235499 1 17741.5 17752 95.1525
12/29/2023 4:42:00 PM 0.0053601757573197 235500 1 17741.5 17751.75 95.1525
12/29/2023 4:45:00 PM 0.00535927005449246 235501 1 17741.5 17754.75 95.1525
12/29/2023 4:48:00 PM 0.0053601757573197 235502 1 17741.5 17751.75 95.1525
12/29/2023 4:51:00 PM 0.00536025124637355 235503 1 17741.5 17751.5 95.1525
12/29/2023 4:54:00 PM 0.00535964739347171 235504 1 17741.5 17753.5 95.1525
12/29/2023 4:57:00 PM 0.00536032673755369 235505 1 17741.5 17751.25 95.1525
12/29/2023 3:55:00 PM STRAT 0.00536032673755369 42400 17762