NT7 and NT8 Optimizers offer different results

[NinjaTrader_Jesse]

Hello guillembm,

If the file is too large it may be too large for email as well, you can try sending it to platform support [at] ninjatrader.com, please include a link to this post or id 3074604. You would otherwise need to use a file sharing service to upload the file and generate a link. You can use any service you already have that creates public links such as google drive or onedrive.


Please let me know if I may be of further assistance.

[NinjaTrader_Jesse]

Hello guillembm,

Thank you for providing that data. I was still able to see the differences here.

Some notes on the tests you are doing would be that the SMA indicator is used in the RSI which causes a difference due to the precision of the value returned. If we break the sample down further to just the standard SampleMACrossOver which uses only the SMA we can still see differences in fills due to the precision. Another note is that the SMA has optimizations in NT8 for the changes in NT8 where the NT7 indicator is more simple. If we take it a step further and use the exact code from NT7 and NT8 we can still see differences due to the precision of the values in NT7.

A simple output example would be the following:

NT8:
4/29/2021 2:50:00 PM SMA(Fast) 4199.6 SMA(Slow)[0] 4200.19
NT7:
4/29/2021 2:50:00 PM SMA(Fast) 4199.6 SMA(Slow)[0] 4200.19000000006

The trailing decimal is truncated in NT8 where it is not in NT7, the NT8 value is more accurate as we see the truncated values truncated down to the 2 places for the instrument being used.

This was tested on 1 minute bars, greater timeframes have less occurrences where we see the calculation differ.

For this specific indicator we would not expect exactly identical results because the Cross conditions will only evaluate true when the cross actually occurs, we can see different fills based on the different placement of the crossing values. While we can get fairly similar results it would still be slightly different for this test and the RSI test as the RSI uses the SMA in its internal calculations. In the sample you provided and the output we can see greater differences based on the end result calculation in the RSI.

Without making specific modifications to account for the differences in values we wouldn't be able to get identical results with the given test. With other indicators or other conditions you may be able to still get identical results so long as all values match identically.

What this test shows is that for fill placement related questions you would need to drill down to the root of what is being done. Making sure all values being used are exactly identical with all decimal places also being identical. If any decimal place or any value is different we will inherently see different results in the analysis of the conditions.



Please let me know if I may be of further assistance.

[guillembm]

Thank you Jesse,

I understand that there have been improvements/changes in NT8 but this still does not explain my two main questions:

1. Why, if these differences are so small (maybe I could even say irrelevant), the results of the optimizations are so big? It is hard to understand why a difference in decimals in the SMA like the one you show can result in discrepancies like the ones I get in the optimization outputs.
2. Why, when using the same parameters obtained in NT7 in a NT8 backtest I get the same results with irrelevant differences as the example I attach?

Following the logic of your arguments, why when using the parameters obtained in NT7 in NT8 differences are so small in a backtest and so high in an optimization?

[NinjaTrader_Jesse]

Hello guillembm,

Why, if these differences are so small (maybe I could even say irrelevant), the results of the optimizations are so big? It is hard to understand why a difference in decimals in the SMA like the one you show can result in discrepancies like the ones I get in the optimization outputs.

Its not irrelevant, if there is any difference in the value of the indicator that will cause other items like cross conditions using that indicator to be affected. If the cross happens on a different bar that may cause large differences in the results.

Why, when using the same parameters obtained in NT7 in a NT8 backtest I get the same results with irrelevant differences as the example I attach?

The indicators being used produce different values when using the same parameters due to the precision (see my last post and the values), while the amount of the difference is small the value is different so it would not be the same exact result.


Please let me know if I may be of further assistance.

[guillembm]

Dear Jesse,

Thanks for your rely but I still believe that these arguments do not explain the enormous differences between NT7 and NT8 optimizer results.

I have been experimenting with both optimizers and I see that, for now, it is unfruitful to continue discussing about this for more time. But, it is an evidence that the results I obtain from NT7 optimizations are better, more complete and it works very well. It is also clear that when I execute a NT8 backtest with the parameters I get from a NT7 optimizer I see the same outcome and this also happens when using market replay, even if I use averages.

So, for me, it will be more suitable to optimize with NT7 and then use these parameters in NT8.

It is a pity that the amazing job that you have been doing developing NT8 gets darkened for this issue. I will continue checking NT8 optimizer often so see if there has been any improvement.

Best regards