NT7 and NT8 Optimizers offer different results

[guillembm]

Hello,

I am still quite astonished to see that Nt7 and NT8 optimizers provide such different results. I insist on testing this because obtaining good parameters is the basis of a strategy that works.

I attach the strategy I created in a NT7 and NT8 format (CompareOptimizerNT7.cs, CompareOptimizerNT8.cs). This is a simple and useless strategy with the only aim to compare results intending to fix this.

I am also aware of the randomness effect of using GO optimizer. To avoid this I repeated many times the execution of the optimizer in both versions obtaining the same results in every execution in NT7 and also in NT8 but still different between both versions (NT7 Results GO Summary.JPG, NT7 Results GO Trades.JPG, NT8 Results GO Summary.JPG, NT8 Results GO Trades.JPG).

I also did the same using the default optimizer that should also offer the same results, but the outcome is still different (NT7 Results Default Optimizer.JPG, NT8 Results Default Optimizer.JPG).

To make sure that I did not set the wrong settings in the panel I took the parameters obtained in NT7 and copied them in the NT8 settings panel. As you can see in this case, results are identical with an irrelevant difference in profits (NT7 Results in NT8 1.JPG, NT7 Results in NT8 2.JPG).

After this, I can conclude that there are still bugs to fix in the NT8 optimizer. I am also surprised that in NT7 I always get better results than in NT8.

My best regards

[NinjaTrader_Jesse]

Hello guillembm,

I see that you had opened a thread about this previously, please ensure to reply back to your existing threads if you have follow up questions on the same topic.

To continue here, were the tests that you ran identical in regard to what data was loaded and processed? I don't see that you have any prints in the script to confirm they had identical data during the test or other items like indicator values. You could add prints to explore this further to gain a better understanding of the differences that you are seeing in the results.

It would not be considered a bug if you are just seeing different results in the analyzer between platforms, that is likely caused by a difference between tests or how the two different platforms execute the tests. You can use prints to dig deeper into the tests if you need to do that type of comparison between platforms.


Please let me know if I may be of further assistance.

[guillembm]

The confirmation that the data is identical is already in my first post. I attach the screenshots again.

As you can see, if I optimize the strategy in NT7 and I copy the parameters obtained to a backtest in NT8 I get the same outcome. This is proof that the data is identical. Data used in NT8 was imported from NT7 after a clean install.

[NinjaTrader_Jesse]

Hello guillembm,

Your images only show the same settings, that is not a confirmation that everything actually was identical. To confirm a script is executing the same you need to use prints in the code.


Please let me know if I may be of further assistance.

[guillembm]

I do not follow your logic. If I optimize a strategy in NT7, I take the resulting parameters and execute a backtest in NT8 with the same parameters and I get the same outcome I would think that the used data is the same. If you realize in the trades screenshot in both versions results are exactly the same.


But I have no problem in printing results in both versions and show them to you. What data would be suitable to print to check this issue?

[NinjaTrader_Jesse]

Hello guillembm,

From the image I can only see that you made the settings the same, that does not tell me everything was actually the same. To know that you need to use prints to address whats happening between the tests to further identify what was different.

To make sure the data is the same you could start by printing the Time[0] object and then checking that both tests had the exact same bars in the test. The next step would be to compare the indicator values in case all the data was identical.

You may want to try adding a day to the amount of data in NT8 loads and also uncheck exit on session close. NT7 and NT8 perform differently in the analyzer so you may need to use prints to find the specific reason the results are different.

Please let me know if I may be of further assistance..

[guillembm]

Dear Jesse,

I printed the data in two files that I attach. As you can see, the results are identical.

Please, try it on your end.


The code of the prints in NT7 is:

if (CrossBelow(RSI(NumBars, SmoothValue), UpperLimit, 1))
{
EnterShort(DefaultQuantity, "");
TradeCounter = TradeCounter +1;
Print("NT7 EnterShort Trade Number: " + TradeCounter + " Date: " + ToDay(Time[0]) + " Time: " + ToTime(Time[0]) + " Close Price: " + Close[0] + " RSI Value: " + RSI(NumBars, SmoothValue)[0] );
}
else if (CrossAbove(RSI(NumBars, SmoothValue), LowerLimit, 1))
{
EnterLong(DefaultQuantity, "");
TradeCounter = TradeCounter +1;
Print("NT7 EnterLong Trade Number: " + TradeCounter + " Date: " + ToDay(Time[0]) + " Time: " + ToTime(Time[0]) + " Close Price: " + Close[0] + " RSI Value: " + RSI(NumBars, SmoothValue)[0] );
}


In NT8 is:

if (CrossBelow(RSI(NumBars, SmoothValue), UpperLimit, 1))
{
EnterShort(DefaultQuantity, "");
TradeCounter = TradeCounter +1;
Print("NT8 EnterShort Trade Number: " + TradeCounter + " Date: " + ToDay(Time[0]) + " Time: " + ToTime(Time[0]) + " Close Price: " + Close[0] + " RSI Value: " + RSI(NumBars, SmoothValue)[0] );

}
else if (CrossAbove(RSI(NumBars, SmoothValue), LowerLimit, 1))
{
EnterLong(DefaultQuantity, "");
TradeCounter = TradeCounter +1;
Print("NT8 EnterLong Trade Number: " + TradeCounter + " Date: " + ToDay(Time[0]) + " Time: " + ToTime(Time[0]) + " Close Price: " + Close[0] + " RSI Value: " + RSI(NumBars, SmoothValue)[0] );
}

[NinjaTrader_Jesse]

Hello guillembm,

From the files you provided the values are different so we should expect different results based on that output. You can use a DIFF tool to compare your files to see that. One example is the following:

NT7 EnterLong Trade Number: 87 Date: 20210413 Time: 203000 Close Price: 253.11 RSI Value: 54.2432516561181

NT8 EnterLong Trade Number: 87 Date: 20210413 Time: 203000 Close Price: 253.11 RSI Value: 54.3051581012081

these are different indicator values so we should not expect identical results.

In general you will see some differences between the two platforms so it would be unlikely for them to exactly match. You can try to get the results fairly similar as you have here but the differences in values with either the indicators or precision of values the indicators report will cause differences in results.


Please let me know if I may be of further assistance.

[guillembm]

You are right. When checking the print results I didn't check the entire file.

There is an issue with imported data. Since November 25th 2020 data does not match with the original. And it happens with all instruments. I wonder if there is an issue when importing data from NT7.

Knowing this I started the tests from scratch. First, I printed the data using bars previous to November 25th 2020 in both versions. I checked in an excel spreadsheet and they match exactly.
Then, I repeated the optimizer tests and the output is different again in both versions. To avoid the randomness of the GO optimizer I used the Default one.

I attach the results.

[NinjaTrader_Jesse]

Hello guillembm,

There is an issue with imported data. Since November 25th 2020 data does not match with the original. And it happens with all instruments. I wonder if there is an issue when importing data from NT7.

I had not observed that problem when I tried importing data myself, have you tried to clear the data and cache then re import it again? Have you also tried to re export the data before doing that process?

Knowing this I started the tests from scratch. First, I printed the data using bars previous to November 25th 2020 in both versions. I checked in an excel spreadsheet and they match exactly.
Then, I repeated the optimizer tests and the output is different again in both versions. To avoid the randomness of the GO optimizer I used the Default one.

I am not quite sure I understand what you are saying here in contrast to the data you provided. The two files are only showing when your conditions became true as far as I can tell and the two files are different. I would also suggest sticking with backtests to help clarify the difference as

If there are any differences in the data or indicator values you should see that the conditions become true at different times leading to different fills and results.

Please let me know if I may be of further assistance.

[guillembm]

Unfortunately, I run twice the backtest. This is why the results appear duplicated. But the results of the prints without considering this issue are exactly the same. Sorry about this.

As you advised, I printed all the bars and I attached it again. I also pasted this in a spreadsheet to be 100% sure that the outcomes are the same.

[NinjaTrader_Jesse]

Hello guillembm,

Can you export the data needed that creates different results, also reduce the test period so it makes a few trades as possible and still demonstrates differences. Please include the samples that have prints that show how the condition evaluated to submit those different trades

Please let me know if I may be of further assistance.

[guillembm]

Sorry for the delay. I have been out these days.

I attach the requested data. I am experiencing issues compressing the requested market data to match your limitations. I tried to split the data but you do not accept the format. Can you provide, please, an e-mail to send it?

Thanks for your support in solving this issue.

[NinjaTrader_Jesse]

Hello guillembm,

It looks like this is just the output and settings,

Can you export the data you had used which creates different results, also reduce the test period so it makes a few trades as possible. You have nearly a year in the image, what is the least amount of data to see the result?

Please also include the samples that have prints that show how the condition evaluated to submit those different trades

Please let me know if I may be of further assistance.

[guillembm]

As I told you in my previous post:

"I am experiencing issues compressing the requested market data to match your limitations. I tried to split the data but you do not accept the format. Can you provide, please, an e-mail to send it?"