Strategy Analyzer Optimization result table shows incorrect data

[wzgy0920]

After I run the Optimization in Strategy Analyzer, the top results are displayed in a table. The table columns such as Net Profit, Percent Profitable, etc.. However the values displayed in the table is incorrect and inconsistent with the actual values shown in the "Summary" when you select a set of parameter. See the screenshot for an example.

[NinjaTrader_ChelseaB]

Hello wzgy0920,

Please update to 8.1.1.7 and confirm you are still able to reproduce the behavior.

To update NinjaTrader

• Shutdown all programs including NinjaTrader *important
• Download NinjaTrader from the link below
  Click here to download NinjaTrader
• Double click the downloaded file to execute it
• Follow the on screen instructions
• Restart NinjaTrader

You can read about changes to this release in the release notes linked below.
https://ninjatrader.com/support/help...ease_notes.htm

[wzgy0920]

Hello wzgy0920,

Please update to 8.1.1.7 and confirm you are still able to reproduce the behavior.

To update NinjaTrader

• Shutdown all programs including NinjaTrader *important
• Download NinjaTrader from the link below
  Click here to download NinjaTrader
• Double click the downloaded file to execute it
• Follow the on screen instructions
• Restart NinjaTrader

You can read about changes to this release in the release notes linked below.
https://ninjatrader.com/support/help...ease_notes.htm

I updated to v8.1.1.7 and I am still having the same issue. See attached screenshot.

[NinjaTrader_ChelseaB]

Hello wzgy0920,

Please test the SampleMACrossover included with NinjaTrader. Are you able to reproduce with this as well?

[wzgy0920]

Hello wzgy0920,

Please test the SampleMACrossover included with NinjaTrader. Are you able to reproduce with this as well?

With the SampleMACrossover Optimization results, I do not get the issue. Is it something wrong with my strategy? How can I troubleshoot?

[NinjaTrader_ChelseaB]

Hello wzgy0920,

This might indicate an issue with the strategy, such as using variables to dynamically call AddDataSeries() or using randomly generated values.

May I test your script on my end to see if I am able to reproduce?

To export a NinjaTrader 8 NinjaScript so this can be shared and imported by the recipient do the following:

1. Click Tools -> Export -> NinjaScript Add-on...
2. Click the 'add' link -> check the box(es) for the script(s) and reference(s) you want to include
3. Click the 'Export' button
4. Enter a unique name for the file in the value for 'File name:'
5. Choose a save location -> click Save
6. Click OK to clear the export location message

By default your exported file will be in the following location:

• (My) Documents/NinjaTrader 8/bin/Custom/ExportNinjaScript/<export_file_name.zip>

Below is a link to the help guide on Exporting NinjaScripts.
http://ninjatrader.com/support/helpG...nt8/export.htm


To troubleshoot an issue, reduce the script. Remove or comment out all logic and test very simple logic like entering when flat.

The uncomment code until the behavior returns.

Below is a link to a forum post on the debugging process.

https://ninjatrader.com/support/forum/forum/ninjatrader-7/strategy-development-aa/99209-date-conditions-not-working?p=791121#post791121

[wzgy0920]

As I am troubleshooting, I am wondering if the issue could be related to the code below in my OnBarUpdate(). I do this to prevent any ghost orders/position when loading my strategy in realtime. However would this cause an issue for Optimization and Strategy Analzyer?

if (State == State.Historical && !IsInStrategyAnalyzer)
return;​

[NinjaTrader_ChelseaB]

Hello wzgy0920,

The suggested code would stop evaluating code if the state is historical and the window is not the strategy analyzer.

You can add prints to see if code is being reached below this point.

[wzgy0920]

I have some new insight, but no solution. I think the issue is related to what is the value set for IsInstantiatedOnEachOptimizationIteration when running Optimization.

• If I set IsInstantiatedOnEachOptimizationIteration=false, some of the values displayed in the table columns as "top results" are different from the actual values show in Summary when selecting the specific result parameter set
• If I set IsInstantiatedOnEachOptimizationIteration=true, the values displayed in the table columns as "top results" are correct and same as the actual values shown in Summary when selecting the specific results parameter set
• When selecting a specific result parameter set and running a Backtest on it, the results are always the same, independent of IsInstantiatedOnEachOptimizationIteration.

I am unsure why this issue is occurring. I have thoroughly reviewed my code to ensure I am resetting class level variables in my strategy as well as the custom indicator my strategy is using. I have also read the following links below:

• https://forum.ninjatrader.com/forum/...-false-problem
• https://ninjatrader.com/support/help...niteration.htm

I don't really understand how the internals of the Optimization works since its using multi-core processing. However, I wonder if the issue could be related to how following ways of how much indicator and strategy works:

• My custom indicator only presents signals based on the price action of the current trading session. It does care what the price action was on the previous day's session
• My strategy's results will differ based on when the strategy is enabled. For example, if I started my strategy at 10am EST it will result in a series of trades that are different from the series of trades that would occur if I were to start the strategy at 10:30am EST. This is because, my strategy will only trade one setup at a time. If it is currently holding a position, it will ignore any new trade signals until it is flat again.

[NinjaTrader_ChelseaB]

Hello wzgy0920,

When IsInstantiatedOnEachOptimizationIteration is false, all variables would need to be instantiated / reset in State.DataLoaded.
Attached is an example.

For assistance understanding the behavior, please provide the output from prints of each value used in the condition being investigated.
https://ninjatrader.com/support/foru...121#post791121

[wzgy0920]

Hello wzgy0920,

When IsInstantiatedOnEachOptimizationIteration is false, all variables would need to be instantiated / reset in State.DataLoaded.
Attached is an example.

I already did that as mentioned in the previous post. Both in my strategy and custom indicator, I made sure to all the variables are being clear or reset in the State.DataLoaded. I followed the guidance in these posts: https://ninjatrader.com/support/help...niteration.htm and https://forum.ninjatrader.com/forum/...-false-problem

[NinjaTrader_ChelseaB]

Hello wzgy0920,

May I have you create a simplified script that can reproduce the behavior.

Make sure this is a new test script that has the bare minimum necessary code to reproduce. All of the unnecessary properties and logic should not be added. (A simple entry / exit would suffice)

To export a NinjaTrader 8 NinjaScript so this can be shared and imported by the recipient do the following:

1. Click Tools -> Export -> NinjaScript Add-on...
2. Click the 'add' link -> check the box(es) for the script(s) and reference(s) you want to include
3. Click the 'Export' button
4. Enter a unique name for the file in the value for 'File name:'
5. Choose a save location -> click Save
6. Click OK to clear the export location message

By default your exported file will be in the following location:

• (My) Documents/NinjaTrader 8/bin/Custom/ExportNinjaScript/<export_file_name.zip>

Below is a link to the help guide on Exporting NinjaScripts.
http://ninjatrader.com/support/helpG...nt8/export.htm

Once exported, please attach the file as an attachment to your reply.

[wzgy0920]

Hi ChelseaB,

I have insured all my class level variables are being cleared and being reset inside the State.DataLoaded. I am still have the issue I reported. I have been trying to troubleshoot the issue and I have some follow up questions.

I noticed that state transitions to State.DataLoaded only occurs once per parameter set during optimization, even though the backtest can span over multiple days. For example, assume I am doing Optimization on ES from 9/18/2023 to 9/22/2023, which is span of 5 days. But when Optimization runs, transition into State.DataLoaded will only occur ONCE per the different parameter set that is being tested. My strategy requires all my variables be cleared and reset on per day trading session. Thus, at 6:00:00pm EST on a trading day (market open time), I want all my class variables to be reset and cleared. In addition, before the end of the trading session, i.e. 4:59:59pm EST, I want all positions to be exited and account to be flat. How can I make sure both these requirements are met, specifically during Optimization and Backtesting?

[wzgy0920]

What exactly happens when IsInstantiatedOnEachOptimizationIteration=false? The online document says "Determines if the strategy should be re-instantiated (re-created) after each optimization run when using the Strategy Analyzer Optimizer.". However, what does it mean by "each optimization run"? What parameters are changed when "each optimization run" happens?

[NinjaTrader_ChelseaB]

Hello wzgy0920,

If you also want to reset variables on a new session, the logic in OnBarUpdate() would need to accomplish this. You can reset when Bars.IsFirstBarOfSession is true.
https://ninjatrader.com/support/help...rofsession.htm

An optimization will run a backtest (iteration) for each set of input parameters being optimized over.

For example with the SampleMACrossover using Fast as 1;5;2 and Slow 1;1;1 will optimize 3 backtest iterations. Fast 1 / Slow 1, Fast 3 / Slow 1, Fast 5 / Slow 1.

When IsInstantiatedOnEachOptimizationIteration is false, NinjaTrader will re-use generated strategy class objects for new backtest iterations and restart the historical data back at bar 0.
When IsInstantiatedOnEachOptimizationIteration is true, a new strategy class object is generated for every iteration which will use more memory and CPU resources.