Usability and performance improvements to Strategy Analyzer

[wzgy0920]

I have questions and suggestions related Strategy Analyzer.

1) Currently the analysis of Optimization results is very very slow and painful. Please improve this. I run Optimization on multiple variables that span multiple weeks. The Optimization itself takes few hours, after which I am presented with top results. When I am analyzing the results, I am clicking/browsing through each of the presented results. However, every time I click on one of the results, Ninjatrader runs a Backtest on those set of parameters which freezes the Strategy Analyzer for few tens of seconds until the Backtest is done running. This is super annoying and painful because it slows down the analysis process. I don't understand why Ninjatrader re-runs the Backtest when it already did the backtest during Optimization. I believe Ninjatrader developers should improve the performance and usability of Strategy Analyzer by caching the backtest results it gathers during Optimization.

2) Is it possible to export the results of Backtest/Optimization and import it to another PC? I would like to do the Optimization on a PC that has higher performance but then transfer the results to my primary PC for analysis.


3) Is it possible to export the Backtest/Optimization results to an HTML format that is browsable without running Ninjatrader? It would be great if this feature can be introduced so we can analyze/share the results (Summary, Graphs, Trades, etc...) without having to view it in Ninjatrader itself.

[NinjaTrader_Gaby]

Hello wzgy0920,

Thank you for your post.

I have submitted a feature request to the Development Team to cache backtest so this is not re-run when selecting an optimization backtest result. It is being tracked with SFT-6074.

Computer memory is important for long resource-intensive tests that generate a large number of trades without your machine stalling. Computer processor speed is important to get optimization iterations done quickly and the number of CPU cores allows for more iterations to run simultaneously.

We should consider the following for memory consumption:

Data * Strategy resources * Number of backtest/optimization iterations * Number of trades * Keep best # of results (when optimizing).

With more complex code can come heavier calculations which may require more processing. This is further impacted by the amount of data being processed, the number of iterations in the optimization, and amount of new objects being created in each iteration.

To improve backtests/optimization, you could reduce the amount of days being tested over to reduce the data being processed. You could also reduce the number of iterations by reducing the range of optimized parameters when running an optimization.

Further, when running an optimization use IsInstantiatedOnEachOptimizationIteration set to false and reset all objects that can be reset or instantiate all objects in State.DataLoaded. By doing so this will re-use memory locations instead of reserving new memory locations.

The IncludeTradeHistory can make a huge performance impact. If the script is not using the SystemPerformance collection, setting IncludeTradeHistory to false in State.Configure can dramatically improve backtest and optimization performance.

https://ninjatrader.com/support/help...inbacktest.htm

If the script is making a lot of Draw method calls, this can also have a large impact. Disabling draw method calls when the script is in the Strategy Analyzer can reduce this impact.

https://ninjatrader.com/support/help...gyanalyzer.htm

See the help guide documentation below for more information about running optimizations.

IsInstantiatedOnEachOptimizationIteration - https://ninjatrader.com/support/help...niteration.htm

Understanding factors that affect optimization performance - https://ninjatrader.com/support/help...ionPerformance


To export the results of a backtest, you can right-click in the log and click on "Export".

Using Logs - https://ninjatrader.com/support/help....htm#UsingLogs

This will be saved as an excel file. You can open this file without having to open up NinjaTrader.

Please let us know if you have any other questions.

[wzgy0920]

Hello wzgy0920,

To improve backtests/optimization, you could reduce the amount of days being tested over to reduce the data being processed. You could also reduce the number of iterations by reducing the range of optimized parameters when running an optimization.

Further, when running an optimization use IsInstantiatedOnEachOptimizationIteration set to false and reset all objects that can be reset or instantiate all objects in State.DataLoaded. By doing so this will re-use memory locations instead of reserving new memory locations.

I already optimized my strategy for better performance for Backtesitng/Optimization by turning off plots and using IsInstantiatedOnEachOptimizationIteration=false during Backtest/Optimization. I cannot reduce the amount of days to backtest on during optimization because I need larger dataset to improve optimization.

One more question: For faster optimization, what is more important? More cores or more threads? What I mean to ask is, does Optimization use multi-threading or multi-core processing?

I would like to re-iterate that NT needs to cache the Backtest results of the top results presented from the Optimization. The results analysis is really slow right now because NT re-runs a backtest on the top results.

To export the results of a backtest, you can right-click in the log and click on "Export".

Using Logs - https://ninjatrader.com/support/help....htm#UsingLogs

This will be saved as an excel file. You can open this file without having to open up NinjaTrader.

Please let us know if you have any other questions.

Is it possible to export the Optimization results in the native Ninjatrader readable format? My use case is that I want to do the optimization on a PC which has a higher memory/CPU speed. Then I want to transfer the results to a regular PC which also has NInjatrader, so I can do further analysis of the results.

[NinjaTrader_Gaby]

Hello,

NinjaTrader 8 is multi-threaded and uses dispatcher thread pool control. From there, the windows operating system designates which logical processor (CPU core) any thread will be run on and the operating system will handle load balancing.

As mentioned, when backtesting computer processor speed is important to get optimization iterations done quickly and the number of CPU cores allows for more iterations to run simultaneously. It relies on CPU resources.

That being said, since you are backtesting over a large number of days this will require very large amounts of memory.

To save backtest results, you can right click in the grid and click on Export, to use excel format you can select that from the dropdown where .csv may be selected. The dropdown has both csv and excel format.

Please let us know if you have any other questions.

[wzgy0920]

To save backtest results, you can right click in the grid and click on Export, to use excel format you can select that from the dropdown where .csv may be selected. The dropdown has both csv and excel format.

Saving the Optimization results in Excel/CSV format is non-native to Ninjatrader. I want to be able to import the Optimization results onto a different PC running Ninjatrader where I want to be able to open the Strategy Analyzer and do further analysis of the Optimization results within Ninjatrader. i.e. I want to be able to view the Summary, Trades, Executions, Graphs of each of the Optimization results in a different Ninjatrader PC.

[NinjaTrader_Gaby]

Hello,

Unfortunately there is no supported method for transferring optimization results to import in NinjaTrader on another computer. If you are interested, I can submit this as a feature request to our Development Team. I can follow up with an internal tracking number for your reference as soon as it is created.

Thanks in advance for your patience.

[wzgy0920]

Hello,

Unfortunately there is no supported method for transferring optimization results to import in NinjaTrader on another computer. If you are interested, I can submit this as a feature request to our Development Team. I can follow up with an internal tracking number for your reference as soon as it is created.

Thanks in advance for your patience.

Yes please can you submit a feature request.

[NinjaTrader_Gaby]

Thank you for your suggestion.

This request is being tracked under the number SFT-6102.


As with all feature requests, interest is tracked before implementation is considered, so we cannot offer an ETA or promise of fulfillment. If implemented, it will be noted on the Release Notes page of the Help Guide.


Release Notes —https://ninjatrader.com/support/helpGuides/nt8/?release_notes.htm

[wzgy0920]

Hello wzgy0920,

Computer memory is important for long resource-intensive tests that generate a large number of trades without your machine stalling. Computer processor speed is important to get optimization iterations done quickly and the number of CPU cores allows for more iterations to run simultaneously.

We should consider the following for memory consumption:

Data * Strategy resources * Number of backtest/optimization iterations * Number of trades * Keep best # of results (when optimizing).

With more complex code can come heavier calculations which may require more processing. This is further impacted by the amount of data being processed, the number of iterations in the optimization, and amount of new objects being created in each iteration.

To improve backtests/optimization, you could reduce the amount of days being tested over to reduce the data being processed. You could also reduce the number of iterations by reducing the range of optimized parameters when running an optimization.

What contributes to a single Backtest run (not Optimization)? How can I make a single Backtest run faster. For example, I am running a Backtest for 2 months of data. However, it takes quite long to finish (~1 min). When the Backtest is running, I do NOT see my CPU maxed out at 100%. In addition, my RAM usage is NOT also maxed out. To be specific, I have 16GB of RAM and it is still at ~8GB used when Backtest is running, while CPU is ~23%

[NinjaTrader_Gaby]

Hello,

A single backtest will run as a single thread which runs on a single logical processor. This will be limited by the speed of that logical processor. On a CPU with 4 logical processors, a single logical processor will max out at 25% of the total CPU.


Please see this forum post which discusses what can have an impact on memory when using the Strategy Analyzer:

https://forum.ninjatrader.com/forum/...ng#post1069339


If you have any questions, please let me know.

[wzgy0920]

Hello,

A single backtest will run as a single thread which runs on a single logical processor. This will be limited by the speed of that logical processor. On a CPU with 4 logical processors, a single logical processor will max out at 25% of the total CPU.

Is this a limitation of NT or Windows? I have 4 cores (8 logical processors) that can run up to 3.4 Ghz. Yet it is surprising to me that a single backtest test takes so long to run.

[NinjaTrader_Gaby]

Hello,

This is a limitation of programming with any operating system including Linux, iOS, Android, and Windows, not something specific to NinjaTrader. A single thread can only run on a single logical processor. When doing an optimization, the backtest iterations are run as separate threads where each backtest iteration is being processed on its own logical processor.

That said, the NinjaTrader optimization engine is optimized to take advantage of as much system resources as possible. From the Help Guide:

"Please insure that you have as much system resources available to the optimization as possible, this usually means making sure all other applications are closed. Furthermore as as the NinjaTrader optimization engine is optimized to take advantage of as much system resources as possible it is advisible not to trigger an optimization during a time where you would need to be using the PC. For example it is not advised to start an optimization while you are managing the exit of a trade."

https://ninjatrader.com/support/helpGuides/nt8/index.html?optimize_a_strategy.htm#UnderstandingFa ctorsThatAffectOptimizationPerformance

Please let us know if you have any other questions.

[wzgy0920]

Hello,

This is a limitation of programming with any operating system including Linux, iOS, Android, and Windows, not something specific to NinjaTrader. A single thread can only run on a single logical processor. When doing an optimization, the backtest iterations are run as separate threads where each backtest iteration is being processed on its own logical processor.

That said, the NinjaTrader optimization engine is optimized to take advantage of as much system resources as possible. From the Help Guide:

"Please insure that you have as much system resources available to the optimization as possible, this usually means making sure all other applications are closed. Furthermore as as the NinjaTrader optimization engine is optimized to take advantage of as much system resources as possible it is advisible not to trigger an optimization during a time where you would need to be using the PC. For example it is not advised to start an optimization while you are managing the exit of a trade."

https://ninjatrader.com/support/helpGuides/nt8/index.html?optimize_a_strategy.htm#UnderstandingFa ctorsThatAffectOptimizationPerformance

Please let us know if you have any other questions.

So does this mean when running a single backtest, it will only use one thread and the speed of the completion will be determined by the processor speed (e.g. 3.4 Ghz)? Is it possible you for you guys to utilize multiple threads in a single backtest or utilize the GPU for further speed improvements? Also, does the speed of the RAM (e.g. 2133MHz) have any affect on how fast a backtest/optimization takes? Also, regarding optimization, is the speed determine by the number of threads available? So if the CPU has total 12 threads, does this mean optimization will do 12 iteration at a time?

[NinjaTrader_Gaby]

Hello,

So does this mean when running a single backtest, it will only use one thread and the speed of the completion will be determined by the processor speed (e.g. 3.4 Ghz)?

Yes, a single backtest will run as a single thread which runs on a single logical processor. This will be limited by the speed of that logical processor.

Is it possible you for you guys to utilize multiple threads in a single backtest or utilize the GPU for further speed improvements?

Unfortunately no, currently a backtest cannot be multi-threaded. However, if you would like we would be happy to submit a feature request to our Development Team to see if this can be improved upon.

Also, does the speed of the RAM (e.g. 2133MHz) have any affect on how fast a backtest/optimization takes?

Yes, the speed of the RAM, processor, hard drive, front side bus, etc. will all have an impact on how fast a processor is able to process calculations.

Also, regarding optimization, is the speed determine by the number of threads available? So if the CPU has total 12 threads, does this mean optimization will do 12 iteration at a time?

Yes, the number of simultaneous backtest iterations in an optimization is limited by the available number of logical processors.

Please let us know if you have any other questions.

[wzgy0920]

Unfortunately no, currently a backtest cannot be multi-threaded. However, if you would like we would be happy to submit a feature request to our Development Team to see if this can be improved upon.

Yes, please add my vote to allow for feature request to speed up Optimization using GPU.