Slow Backtest & Optimisation with Tick Replay

[Nystagmus]

Hi,

Trying here to decrease the backtest and optimization time to as low as possible and need some help.

1) I am have my own strategy and I am not using any type of code which could increase the speed because first I need to make sure its using the full hardware capability which is not.
Would need some advice in what ideas could you give so I can include in the code. Some articles maybe or smth else.

2) The main question:

I downloaded the needed data from beginning of 2023 as seen as bellow:



For better accuracy I use "tick replay" always for backtest and optimizations.
==========================
- Specifications of my PC:

Windows 10 Pro
Processor - AMD Ryzen 7 2700 8-Core
Logical Processors - 16
Base Speed - 3.2 GHz
RAM - 16GB

Always when running the backtest or optimizations the CPU does not go above 10% and RAM does not go above 30% and I am wondering WHY?
My understanding and wish would be to use at least 90% of everything and why not 100%.
So in the end the backtest takes around 2.5 hours not speaking of optimizations which take around 6 days for 60 iterations.

====================================

Same on my laptop which is less powerful on specifications.
While tests, they never reach even 60% on CPU or RAM.
The laptop as seen bellow is working in a "warm mode" and because of this, a tick replay of 6 months of GC takes around 2.5 hours also as on my PC which I assume if running at 100% would take around 45min.



So, can anyone give any hints? Maybe there are some settings in windows stopping NT8 of using full power?

Thanks in advance.

[NinjaTrader_BrandonH]

Hello Nystagmus,

Thanks for your post.

There is no supported or documented code we could provide to increase the speed of backtests and optimizations in the Strategy Analyzer.

Computer memory is important for long resource-intensive tests that generate a large number of trades without your machine stalling. Computer processor speed is important to get optimization iterations done quickly and the number of CPU cores allows for more iterations to run simultaneously.

We should consider the following for memory consumption:

Data * Strategy resources * Number of backtest/optimization iterations * Number of trades * Keep best # of results (when optimizing).

With more complex code can come heavier calculations which may require more processing. This is further impacted by the amount of data being processed, the number of iterations in the optimization, and amount of new objects being created in each iteration.

It is important to note that the Tick Replay property implies that more PC resources are used to calculate your indicators and strategies also and as a result will lead to a performance impact. This is noted on the Tick Replay and Performance Tips help guide pages below.

Tick Replay: https://ninjatrader.com/support/help...ick_replay.htm
Performance Tips: https://ninjatrader.com/support/help...ance_tips2.htm

To improve backtests/optimization, you could reduce the amount of days being tested over to reduce the data being processed. You could also reduce the number of iterations by reducing the range of optimized parameters when running an optimization.

Further, when running an optimization use IsInstantiatedOnEachOptimizationIteration set to false and reset all objects that can be reset or instantiate all objects in State.DataLoaded. By doing so this will re-use memory locations instead of reserving new memory locations.

See the help guide documentation below for more information about running optimizations.

IsInstantiatedOnEachOptimizationIteration — https://ninjatrader.com/support/help...niteration.htm
Optimization Tips — https://ninjatrader.com/support/help...ionPerformance
Walk Forward Optimization — https://ninjatrader.com/support/help...ss_metrics.htm
Genetic Optimization — https://ninjatrader.com/support/help..._algorithm.htm

This forum thread will also be open for other community members to share their insights on the topic or unsupported code or Windows settings you could try.

[Nystagmus]

Hello Brandon,



1) All the above info is known to me and all those articles I have seen in other Forum answers, but my question is different, why is it not using 100% of my PC, laptop and even a VM resources.
The results always the same- not using full power.
By the way 3 of the links are outdated and not found so this seems to be a template answer...
I would really like someone to investigate this and help me.

2) During 5 months on backtest I have 132 trades.

3) You also say "It is important to note that the Tick Replay property implies that more PC resources are used"
This seems to not be true because with tick replay or without, the % of CPU or RAM does not differ and not getting higher or lower.
I would really like someone to investigate this and help me.

4) I am not very sure what do you mean by " * Strategy resources"

5) For the "IsInstantiatedOnEachOptimizationIteration" I would need to know when the variables of a cycle are to be reset, I mean the reset of variables occurs in every trade or at the end of the backtest.

So the question is when do I have to reset the variables.
This is important because I have variables that are reset for each trade and at the beginning of the day in the strategy.

In the example from Article the reset is done in DataLoad.


Regards.

[NinjaTrader_BrandonH]

Hello Nystagmus,

Thanks for your notes.

You would not be able to control how NinjaTrader utilizes CPUs and memory on your machine.

NinjaTrader 8 is multi-threaded and uses dispatcher thread pool control. From there, the windows operating system designates which logical processor (CPU core) any thread will be run on and the operating system will handle load balancing.

Data, orders, backtests, optimizations, are all processed separately in different threads than the consumer script's thread.

IsInstantiatedOnEachOptimizationIteration could be used in your script to improve the performance of Strategy Analyzer runs.
https://ninjatrader.com/support/help...antiatedoneach optimizationiteration.htm

Objects should only be initialized in State.DataLoaded when absolutely necessary, existing objects that can be reset like arrays should be cleared / reset instead of new objects initialized.

The IncludeTradeHistoryInBacktest property can make a huge performance impact. If the script is not using the SystemPerformance collection, setting IncludeTradeHistory to false in State.Configure can dramatically improve backtest and optimization performance.
https://ninjatrader.com/support/helpGuides/nt8/NT%20HelpGuide%20English.html?includetradehistoryi nbacktest.htm

If the script is making a lot of Draw method calls, this can also have a large impact. Disabling draw method calls when the script is in the Strategy Analyzer can reduce this impact.
https://ninjatrader.com/support/helpGuides/nt8/index.html?isinstrategyanalyzer.htm

Below is a link to an example that demonstrates the effect of IsInstantiatedOnEachOptimizationIteration.
https://ninjatrader.com/support/foru...995#post802995

As far as the total utilization, there may be other factors about the machine that may come into play such as the windows power settings. NinjaTrader is designed to use all logical processors available; specifically, it creates new threads and thread pools for the operating system to manage and does not limit or throttle the CPU core usage.

[Nystagmus]

Hi BrandonH,

Thanks a lot for the answer.
More things are clear now but unfortunately the only option that I will be able to include in the strategy is the "IsInstantiatedOnEachOptimizationIteration" due to the structure of the code and strategy itself.

The most annoying thing will still remain, that a powerful machine with 8 Cores and 16 threads acts exactly the same, timewise, as the 4 Cores 4Threads.
My today test showed that that the machine which is more then 2 times powerful was only 5 seconds faster for the backtest with 100% same settings.

Would be very cool to have this improved in the next NT updates.
You know, some parameters with checkmarks about how much power you allow NT to use.

Anyway, thanks a lot for the time spent.
If anytime you think there are some other things that can be added in the code or other method, please feel free to add in this discussion.

Very much appreciated.
Have a good one.

[NinjaTrader_BrandonH]

Hello Nystagmus,

Thanks for your notes.

We are tracking interest for improving the performance of backtest/optimizations in an internal feature request and I have added your vote and notes to this feature request for the Development team to take into consideration. This request is being tracked under the number SFT-5223.

As with all feature requests, interest is tracked before implementation is considered, so we cannot offer an ETA or promise of fulfillment. If implemented, it will be noted on the Release Notes page of the Help Guide.

Release Notes — https://ninjatrader.com/support/help...ease_notes.htm

[rayko]

Hi Nystagmus & Brandon

I’ve made this short 5-minute video showing how my PC CPU/memory responds while running Strategy Analyzer sitting next to a Windows TaskMgr.

Camtasia Screencast: https://app.screencast.com/4HEZV6Sk6dvBo

In Summary: The video shows how Strategy Analyzer testing can accumulate PC Standby memory till the User must Restart Ninjatrader in order to reclaim memory and prevent failure.

I can only run 3 consecutive Strategy Analyzer reports before I must Restart NT

I’ve provided a lot of details and effort because I would really like this to be fixed. I use the Strategy Analyzer reports to build Excel Pivot tables of my results to give me an edge on what to trade during the week, so this is important to me.

What I’m really hoping is that Brandon and the good folks at Ninjatrader can provide a simple fix. I really don’t care what additional coding I must do, I'm tired of having to restart NT over and over to reclaim memory.

Method: The same strategy is executed 4 times with Windows 11 on an 11th Gen Intel(R) Core(TM) i9-11900 @ 2.50GHz.

The strategy is being run on a PC (beside me) connected remotely by another PC (also beside me) that is filming the test using Camtasia.

About the test: The strategy was executed on 3 Wednesdays for the past 3 weeks between 2023-05-17 and 2023-06-01 using Minute Data iterating from 1 minute to 5 minutes by 1 minute.

The same test Date Range is used for each of the 4 tests.

Note: The fourth test was to take about 20s to execute. It was able to perform 5 iterations before the strategy imploded to become unresponsive forcing NT shutdown.

About the strategy: One Optimization normally takes 18 seconds and 75 iterations. Here is a snippet of my code at State.SetDefaults:

Calculate = Calculate.OnEachTick;
StartBehavior = StartBehavior.WaitUntilFlat;
TimeInForce = TimeInForce.Gtc;

ShowTransparentPlotsInDataBox = true;

TraceOrders = false;

RealtimeErrorHandling = RealtimeErrorHandling.IgnoreAllErrors;
StopTargetHandling = StopTargetHandling.PerEntryExecution;
BarsRequiredToTrade = 20;
// Disable this property for performance gains in Strategy Analyzer optimizations
// See the Help Guide for additional information
IsInstantiatedOnEachOptimizationIteration = false;

// Exclude trade history in a backtest to benefit from memory savings
IncludeTradeHistoryInBacktest = false;

DisplayInDataBox = true;
DrawOnPricePanel = true;

The strategy uses the “Managed” order approach and the following order types;
EnterLong, EnterShort
EnterLongStopMarket, EnterShortStopMarket
ExitLongStopMarket, ExitShortStopMarket
ExitLongLimit, EnterShortLimit

About the Video:

• The first run took 18 seconds to complete and there is 10,416 MB of Standby Memory left over. (at 32 seconds in video)
• After the second execution took 21 seconds to complete and there is 19,516 MB of Standby Memory left over. (at 1:28 in video)
• After the third execution took 20 seconds to complete and there is 25,562 MB of Standby Memory left over. (at 2:23 in video)
• In between executions, I left at least 15 seconds for the machine to begin to recover memory, before beginning the next test execution.
• At 2:30 in the video, memory begins to recover.
• At 3:27, Standby memory is 17,211 MB decreased (25,562-17,211) by 8,351 MB).
• At 3:43, Standby memory is 25,846 MB increased (25,846 -17,211) by 8,635 MB).
• At this point, there is no more free physical memory.
• At 3:50 I start another test of the same strategy and parameters.
• At 4:05, Ninjatrader implodes and is unresponsive.
• At 4:21, NT using 36% cpu and all memory, and unresponsive.
• At 5:01, NT using 31% cpu and all memory, and unresponsive.
• At 5:54, terminated NT using Windows taskmgr
• At 6:02, Windows memory is recovered and NT is down.

.

[NinjaTrader_BrandonH]

Hello rayko,

Thanks for your notes.

Yes, optimizations use a significant amount of memory and yes, the memory is not decommitted right away.

My colleague Jim has created a video demonstrating how the SampleMACrossover strategy, a simple strategy that utilizes IsInStantiatedOnEachOptimizationIteration=false for efficiency, can still quickly utilize memory resources.

Demo — https://drive.google.com/file/d/15pz...w?usp=drivesdk

As we can see there are a number of factors that are involved and memory utilization can climb very quickly depending on a few of these factors.Once memory gets maxed out, we can experience short freezes where memory is decommitted, stored to disk, and then new resources are committed before the backtest/optimization is resumed.

There are no supported means for forcing garbage collection and NinjaTrader 8 just lets .NET handle it.

Something else to consider with optimization backtests: By default NinjaTrader will create new instances of the strategy for each optimization iteration. A strategy can be programmed to use IsInstantiatedOnEachOptimizationIteration=false (class level variables will need to be reset in State.DataLoaded) and the optimizer will not create new instances of the strategy for each optimization iteration. The additional optimization iterations will increase the amount of memory utilized, and will increase the time needed to complete the backtest/optimization.

IsInstantiatedOnEachOptimizationIteration - https://ninjatrader.com/support/help...niteration.htm

We are tracking interest in having NinjaTrader implement a way to force garbage collection in an internal feature request and I have added your vote to it. This request is being tracked under the number SFT-2111.

As with all feature requests, interest is tracked before implementation is considered, so we cannot offer an ETA or promise of fulfillment. If implemented, it will be noted on the Release Notes page of the Help Guide.

Release Notes — https://ninjatrader.com/support/help...ease_notes.htm

[rayko]

Hello Brandon

I hope you noticed my strategy does in fact have IsInstantiatedOnEachOptimizationIteration = true. You didn’t need to paste that part. It's already in the strategy see above.

I look forward to seeing garbage collection implemented into Strategy Analyzer.
Right now SFT-2111 is the only thing on the MENU that matters to me

Best Regards,