Optimizer Really Slow Suddenly

[proptradingshop]

From a couple days ago to today all of a sudden the strategy optimizer is running extremely slow. 1000 iterations used to take several minutes now takes closer to an hour. There has been no code changes to the strategy itself. It seems to run fast for like 30-40 iterations then gets hung. Then runs fast for 30-40 iterations and gets hung and repeats this with longer hang times until optimization is complete.

It seems that CPU usage of the optimizer has drastically increased and now hits 100% where before it would not. I am simply testing the sample ma crossover to monitor the issue.

I have tried removing my workspace and opening a new workspace.
Deleting cached files, db, bin, obj, recompiled in NT8
Restarted NT8 several times
Rebooted my PC
Reset & Repair DB

I am using IQ data feed and looking at the NinjaScript utilization monitor didn't really provide much insight.

I also ran the same optimization on a machine with lesser resources and I can see it performing as expected.

My question is, is there anything else I can try to figure out the root cause of the issue before resorting to completely wiping NT8 off this machine and doing a fresh install.

[NinjaTrader_ChelseaB]

Hello proptradingshop,

The strategy analyzer becoming slow typically means the computer has run out of memory.

Below is a link to a support article with some helpful tips on improving the performance.

https://ninjatrader.com/support/forum/forum/ninjatrader-8/platform-technical-support-aa/1069302-memory-bug-when-wfa-optimizing?p=1069339#post1069339

[proptradingshop]

Hi Chelsea, thanks for the response. I have 64gb RAM and memory never goes above 50% when running the optimizer. It is the CPU spike that started happening for the last couple days. I am only keeping the best 10 results of the optimization and also removed all drawings. This seemed to come out of no where but I don't see any signs pointing towards a memory issue as I mentioned memory is consistently under 50% during optimization. Is there anything else? I did try both IsInstantiatedOnEachOptimizationIterationtrue and false and that did not make any difference.

I have looked at all the logs and nothing really stands out as far as exceptions or code issues.

I am looking for how I can determine what is the root cause of the sudden change in performance, what other debugging steps can I take?

[NinjaTrader_ChelseaB]

Hello proptradingshop,

For debugging steps you can comment out all logic and test to confirm the behavior does not occur then uncomment one line at a time and test for the behavior after uncommenting each line to see which line is causing the performance impact.

"It is the CPU spike that started happening for the last couple days."

The market data is always changing. Unless you are testing the same instrument with the exact same code on the exact same date range, the results and performance is going to be different.

If the script is processing tick data (TickReplay or an added 1-tick series) more ticks means more data points which means more processing.

[proptradingshop]

Thanks for the suggestions. The script is processing on bar close. It is the sample MA script with drawings commented out. It is being ran for the same date range and tickers as before that used to take a few minutes but now takes a few hours. This is not a slight difference, this is a major difference hence why it caught my attention.

One thing I noticed is that when logging the strategy instance I see Strategy 'Sample MA Crossover/-1':
What is the -1 in this case and what is it coming from? I can recreate this same log when running other strategies and I don't remember if this -1 was there. For example Strategy 'OpeningRangeBreakout/-1': An order has been ignored since the stop price ‘78.06’.

I am just mentioning the -1 because I don't remember it showing up in logs before and I see it now so don't know if it is related or not. Just running out of ideas at this point.

What would be the next steps in identifying the root cause of this issue?

[NinjaTrader_ChelseaB]

Hello proptradingshop,

The Sample MA Crossover included with NinjaTrader does not call any Draw methods. Further, this is a locked script you cannot modify.

Did you create a copy and modify the copy?

What is the bar type and interval used for the test?

What are the optimization ranges of the two inputs?

What is the Start and End dates?

[proptradingshop]

Hi Chelsea, it is a copy of the script and I removed the AddChartIndicator calls.

1. It doesn't matter, this is not occurring on just a single strategy optimization. I am comparing how long the optimization runs would take me a couple days ago versus today but they are mostly some variation of 5 - 60 minute bars dating back 1-2 years.
2. I am running 1000 iterations as a benchmark.

I hope you have an understanding of the problem, I was running optimizations that would take 5-6 hours a few days ago and today those same optimizations show that it will take days.

The optimizer seems to go through 30-40 iterations quickly then gets hung for a minute and repeats this pattern with each hang increasing in time. There has been no code changes in any of my strategies and this just started appearing. Each OnBarUpdate is running typically in ms so the problem is not in the strategies logic. Like I mentioned the CPU has began to spike which it was not dong a couple days ago for the same optimization runs.

I am looking for a little more advanced debugging to find the new bottleneck as I have gone through all the surface level possibilities.

Thank you for the support, please let me know what additional steps you would take to help identify the issue.

[NinjaTrader_ChelseaB]

Hello proptradingshop,

If you have commented out all of the logic and the behavior persists, this may indicate that the computer itself may be experiencing hardware issues.
I would recommend obtaining a second computer to test on and compare performance.

Note, 2 years of minute bars is a lot of data and it would be expected this test would take a while depending on the single threaded performance and logical processor count on the computer.

[proptradingshop]

Hi Chelsea, I did mention in my responses that it is running faster on a machine with less resources. There is no hardware issues on the machine that is slow now as I have run diagnostics on memory and cpu and everything is as expected. I have benchmarks for how long it used to take versus how long it takes now so the amount of data is irrelevant. I want to specifically know what aspect is all of a sudden taking long.

Can you please provide me a mechanism to help identify why it is running quickly for 30-40 optimizations then hanging? I would like to know what it is doing when it is hanging.

Thank you

[NinjaTrader_ChelseaB]

Hello proptradingshop,

With all of the logic commented out and still reproducing the behavior, my only suspicion would be failing hardware on the computer.
(such as a failing memory or a failing harddrive)

It would not be an issue with the platform.

And since you have commented out the logic as directed, I don't suspect it is an issue with the script.

A hang typically always means the computer has run out of resources (ether memory or CPU).

Code like infinite logic loops or large amounts of data processing and calculations can cause this behavior, however this would not apply in this case as you have commented out all of the logic as directed and you are still able to reproduce.

Unfortunately, I would not have any further direction I can provide. This is why I have suggested testing another computer, which if another computer is not able to reproduce, this gives you a better sense that the computer hardware is the issue.

[proptradingshop]

There is no hardware issue. I am happy to do a debugging session and show my findings. I can guarantee that if I uninstall and reinstall a fresh NT8 instance on the machine the issue will be fixed. I have been through this several times and would like to avoid having to resort to this solution because we can not get better platform support to help identify performance bottlenecks. I appreciate the effort.