Limits of Optimizer Multi-Threading

[kevinenergy]

Are there any limits on the Multi-Threading capabilities of the Ninjatrader Optimizer? If I was to upgrade my CPU to a 32 Core, Like the new Ryzen Threadripper 2990WX will the optimizer be able to make use of all of the cores; or is there a limit on the number of threads that can be utilized?

Has any benchmarking been done at massively multi-threaded optimization? Does memory become a bottleneck or any thread starvation due to locked resources etc?

Just wanted to check if anyone has any info on this before I spend $2k on a new CPU.

[NinjaTrader_Jim]

Hello kevinenergy,

Thanks for your post.

NinjaTrader uses a thread pool for processing historical data and creates threads in line with the number of logical cores on your CPU. Multiple threads helps greatly with optimizations. I am not aware of any noted profiling done and I am not aware of particular bottlenecks on a large scale.

I do have similar hardware on my home PC. I'll update this post with some comparisons using a test script and share my results when I get a chance.

Please let me know if there is anything we can do to assist.

[kevinenergy]

That would be really helpful - thanks you Jim! It doesn't need to be a comprehensive test. But what I'd like to know is: if you switch the performance Tab on Task Manager to "Logical Processors" as below; is the Optimizer still pushing each one to 90% with a huge number of threads running. I'm concerned that maybe the north bridge will become a bottleneck with so many threads pulling in data from RAM which would be the likely cause if the CPU utilization is lower.

Thanks again for offering to test this for me!! Really appreciated!

[NinjaTrader_Jim]

Hello kevinenergy,

I ran a quick optimization this morning using SampleMACrossover with Fast 1;200;1 and Slow 1;200;1 against 1 minute data since the beginning of the year on ES. Really just wanted to max out CPU usage with a simple script so number of iterations could be focused on (since you are inquiring about thread usage.) All my logical cores are being utilized pretty well, and max CPU usage does even out between 70% and 90%. I'm sure a more complex strategy will keep overall utilization at a higher percent. Memory is also being heavily utilized, but I do have other applications running.

Here's a screenshot:



Please let us know if there is anything else we can do to help.

[kevinenergy]

Jim! Thank you for taking the time to do this. Looks like a good result to me. How fast is your RAM in that rig if you don't mind me asking?

[spottysallrite]

Why not do this in the Cloud, such as Google or AWS? I do extensive optimizations and gotten very good results. GC offers 96 cores.

[kevinenergy]

I looked into it. But $4.50/hr quickly adds up. That's $800/week.

Why not do this in the Cloud, such as Google or AWS? I do extensive optimizations and gotten very good results. GC offers 96 cores.

[NinjaTrader_Jim]

Hello kevinenergy,

The RAM is earlier DDR4 RAM. I don't remember the speeds off the top of my head, and I would like to leave this thread in the direction it has taken rather than talk about PC specs.

If there is something else we can assist with, please feel free to open a new thread.

[kevinenergy]

Reason I was asking is I wondered if I could squeeze the Optimizer up to MAX core utilization with faster RAM. Earlier DDR4 so probably 2800-3000Mhz. There's 4200Mhz availabe now so hopefully that will squeeze even more out of it. Wasn't just generally discussing PC specs, don't worry haha. I'm not THAT much of a geek.

I will update this thread when I get my new PC and have done some benchmarking.

[spottysallrite]

The optimal number of cores for me seems to be around 16, and that's using 5 years of single-tick data on the ES, and optimizing 27 params with 3 values each.

How about spending something less than $2k and overclocking and cooling using an isopropyl alcohol / dry ice bath?

[Oracletrades]

I do realize this thread is super old, however I do have a setup with AMD CPU Ryzen 5 3600 and my strategy with nearly 14 parameters maxes out at around 35% which is terribly poor. Below is a VM but it's on a local machine with full pass through (even GPU of the host). When I run it on another PC with AMD Ryzen 9 3900x, same problem..

Just in case anybody else has similar issues. I don't seem to be able to find a solution. Intel CPUs are much faster with NT8 from what I can see.. (I have an intel workstation which maxes out at 98%..).

[NinjaTrader_Jim]

Hello Oracletrades,

We do see the the workload is being disturbed across all cores. I would suggest performing a very CPU intensive optimization with the SampleMACrossover strategy like I have in post #4 for an apples to apples test, however. If there are not as many optimization iterations and if the strategy is not very complex, I may not expect the optimizer to hit full CPU utilization.

If you perform the same test I do in post #4 with the SampleMACrossover strategy on ES 1 minute data from 2020 to 2021, do you see similar utilization as I am after letting the test run for some time?

After performing the test above, do you see differences through your VM and on your PC that suggest the VM is throttled?

I look forward to hearing from you.

[Oracletrades]

We're discussing this matter here, apologies for posting twice, I thought the other thread may be "dead"..

https://ninjatrader.com/support/forum/forum/ninjatrader-8/platform-technical-support-aa/1048913-limits-of-optimizer-multi-threading#post1137190