Only 32% CPU usage

[crimsonite]

http://www.ninjatrader.com/support/forum/showpost.php?p=198816&postcount=7

I'm having the exact same problem as this guy. Is NT7 limited to a certain level of processing power?

[sledge]

http://www.ninjatrader.com/support/f...16&postcount=7

I'm having the exact same problem as this guy. Is NT7 limited to a certain level of processing power?

Post your windows system score picture.

[crimsonite]

7.9 across the board.

What's the bottom line? It does use all 32 cores evenly, but sticks strict at 32-33% usage. Will I never be able to go past this limit and somewhat compete with institutional firms on a piece of software I'm paying $55/mo for?

Would like to hear a clarification, insight, or a note that it's being worked on. I really don't mind the bugs in Ninja, I completely understand and support the efforts and mantra of having a value program for retailers. Something like not utilitizing expensive hardware eeks on the unacceptable side, especially for software that blows esoteric autodesk or adobe out of the water in terms of price. Frankly I'm getting the feeling that Ninja is more spaghetti code and advertisement than a strong effort to (example) get to multicores - something that took awhile back then.

Much appreciated...:cool... C

[crimsonite]

Hello sledge... I thought I'd share some updates with this issue.

1.) I switched to just one processor, same one the other guy is having a problem with - only runs about 33% of the processor.
2.) I then connected to a supercomputer I have access to and partitioned 668 cores, bringing the total to 700 cores. This increased Ninja to about 45% CPU usage .
3.) I switched down to 296 cores - Ninja uses around 39% CPU.

I have no idea what the correlation is, if any. I thought of other potential bottlenecks, but don't think any of them are the issue:

1.) Disk: The disk usage is not at its max
2.) RAM I/O: It is not at it's max
3.) Network: (Obviously) not the bottleneck

I've also successfully run other intensive disk, RAM and CPU programs, such as WinRAR and Intel Burn Test. All achieved 100% CPU.

It looks like it's in the software. I think the issue might be Ninja's method of clearing out the RAM very often. A very easy fix for this would be to allow users to adjust the RAM cleanup in the options - most Ninja users have lots of available RAM (this isn't 512mb RAM era anymore. My laptop is even running 8gb, and Ninja is at most running 1.2gb. On the cluster, I'm running a little under 72gb, and Ninja is using 1.6gb max.

This may seem like an esoteric problem, but other guys (link in the OP) are having the same problem, even with their high end single core workstations. It looks like an easy fix, and with all of the resources available to Ninja professional developers, I think it can certainly be done.

In terms of dollar-for-dollar value, Ninja is by far the most comprehensive and also intuitively designed finance software I've ever used, including Bloomberg. Have you guys thought of running donations or a program where Ninja is given away for free to finance students? After all, this is a sector with allot of rich guys . Ninja could just as easily be the Microsoft of high value retail platforms for individuals!

[sledge]

I may have missed it, but what are you doing exactly in NT when examining CPU speed?

[crimsonite]

I'm doing optimizations.

I noticed that Ninja only uses all cores for CPU intensive tasks... could it be that it's detecting a huge computer, and confusing a huge task as an easy task relative to processing power (you follow?) :confusedL

[NinjaTrader_AdamP]

Crimsonite,

CPU usage is dependent on many things. Its possible the optimization task simply doesn't need 100 % of the capacity a processor has on average per cycle, or a scheduler could be limiting this, as could the method being used. CPU usage total isn't necessarily the best metric for determining performance, it would be better to use total time for a job as well its average CPU usage.

CPU usage means that a particular process used X percent of the CPU's total available calculations per cycle on average at that time. A more efficient method would actually use less percentage of the processor per cycle, so long as the total time it takes to complete all necessary calculations is less than or equal to the previous method you tested.

An example :

Suppose we have 2 methods, named "Method A" and "Method B" respectively. Suppose each is a different method for producing the same output from a set of inputs. Method A produces its output using 30 percent CPU usage on average in 2 hours, Method B produces its output using 20 percent CPU usage on average in 2 hours. Obviously Method B is a more efficient method since it produced the same results in the same amount of time as Method A using less of the processor capacity on average.

1.) I switched to just one processor, same one the other guy is having a problem with - only runs about 33% of the processor.

This isn't necessarily an issue. Please see the following article :

http://blogs.technet.com/b/winserverperformance/archive/2009/08/06/interpreting-cpu-utilization-for-performance-analysis.aspx

2.) I then connected to a supercomputer I have access to and partitioned 668 cores, bringing the total to 700 cores. This increased Ninja to about 45% CPU usage .

3.) I switched down to 296 cores - Ninja uses around 39% CPU.

Please keep in mind the many processors are communicating with the main system, perhaps needing to wait for one task to complete before one can communicate its results. Keeping all processors you are using in-sync requires additional resources the more processors you are using. Generally the trade off is that you get more performance overall since to complete X amount of work will take much less time.

Also, I am not familiar with what you mean when you said that you "partitioned" these processors. In my experience with super computers we used a scheduler to request X number of processors, and Y amount of RAM for Z amount of time when we submitted a job, then the job would run, and complete when our code finished running.

I have no idea what the correlation is, if any. I thought of other potential bottlenecks, but don't think any of them are the issue:

1.) Disk: The disk usage is not at its max
2.) RAM I/O: It is not at it's max
3.) Network: (Obviously) not the bottleneck

1) NinjaTrader should mostly be using RAM over a hard drive, so disk usage isn't necessarily even an issue here.

2) RAM is where the majority of data that is being used by real-time processes is stored. Please keep in mind that .NET has a cap on how much RAM can be used. The input/output speed of RAM would be set by the hardware, as the CPU works directly with RAM.

3) A network can most definitely be a bottleneck for clusters since each node must communicate.

What's the bottom line? It does use all 32 cores evenly, but sticks strict at 32-33% usage. Will I never be able to go past this limit and somewhat compete with institutional firms on a piece of software I'm paying $55/mo for?

Would like to hear a clarification, insight, or a note that it's being worked on. I really don't mind the bugs in Ninja, I completely understand and support the efforts and mantra of having a value program for retailers. Something like not utilitizing expensive hardware eeks on the unacceptable side, especially for software that blows esoteric autodesk or adobe out of the water in terms of price. Frankly I'm getting the feeling that Ninja is more spaghetti code and advertisement than a strong effort to (example) get to multicores - something that took awhile back then.

I hope I have demonstrated there really is no "limit" related to usage of resources in NinjaTrader as far as "CPU usage" is concerned. Our optimization engine does use multiple cores, but using "more" of a core can be less efficient depending on the time the entire job takes to complete.

Institutional firms most likely engineer all their own software or use existing things like Unix, Torque, etc. heavily customized with a computer cluster. They also have more direct access to markets electronically. This is something NinjaTrader could not provide for you, as we are not a data provider nor a trading institution. We provide software that is easy to use for a trader, yet powerful in its customization for advanced programmers while not sacrificing the ease of use for the average individual. NinjaTrader is built for everyone.

Please let me know if I may assist further.

[danluc]

I think it's not a new information, but, in my case, the optimizations with any multi timeframe feature in the strategy (like a simple EMA rising from another timeframe), will result in 10 to 12% of CPU performance (it needs almost two days to optimize in a core i7 2600K, 16 gb ram) and, without the indicators from another timeframe is running about 60 to 70%.

Ninja support told me this is because the nature of Windows.

[chancehero]

why don't you just write another program which gets feed your data input , does the optimization (were you can easily make use of all your cores) and spits it back out to the ninja trader code?

looks like someone needs good old Fortran :P

[crimsonite]

Thoughts on Ninjatrader & Strategy Analyzer

Ninjatrader AdamP,

I get the impression that you're disingenuously sidestepping the problem. Let me be very clear:

Ninjatrader Strategy Analyzer does not work to use all of the processor.




There is no computer hardware problem like you're implying. Oracle does not have bottle necks on their network. 32% CPU is NOT the optimal CPU usage (your link doesn't address this at all). None of the computer resources are fully loaded, including RAM, CPU, Disk (despite how much disk IO resources Ninja seems to require) and Network. I kind of feel the same way as I did when I read the first customer service responses in the false backtest data thread, saying that there was no issue.

Sadly I don't expect this issue to be corrected, because all of the other (and lesser) bugs have been carelessly "forgotten" about. Personally I still don't know if Ninja is giving me false backtest data or not, because nobody is being clear about whether the ask issue that received significant attention a few weeks ago has been addressed or not.

Most likely, this is a very easy problem to fix. Is there something in Ninja's source code that doesn't trigger a 100% CPU usage unless a strong process is executed? This would be logical because Ninja would view lots of CPU power as 'enough' and not execute full 100% CPU.

On the contrary, I'm excited about NT7 growing to dominate the value retail software scene... it can be used to trigger students' interest in finance, and assist the retail home trader looking to meet his trading goals all the way. I still don't think any value platform under a few hundred bucks a month is as intuitive as NInja... it's just things have to be fixed. Spaghetti code isn't a big deal if special attention is paid to working out bugs so that the software works on almost all customers' hardware (that includes us with high end equipment). Good luck, and I look forward to reading about the next release of Ninja soon.

[NinjaTrader_Brett]

Hello,

Thanks for the feedback.

We are always looking to make performance improvements in our next major release of the software.

Unfortunately I could not give an expected release for this other then it is still some time down the road. However if you have optimized the hardware correctly as you say you have then you are achieving the fastest optimize that you possibly can with NinjaTrader at this time.

-Brett

[koganam]

If I may be so bold, I do not understand why one would even want anything to use 100% CPU in a multitasking operating system. The whole idea of a preemptive OS scheduler is to prevent any process from hogging the processor, precisely because other programs will then be starved of resources.

Window's scheduler, while a vast improvement over what came before, does not do a very good job of deciding that with very few other programs running, it should dynamically adjust priorities to give more time to the process that seems to need it most.

However, you can always manually adjust the priority of any process. Just be wary if you do. In my days, I have caused many a crash by adjusting scheduler priorities. Then again, I belong to the computing school that says: "If your computer ain't broke, then you have not installed enough beta programs!"

[sledge]

I think it was suggested below..

A real test would be to get a stopwatch, and run his optimization on 1 cpu, then 2, ... to how many ever cores he can get.

I've never ran that PI cpu test program, but does that eat all of the CPU?

I haven't done optimizations in NT and don't really plan to.

I know my video editing eats up CPU.

If I may be so bold, I do not understand why one would even want anything to use 100% CPU in a multitasking operating system. The whole idea of a preemptive OS scheduler is to prevent any process from hogging the processor, precisely because other programs will then be starved of resources.

Window's scheduler, while a vast improvement over what came before, does not do a very good job of deciding that with very few other programs running, it should dynamically adjust priorities to give more time to the process that seems to need it most.

However, you can always manually adjust the priority of any process. Just be wary if you do. In my days, I have caused many a crash by adjusting scheduler priorities. Then again, I belong to the computing school that says: "If your computer aint'broke, then you have not installed enough beta programs!"

[crimsonite]

Hello,

Thanks for the feedback.

We are always looking to make performance improvements in our next major release of the software.

Unfortunately I could not give an expected release for this other then it is still some time down the road. However if you have optimized the hardware correctly as you say you have then you are achieving the fastest optimize that you possibly can with NinjaTrader at this time.

-Brett

Yes, I mentioned I don't really expect it, but I'll nonetheless be looking forward to the next release (regardless of what is done to be improved). With that said, I have a strong suspicion that it would not be a difficult thing to correct. Many are using high performance CPU's, with Dell's at Costco with 16gb RAM going for under a couple thousand bucks.

If I may be so bold, I do not understand why one would even want anything to use 100% CPU in a multitasking operating system. The whole idea of a preemptive OS scheduler is to prevent any process from hogging the processor, precisely because other programs will then be starved of resources.

Window's scheduler, while a vast improvement over what came before, does not do a very good job of deciding that with very few other programs running, it should dynamically adjust priorities to give more time to the process that seems to need it most.

However, you can always manually adjust the priority of any process. Just be wary if you do. In my days, I have caused many a crash by adjusting scheduler priorities. Then again, I belong to the computing school that says: "If your computer ain't broke, then you have not installed enough beta programs!"

Usually I will run Ninja at "below normal", meaning Windows will run it at 100%, but give the right away to other programs. This allows it to be as fast as possible, while leaving the OS and other software as fluid as possible.00

I think it was suggested below..

A real test would be to get a stopwatch, and run his optimization on 1 cpu, then 2, ... to how many ever cores he can get.

I've never ran that PI cpu test program, but does that eat all of the CPU?

I haven't done optimizations in NT and don't really plan to.

I know my video editing eats up CPU.

It runs on all cores evenly... and yes, it won't increase core usage if I turn off a core or several.

[sledge]

But does it run faster or not?

I'm only at 2 cores and don't have access to more yet.

Yes, I mentioned I don't really expect it, but I'll nonetheless be looking forward to the next release (regardless of what is done to be improved). With that said, I have a strong suspicion that it would not be a difficult thing to correct. Many are using high performance CPU's, with Dell's at Costco with 16gb RAM going for under a couple thousand bucks.




Usually I will run Ninja at "below normal", meaning Windows will run it at 100%, but give the right away to other programs. This allows it to be as fast as possible, while leaving the OS and other software as fluid as possible.00


It runs on all cores evenly... and yes, it won't increase core usage if I turn off a core or several.