NT taking up all CPU and slow

[NinjaTrader_Dierk]

You'll find this in your traces file:
2009-07-07 12:59:27:328 Processor='Procesador Intel Pentium III Xeon'

[NinjaTrader_Dierk]

@JS999: Could you please provide a simple as possible scenario which would demonstrate the load issues you're experiencing. At best you sent your workspace files to "dierk AT ninjatrader DOT com".

I'd like to try 2 things:
a) see how your setup would perform on my ~3years old office machine
b) use your setup as reference for our internal NT7 performance tests

To clarify: performance gains in NT7 Ray or I reported earlier in different threads are related to management of historical data and have nothing to do with real-time data issues reported in this thread.

Thanks in advance

[JS999]

@JS999: Could you please provide a simple as possible scenario which would demonstrate the load issues you're experiencing. At best you sent your workspace files to "dierk AT ninjatrader DOT com".

I'd like to try 2 things:
a) see how your setup would perform on my ~3years old office machine
b) use your setup as reference for our internal NT7 performance tests

To clarify: performance gains in NT7 Ray or I reported earlier in different threads are related to management of historical data and have nothing to do with real-time data issues reported in this thread.

Thanks in advance

First of all, you aren't going to notice performance problems under normal market conditions. If you just open up NT and run it you will probably notice nothing wrong, since most of the time the market isn't moving very fast and the volume isn't that high.

Here is my suggestion:

Set up a fresh copy of NT on two similar machines side-by-side, connect both of them to Zen-Fire. On each of them, open up one single chart: a 1 minute chart on one of them, and a 2-Range chart on the other. Then watch the market right at the open on the US indices on the Russell (TF). You can also watch Crude Oil (CL) at around 2:25 to 2:30 EST during the closing minutes of the pit trade, soybeans (ZS) between 2:10 to 2:15 EST, or the US indices again from 3:57 to 4:00 EST. Try to watch before any big news releases as well - I believe this week we have unemployment claims tomorrow at 8:30 AM EST which should move the market, as well as GDP numbers on Friday at 8:30 AM EST which should do the same.

Observe that the range chart copy lags behind the minute chart copy. Make sure that you have Task Manager open so you can monitor the CPU usage for each process and see how it jumps. Time + sales is another thing that really eats up the CPU - in fact, I can't actually use it at all because during heavy volume periods it tends to lock up the UI interface a bit, at least on the demo. You may have to do this type of testing a few times, since you can't 100% predict exactly how much any given event is going to move the market... you should really be watching it during the heavy volume spikes that we sometimes get, usually on those big trend days like what we had a few times over the past 2 weeks.

The problem with giving you my setup is that it's already about as optimized as I can get it... I can send it to you, but I don't think you should use it as reference - you should be using a standard default fresh copy of NT as your reference, since that is what most people install. The main trading copy of NT that I have using the 10 second and 2 minute charts is more or less OK at this point (I think, I still have to test it on some heavy volume times which has been tough the past few days since things have been dead), but range charts lag for sure on heavy volume (and spike the CPU to 100%). Try it out for yourself and run some tests, you'll see what I am talking about.

[NinjaTrader_Dierk]

We can tweak the sim feed to produce any load we'd like. We are not reliant on actual market conditions for our internal performance tests.

Please feel free to send any setup which you think makes sense as per below for further investigation.

Thanks

[JS999]

We can tweak the sim feed to produce any load we'd like. We are not reliant on actual market conditions for our internal performance tests.

Please feel free to send any setup which you think makes sense as per below for further investigation.

Thanks

Ok, but if you think about it, wouldn't it just confuse things for me to send you my setup at this time? I mean, if I am claiming that the problems are internal to NT 6.5 and have nothing to do with my setup, then what is the point of seeing it? You guys should just run a range chart on your setup and make it run under heavy load, and compare that to exactly the same thing running on another copy with only a 1 minute chart. If you see the slowdown, then you'll realize that the problem is NT and my setup won't matter. If on the other hand you see perfectly normal performance, then obviously there is something I am doing (which I highly doubt, since I have been trying this out for months).

If I just sent you my setup and you tested that, what would it prove? Suppose it lags - great. But then you haven't narrowed down the problem at all, since you won't be testing it on a completely clean copy, and won't be able to definitely say that the range charts are the problem. If you see the lag with your own clean copy of NT then you'll know it's the code. Does that make sense?

[NinjaTrader_Dierk]

We are not experiencing anything similar to your reports in our internal tests so far. Please let us know as you would be willing to provide the information (workspace) we need to isolate what you experience.

Thanks

[Mike_32]

Well I can tell you, I am currently working on changing broker. In fact I might just scrap NT altogether and use something else. The CPU spikes on an intel 3ghz quad core 1333fsb with 4gig 1333mhz DDR3 ram are the same as on a pentium 4 2.8ghz with 2gb of 533 ram. You would think it would be better, but it seems to just be the norm with NT. I don't believe its the ZF feed anymore I believe it to be NT itself because ZenFire is just a price ticker, its NT that makes it into bars. And it causes the bars to get delayed, sometimes by 50 seconds in the middle of the volatile US session which causes losses on getting out in time or getting filled on time. I thought this was manipulation but since with discussions with the broker have found its due to NT using to much CPU ALL the time. Even with no custom indicators installed, no templates, no workspaces, only 1 chart open, it still regularly spikes the CPU usage and slows down the displaying of bars.

[NinjaTrader_Ray]

Guys,

There are many variables that can impact the client side performance.

- Our software
- Your hardware
- Your network/ISP
- Broker technology and network

We run internal tests where we flood our software with high frequency number of ticks to review load and efficiency. These tests meet our expectations.

Hardware, I will just assume you run adequate hardware.

Our software, we provide options to process our UI's on each coming tick or based on timers. Most software applications don't provide this granularity and most drive their UI's off of timers since otherwise their UI's would get overloaded.

Most market data vendors and/or brokerage technologies throttle/filter their data so you don't get every tick update but rather a several snapshots per second. They do this for several reasons. To reduce the amount of data that is being sent over the wire, to control the frequency of data updates sent through the wire. All of which has a positive impact on performance and can save them costs.

In situations where you are connected to an unfiltered feed, if you have a poor connection to the internet, you can easily and quickly build data buffers on the server side. This simply means that the sheer amount of data that is trying to be sent to you is queued since your connection can't handle it for some reason. If this happens, you get data bursting as large chunks of lagging data make it through in a split second which will immediately overwhelm NinjaTrader since it has to process all of this data in a split second. In the event that a buffer is too large or its duration is too long, the network engineers at Zen-Fire will actual flag this and action is taken to inform the customer. Most of the time, these buffers clear out quickly as they generally happen in volatile times.

In contrast, vendors who throttle their data generally don't see server side buffering since they limit the amount of data that is being sent hence, you get a even flow of data that will not overwhelm any connected client.

I am not suggesting this the issue but merely trying to provide some ideas as to why you might see some locking/freezing during volatile periods.

That being said, we have made improvements in memory consumption and CPU usage across the board in NT7.

[salakazam]

You'll find this in your traces file:
2009-07-07 12:59:27:328 Processor='Procesador Intel Pentium III Xeon'

I have not clue why this happen, because my processor is an Intel(R) Core(TM)2 Duo CPU T9300 @2.50GHz (open my pC to see). But no problem, because I solved my issue.

Yes, is not a problem with my chip, is with some properties of my nView Desktop Manager. You did right when pointed the icon, not the icon but I disconnect all properties and now is working fine. I test in other PC with same problem, deactivate this properties and works again.

Maybe other user with this problem can solve making the same (see "control panel" "nView Desktop Manager", deactivate properties in desktop manager tab).

If any user can find exactly which property crash with NT, please post here.

Thanks for your advices. Especialy to poin the Nvidia issue, because all the other tips are unnecessary, I´m working now with all my workspaces, indicators, calculate on bar close to false, etc...

Regards.

[JS999]

Guys,

There are many variables that can impact the client side performance.

- Our software
- Your hardware
- Your network/ISP
- Broker technology and network

We run internal tests where we flood our software with high frequency number of ticks to review load and efficiency. These tests meet our expectations.

Can you be more specific by exactly what you mean when you say "meets your expectations"? I'd like to know what hardware you ran on, how many charts you had open, if you had a time and sales window open, and/or a DOM, what was your lookback period, what indicators you had on there, whether your chart refresh interval was set to 0 or to some other timer value, and so on. Saying that "it meets your expectations" doesn't tell us anything, because your expectations might not be my expectations as a scalper. I want to know exactly what you ran that you think is acceptable.

Our software, we provide options to process our UI's on each coming tick or based on timers. Most software applications don't provide this granularity and most drive their UI's off of timers since otherwise their UI's would get overloaded.

On this note, I have a feeling that the slowdowns people are experiencing may have to do with something in your screen redraw routines. I have noticed a relatively significant performance improvement when I removed all the horizontal grid lines from my charts (I used to have horizontal grid lines specified with a value of 1, meaning it would put a line at every tick). Is it possible that this is causing slowdowns when the market is running because it attempts to redraw all of these lines at every tick?

In situations where you are connected to an unfiltered feed, if you have a poor connection to the internet, you can easily and quickly build data buffers on the server side. This simply means that the sheer amount of data that is trying to be sent to you is queued since your connection can't handle it for some reason. If this happens, you get data bursting as large chunks of lagging data make it through in a split second which will immediately overwhelm NinjaTrader since it has to process all of this data in a split second. In the event that a buffer is too large or its duration is too long, the network engineers at Zen-Fire will actual flag this and action is taken to inform the customer. Most of the time, these buffers clear out quickly as they generally happen in volatile times.

So what you are saying is that large chunks of unfiltered data streamed into NinjaTrader can overwhelm the system. In this example you are citing that happening as a result of lagging buffered data which gets sent all at once in a burst, and that's fine.... but sometimes you just legitimately have large volumes of unfiltered data coming in during real-time because that's what the market is doing. Not every data burst is buffering, sometimes the market is just going crazy and there are a lot of trades to process, correct?

So whether it gets buffered or not is not the issue - the issue is what happens when NT receives a large chunk of unfiltered data, whether it was delayed or not. If you can't handle an unfiltered data feed when it bursts large amounts of data, then this is a problem and you shouldn't be advertising this software as being capable of handling an unfiltered feed.

In contrast, vendors who throttle their data generally don't see server side buffering since they limit the amount of data that is being sent hence, you get a even flow of data that will not overwhelm any connected client.

Agreed, but the bottom line is that this piece of software is being advertised as working with an unfiltered data stream like Zen-Fire, so it needs to work properly with it. Are you saying that it's impossible to write a piece of software that can handle these unfiltered data bursts properly without lagging, even on the most powerful machines we have today? If so, then Zen-Fire needs to be changed somehow, because they are providing data that is impossible for vendors to work with properly. If not, then the problem is on your end. The bottom line is that the system has to work on first install, with typical settings, without significant delays, whether the market is running or not. Saying that other vendors don't provide unfiltered data doesn't change the fact that NT is trying to run with it and getting overwhelmed. The final result is the thing that matters here, and whether it's your fault or Zen-Fire's fault you guys need to work with it to get it resolved.

I am not suggesting this the issue but merely trying to provide some ideas as to why you might see some locking/freezing during volatile periods.

I am almost 100% certain that the locking and freezing has to do with screen drawing... the system is perfectly able to handle things without any lag when you just have a time and sales window and DOM open... it's when you add more and more charts that things get bad under heavy load.

That being said, we have made improvements in memory consumption and CPU usage across the board in NT7.

Hopefully you have spent time trying to optimize those routines, especially the ones that draw constant volume/tick/range bars, as these tend to lag minute bars in terms of performance.

[eDanny]

"I am almost 100% certain that the locking and freezing has to do with screen drawing... the system is perfectly able to handle things without any lag when you just have a time and sales window and DOM open... it's when you add more and more charts that things get bad under heavy load. "

I think you probably are 100% wrong on this point. Modern video cards are so fast when it comes to 2D drawing, I don't think you could significantly slow them down. The problem with multiple charts and such during heavy load is because of all the calculation being done by many indicators BEFORE the charts get re-drawn. Just my 2 cents.

[JS999]

Ray, I don't know how your software is set up, but there should be the absolute minimum necessary processing being done before the screen redraw happens, since that is what the user sees and uses to make decisions. Are you currently writing to disk and drawing to the screen all in one thread? If so, you could change it so that after screen refresh (which takes priority and happens first), data is placed into a memory buffer somewhere where another thread polls it and periodically flushes it to the HD. I don't know if you're already doing something like this or not, but it's a suggestion...

[JS999]

"I am almost 100% certain that the locking and freezing has to do with screen drawing... the system is perfectly able to handle things without any lag when you just have a time and sales window and DOM open... it's when you add more and more charts that things get bad under heavy load. "

I think you probably are 100% wrong on this point. Modern video cards are so fast when it comes to 2D drawing, I don't think you could significantly slow them down. The problem with multiple charts and such during heavy load is because of all the calculation being done by many indicators BEFORE the charts get re-drawn. Just my 2 cents.

Maybe - and that was my initial impression too, which is why I never thought of removing the horizontal lines. All I know is that removing them improved performance... something seems to be happening there.

[eDanny]

Again, it is not the video card lagging from drawing a few lines. The chart is also calculated and readjusted and more horizontal lines could mean more cpu usage also. Just guessing on this point but it makes sense since the chart draw routines are, of course, just more code. Notice during freezes the charts are not half drawn or misshapen in any way. They just stop, waiting for the video card draw routines which are waiting on info from the backed up data.

[JS999]

Again, it is not the video card lagging from drawing a few lines. The chart is also calculated and readjusted and more horizontal lines could mean more cpu usage also. Just guessing on this point but it makes sense since the chart draw routines are, of course, just more code.

I wasn't suggesting that the video card itself was responsible for the lag... that would be ridiculous in this day and age, considering what modern cards can do. The lag is obviously CPU related, so maybe NT is doing something prior to redrawing those lines that is taking up the time. I can't see their code so I have no idea what they could be doing...