High Volatility Environment and NT Performance

[aviat72]

During this week's high volatility environment, I noticed that NT charts would fall behind the market significantly. What was peculiar was that some instruments were current while others were more than two minutes behind. I have a IQFeed tick true feed. They also send market internals feed which is sampled at 1second interval.

Today NT kept pace with the market internals. However for the instrument to trade, TF in this case, NT charts fell 2 minutes behind the market! Please see attached picture. This was true not only of the instruments but also the indicators plotted on them. This clearly seems to have to do with speed of the incoming data. The tick-true data (TF) fell behind but the 1 second input stream (internals) kept pace with the market. I can understand the behavior during the Thursday's plunge, but Friday's 10:30AM Dive was not that abnormal.

Though this could be an IQFeed issue (i.e. their tick feed falling behind) it is unlikely since they are reputed to be one of the most efficient data-feeds out there and no one else has reported such delays with IQFeed.

Clearly any strategy running live would have been screwed by this lag to the market.

I wanted to get a sense of what I can do to minimize the lag.

(1) Does creating Tick and Range charts contribute to the processing slowdown? i.e. will a 1 or 5 second chart be preferable to a 200 tick chart?

(2) Also is it possible for NT to warn the user that it is falling behind in processing real-time data? I can switch to the IB (Interactive Broker) sampled feed when market gets volatile since it does not increase the throughput which a tick-true feed does.

(3) Ideally an option to force NT to go into a sampled tick mode when it starts falling behind would be great. Alternatively, an option to chose the data-feed provider in high volatility environment will also help.

DefaultProvider = Provider#1 (IQfeed in my case)
HighVolProvider = Provider#2 (IB in my case)

If NT detects that it is falling behind Real Time by more than a few seconds, it automatically switches to the HighVolProvider's data-feed.

(4) In general greater user control on which data-provider to use for which instrument will really help. The current system where it tries them in the connection order is restrictive. In my case I have Real Time sub with DTN for Futures and not for equities; DTN sends a delayed feed for them. However NT will still chose that feed over the IB feed which is real-time (but sampled) since DTN is my first connection.

A connection provider choice at the instrument level will be great and should be easy to implement.


I expect markets to be very volatile in the next few months and these issues need attention.

[tony98]

Live Strategies

I run live strategies in Ninja 6.5 and got royally screwed on Thursday(May 6 2:40 pm market tank) due to NT freezing up!

The system was running on a new i-core 7 860(win 7 64 bit 8GB RAM) that I bought in preparation for ninja 7. I was hoping that NT 7 would have been able to handle this properly.

My NT was connected to zenfire, TT and IB at the time - a multibroker license. I was able to do some damage control with IB because of the TWS interface, however zenfire and TT required NT to be running. A bad situation.......

I would like to hear more from others about their experiences and from NT about how to handle/configure systems optimally for NT 6.5 and 7.

Thanks.

Tony

[aviat72]

Tony

In general I always keep a window which allows manual trades open irrespective of what software I am using to manage trades.

There is absolutely no guarantee that the software you are using will crash or hang at the worst possible time.


Going forward I am going to set very lose stop-limits price on my long positions or just use stop market with IB handling the execution along with GLOBEX rules.

[NinjaTrader_Austin]

aviat72, if all the other instruments were being updated correctly as indicated by your screenshot, that means NinjaTrader was keeping up with the data being fed to it. The data feed is most likely the culprit here. I will have someone get back to you on Monday with more information.

[tony98]

Aviat72,

I use IB interface TWS, but zenfire and Trading Technologies feed have no interface from my brokers. The strategies I employ release bracket orders(stop and profit) after being filled. In this case my entry orders were out and NT crashed. I was subsequently filled and had no protection until I could get NT unfrozen and up again.

NinjaTrader Austin,

Can you comment on the experiences of others with NT during this high speed market period? How can I do more to optimize my system or improve its performance?

Thanks.
Tony

[NinjaTrader_Austin]

Tony, I'm a full-time engineering student in addition to working for NT and as such I don't trade and haven't really heard any experiences lately. There is a section in the NT7 help guide about how to increase your performance. It can be found at Help -> Help -> Contents -> Configuration -> Performance tips.

[aviat72]

Any update on this issue? Can development comment on what is happening under the hood?

[NinjaTrader_JoshP]

Firstly what kind of computer are you using?

Have you confirmed that it is actually your computer maxing out on resources as opposed to simply the data not coming across the line? When multiple instruments are being used and one is not having any issue this is more of an indication of a data feed issue. Reason I say this is because if NT was actually hitting the wall performance wise you would see it across the board.

Another possibility is to check your network congestion. If your internet bandwidth is being used up at that time you may not be getting updates to your prices as you would normally expect.

Unfortunately there is no way NT can detect feeds being slow or fast. NT acts like a radio receiver. If data is received, data will be charted. If no data comes across the line, NT has no way of knowing if data should have come across the line or not.

Thank you for your suggestions on control over the feed uses.

[aviat72]

Josh:

Thanks for your reply.

There is a clear symptom that the panels which have second based instruments are current; while the panel based on tick based instrument fall behind.

What I would like are more specifics about where the problem may be occurring based on the obvious clue that some feeds are current (time based) while some are not (tick based).

There are two broad scenarios:

(1) Data Feed falls behind for tick based instruments (i.e. the TF Future) while the data-feed is current for second bars. NT processes the feed as it comes and displays that. This would mean that the tick based feed fell behind almost 2 minutes!

(2) Since the number of ticks is huge NT hits pure computer horse-power issues. Since there are tick based indicators running, and ticks coming through at hundreds per second, it just takes more time to process everything.

Based on how NT is implemented under the hood what is more likely especially given the fact that some charts were current? Development might be able to shed more light on this.

[aviat72]

On a different note regarding NT's behavior in high tick-rate environment.

This is a straight-forward Q through-put issue and NT can detect if the current tick being processed is not within a second or two of real-time.

If there is a data-feed issue, then a test to check whether the tick-time Time[0] is behind the Time.Now (real PC time) can detect the slow-down.

If there are horsepower issues, the size of the internal Qs to store tick values will start increasing; or the time gap between the first and the last member of the tick Q which is waiting to be processed will go up.

However independent of the root-cause, once NT detects performance issues it can potentially react to it. If it is a data-feed issue it can use the fall-back feed; if it is a horsepower issue it can start using a sampling mechanism (essentially skip ticks).

Given Euro-Tarp etc we are likely to be very volatile going forward. Being able to react to it is important for automated trading systems.

[NinjaTrader_JoshP]

1. Not sure I follow your distinction. Second charts and tick charts both run off of tick events. When a tick comes in it will update both charts.

2. As mentioned, this does not look to be performance related. If some instruments are completely loaded and your TF is not, this is indicative of you simply not receiving data from your provider and not NT performance. If NT was really running into issues with performance you would see the whole chart lag and become choppy. If you brought up the Task Manager you would see maxed out CPU. I do not suspect this is your case. If your CPU is not pegged to max, NT is having no issues processing all of the incoming tick data.

I suggest you check your network connection as this sounds to me as if it is network congestion slowing down packets from traveling through your internet line. Or it could even simply be that was what the data provider provided at that point in time.

There is no check of Time[0] versus PC time possible. Many data providers do not push their ticks timestamped and as such they are timestamped by the PC clock.

[MXASJ]

I'm not smart enough to know if this helps or hinders, but I spent today coding this and it had some interesting results during news times and market open. NT7b15 compile.

I'm running it off the Strategies tab (not an open chart) using ES 1 sec as the dataseries.

Open to input as to whether this is relevant, and if it is, how to make it better.

[NinjaTrader_JoshP]

MXASJ,

local = DateTime.Now;
exchange = Time[0];

Not relevant for many data providers as Time[0] would = DateTime.Now of your incoming tick. For tests on data providers that actually do push their own timestamps, PC clock sync is of utmost importance.

[aviat72]

1. Not sure I follow your distinction. Second charts and tick charts both run off of tick events. When a tick comes in it will update both charts.

Josh

As I mentioned multiple times in the posts above, the instruments which are current are those where the data-provider is updating once a second; i.e. the tick occurs once every second. These are market internals and they are no trade ticks involved; they are sampled.

The behavior where the tick based data is falling behind while the second (time) based data is current is peculiar.

Think of the data as a Q. All the data from the exchange is being pumped into the Q by IQFeed in sequence. Some of the data is sent once every second, and some is being sent when the tick occurs.

It would be very odd if the tick based data falls behind 2 minutes the second based data simply because of network congestion. If network congestion was an issue the second based data would also be delayed. And even in high tick environments the data rate is in the order of 10s or 100s of KBs hardly something which will be causing network congestion.

This points to a processing bottleneck somewhere.

It could be at the data-vendors servers which just get overwhelmed by the amount of tick data. So a lot of time elapsed before the tick could be sent
OR:
It could be at the NT/Local PC level which runs out of horsepower and is just not able to keep up with the number of ticks and the numerous tick based indicators.

There is no check of Time[0] versus PC time possible. Many data providers do not push their ticks timestamped and as such they are timestamped by the PC clock.

For those which do it would certainly help.

If nothing else a check which compares instrument A and against instrument B. In this case the market internals were 2 minutes ahead of the future instrument which tracked them. As the charts show, NT clearly knew that the TF data it was processing was 2 minutes behind the internals data which was current.

If NT was just time-stamping based on the time it received the data, and the data was late, the TF chart would still be showing current (which would be grossly inaccurate!).

----------------------------------------

Josh, do give this deeper thought and look into what is happening under the hood.

What I want to know where the bottleneck could be and how I can setup the system so NT can actually keep up. This includes switching data-providers when needed or going into a sampled tick mode.

We can speculate on what could be going wrong but that would be pure speculation; it is not addressing the issue.

And unless the issue is addressed putting live dollars at risk is too risky.

[NinjaTrader_JoshP]

aviat72,

There is no point theorizing on what could/could not be the case. Please check your CPU usage in the Task Manager when you see such a situation. Only then will we be able to know what is happening. If the CPU is maxed out then to proceed will be to consider some of the performance tips Austin has provided earlier. If the CPU is not maxed out then performance is not the issue to blame and instead it could be either that was simply what your data provider was offering or network congestion or dropped packets, etc.

If you are worried about horsepower please be sure your computer meets minimum system requirements: http://www.ninjatrader-support.com/H...lationAndSetUp