Critical processing speed in milliseconds

[pstrusi]

Hi,

After a long time doing live simulations, I went totally "live" through Interactivebrokers with caution. Since I noticed that Algo seemed to react too slow I decided to stop it and analyze action so far with amplifying glass. In the following very concise trade example you could see the "long time" it takes since the Algo detects a signal, send the order and updates its status:

2017-07-09 21:32:24:988 (Interactivebrokers) IB.Adapter.Submit: Id=29
2017-07-09 21:32:25:029 (Interactivebrokers) IB.OrderRequest.Send:
2017-07-09 21:32:25:030 (Interactivebrokers) New state=PendingSubmit
2017-07-09 21:32:25:030 (Interactivebrokers) Cbi.Order.Update1: NewState=PendingSubmit
2017-07-09 21:32:25:035 (Interactivebrokers) Cbi.OrderStatusEventArgs.Process
2017-07-09 21:32:25:129 WARNING: Order Event Warning:Attribute 'Outside Regular Trading Hours' is ignored ( unnecesary I trade 24/7)
2017-07-09 21:32:25:129 WARNING: The closing order quantity is greater than your current position. 2137 ( unnecessary I reverse positions with one order )
2017-07-09 21:32:25:384 (Interactivebrokers) IB.OrderRequest.ProcessOpenOrders
2017-07-09 21:32:25:385 (Interactivebrokers) IB.OrderRequest.Process: State=PendingSubmit
2017-07-09 21:32:25:385 (Interactivebrokers) Cbi.Connection.ProcessEventArgs New state=Accepted Instrument
2017-07-09 21:32:25:386 (Interactivebrokers) Cbi.Order.Update1: New state=Accepted Instrument='$EURUSD'
2017-07-09 21:32:25:386 (Interactivebrokers) Cbi.Connection.ProcessEventArgs New state=Working Instrument
2017-07-09 21:32:25:386 (Interactivebrokers) Cbi.Order.Update1: New state=Working Instrument='$EURUSD'
2017-07-09 21:32:25:389 (Interactivebrokers) Cbi.OrderStatusEventArgs.Process: New state=Accepted Instrument='$EURUSD' A
2017-07-09 21:32:25:390 (Interactivebrokers) Cbi.OrderStatusEventArgs.Process: New state=Working Instrument='$EURUSD'

1. I need to confirm something that I'm not used to. Reading the trace file, Is the order submit and working in IB servers at 21:32:25:129 when it reads "PlaceOrder is now being processed" after the system warning ? because otherwise it took almost 400 milliseconds to process, unnaceptable.

2. Could it be the warnings at 21:32:25:129 that may be interfering the speed?

3. Any ideas or suggestions to solve this unacceptable delay?

There are still an ongoing analysis but let's start with this "easy one " which repeats always.

By the way: NT simulation engine does barely some approach, but it fills are not very real when you trade fast and often, cause it seems not to respect the queue in bid-Ask volumes before your orders requesting to be filled. So that needs a reform.

Thanks in advance

[NinjaTrader_PatrickG]

This amount of delay can be expected with any provider when submitting orders. There are many factors, however, most importantly would be the quality/speed of your internet connection, the physical distance from your broker's/the exchange's servers, and the state of the path between you and those servers at the time of order submission.

If you're trying to improve your latency you could consider a VPS service, however, since the delay is less than one second there will likely be little or no improvement in latency.

I appreciate your feedback on the simulation engine and I'll submit a feature request to improve the simulation engine to be more realistic with ask and bid volume. I'll update this thread when I receive a tracking number.

EDIT - the tracking number is SFT-2512.

[pstrusi]

Hi Patrick, Thanks for attending my post.

What you address in the first place is a very logical step and NT has nothing to do it, it's just my responsibility, but I'm not asking that, please take a little time and stay to what I'm actually asking.

My actual latency to IB servers ( ping roundtrip is about 91 milliseconds ) so I'd expect "working status" on orders a little bit after that time. As you can see in the trade example extracted from the trace file, it takes around 130 milliseconds from the signal is triggered until the warning that states:

2017-07-09 21:32:25:129 WARNING: Order Event Warning: Attribute 'Outside Regular Trading Hours' is ignored based on the order type and destination. PlaceOrder is now being processed. 2109

As you can see highlighted in bold letters it says: "PlaceOrder is now being processed" afterward NT seems to open the IB's orders book to read or update status, this happens after 21:32:25:384 so, what I'm asking for real is

When are my orders really "working" in IB servers after the warning at 21:32:25:129
or after the update from orders book at 21:32:25:390 ?????

Looking forward to your critical technical answer

[NinjaTrader_PatrickG]

'Accepted' indicates that NinjaTrader received an order confirmation from your broker. 'Working' indicates that NinjaTrader received an order confirmation from the exchange (provided by your broker). More information on order states can be found here:

http://ninjatrader.com/support/helpGuides/nt7/order_state_definitions.htm?zoom_highlightsub=state

The 'working' update comes at 21:32:25:386:

2017-07-09 21:32:25:386 (Interactivebrokers) Cbi.Order.Update1: New state=Working Instrument='$EURUSD'

NinjaTrader does not control the speed of these updates. NinjaTrader behaves like a radio and relies on signals from your provider's API to then display updates in the platform.

[pstrusi]

Let me put it clearer with the same example:

The triggered signal comes at 21:32:24:988 but it seems that NT needs 141 milliseconds to process it through its servers as the Warning before "send" comes at 21:32:25:129 ( this delay shouldn't be that long, it's an eternity ) ...BUT not only that, it takes additional 155 milliseconds to receive "order status" from IB, and I can hardly believe that IB takes so long to process, so all aims that there's some critical delay in NT servers and this is a problem for everyone.

So, do you realize what I'm saying ? since my latency is 91 mls, I cannot accept easily that all of my orders processing always through NT servers takes more than 400 mls everytime, that's not right and it should be addressed urgently because it's critical. If you were in my shoes you'll think the same.

I really hope you could come with some idea, solution, improvement

Thanks

[NinjaTrader_PatrickG]

I'd like to clarify that at no time is an Interactive Brokers order handled by any NinjaTrader server.

Your ping to Interactive Brokers' servers is not a fair representation of the expected delay. NinjaTrader communicates to Interactive Brokers through an API. The signals sent from NinjaTrader to the API must be processed by Interactive Brokers and then a return signal sent through the API to NinjaTrader.

I tested this on my end with the edemo Interactive Brokers paper trading account and experienced similar round-trip times:

2017-07-10 10:46:49:718 (_IB Any) Cbi.Account.CreateOrder: orderId='1019cc2405bc4523b94a6e8bf47107f7' account='DU15022' name='' orderState=Initialized

2017-07-10 10:46:50:014 (_IB Any) Cbi.Account.OrderUpdateCallback: realOrderState=Working orderId='429108633' account='DU15022' name='' orderState=Working

300ms is not an unexpected amount of time.

[pstrusi]

I'd like to clarify that at no time is an Interactive Brokers order handled by any NinjaTrader server.

Your ping to Interactive Brokers' servers is not a fair representation of the expected delay. NinjaTrader communicates to Interactive Brokers through an API. The signals sent from NinjaTrader to the API must be processed by Interactive Brokers and then a return signal sent through the API to NinjaTrader.

I tested this on my end with the edemo Interactive Brokers paper trading account and experienced similar round-trip times:

2017-07-10 10:46:49:718 (_IB Any) Cbi.Account.CreateOrder: orderId='1019cc2405bc4523b94a6e8bf47107f7' account='DU15022' name='' orderState=Initialized

2017-07-10 10:46:50:014 (_IB Any) Cbi.Account.OrderUpdateCallback: realOrderState=Working orderId='429108633' account='DU15022' name='' orderState=Working

300ms is not an unexpected amount of time.

Thanks for your answer, this is totally discouraging

[pstrusi]

Since I need to look for a solution to get better processing times, the first thing I did was to consult the basic problem to IB support, so here I share its answer in order to inform and help next users like me that trades often:

Dear Mr. Strusi,

Thank you for contacting Interactive Brokers.

Generally speaking, a direct marketable order that sent out from IB server to exchange for execution would take a round trip time of ~25 ms only on IB side. Order latency would mostly come from the transmission from your local network to IB server. For clients who connect via the public Internet, round trip time of 400 ms is actually considered as good performance.

If you need us to look into any specific order further, please provide the order details, such as the Time, Symbol, Quantity, Buy/Sell and Price etc.

Regards,

Steven J.
API Support, Technical Assistance Center
Interactive Brokers

[pstrusi]

T..........
I appreciate your feedback on the simulation engine and I'll submit a feature request to improve the simulation engine to be more realistic with ask and bid volume. I'll update this thread when I receive a tracking number.

The simulation engine is a good tool to begin trials before progressing to paper account tests, however, it needs to address two issues to be more real:

1. It should take into account a minimum of 400ms before an order is considered "working"
I had overlooked the simulation engine setting for comms lag, so this point 1, may be considered solved
2. Orders, after "working status" should respect the bid-ask queue flow

Thanks

[sledge]

The simulation engine is a good tool to begin trials before progressing to paper account tests,

1. It should take into account a minimum of 400ms before an order is considered "working"

It is not that slow in the states with Rithmic so that's no solution.

[eDanny]

Maybe a setting to allow entry of a lag parameter to match your local lag time?

[NinjaTrader_PatrickG]

eDanny, that is an excellent suggestion. I have included it in SFT-2512

[pstrusi]

I want to use this post to consult a doubt.

I'm "playing around" with simulation engine settings found within "Tools / Options". I find two boxes:
Delay comm (msec)
Delay exchange (msec)

Despite I set correctly "a lag processing time" about 500ms in total, still it fills very quickly all orders triggered, So:

Are there other parameters that I should set as well to have more real simulation?

Thanks in advance

[NinjaTrader_PatrickG]

These settings do not function as intended in NinjaTrader 7 and do not exist in NinjaTrader 8.

The development life cycle of NinjaTrader 7 has ended and there are no plans to make any changes to NinjaTrader 7 outside of critical updates. Any improvements to any feature of the platform would be made in NinjaTrader 8.

[pstrusi]

These settings do not function as intended in NinjaTrader 7 and do not exist in NinjaTrader 8.

The development life cycle of NinjaTrader 7 has ended and there are no plans to make any changes to NinjaTrader 7 outside of critical updates. Any improvements to any feature of the platform would be made in NinjaTrader 8.

Thanks for your answer Patrick, it's a pity then. However I wonder: since the simulation engine is a basic tool to begin test, and it has a clear purpose,

why doesn't it exist in NT8 ? is it because it's better to work with paper accounts directly ?