overfills what and why

[pbcah]

Hi,

We've been system testing our strategy with NT7 15 (recently upgraded to NT7 16) for a couple of weeks in combination with IBs TWS. During that period we have experienced 2 unexplained "Overfill" errors that result in an Buy order (after being short) being left in the market immediately following the overfill dialogue error. We see this as our only 1 of 2 "A" class error outstanding in the overall system test.

As these events are very infrequent, I have not been able to fully diagnose the exact sequence of events that cause them. Our Strategy employs an EnterShortStop order, plus StopLoss and ProfitTarget bracket orders that are placed and adjusted inside the OnExecute method as recommended. (As I've outlined in another thread this is not an atomic operation and we would prefer to submit these 3 orders as one block and will redevelop our order management handling to use Unmanaged orders over the next couple of weeks. This is the other "A" class problem outstanding).

Once there is a position and the 2 bracket orders are in place, the only thing that is changed is the SL price from time to time when an indicator needs to move the stop. When the overfill error occurred, it appears apparent that it was not necessary to move the SL price and we don't see any related event information that points to this happening.

Can you outline the circumstance where an Overfill happens (ala is it a race problem where IB is closing an order while the SL is being updated etc.)?

Can you also outline or even better send some example code to show how to correctly handle overfill events?

Are there still any issues with Overfills as listed against NT7 14 in the latest NT7 16 release?

Regards,

Paul

[NinjaTrader_Bertrand]

Hi Paul, I'm not sure which 'overfill issues' you refer to - working with the unmanaged approach always had the potential implication of having to deal with overfills / inflight executions as they come in - so you would need to monitor this new IOrder property and rectify as needed in your custom code, unfortunately we would not have a sample handy for this (if you choose to set IgnoreOverFills to true as I would imagine to not halt the strategy - http://www.ninjatrader.com/support/h...reoverfill.htm)

[adamus]

Can you outline the circumstance where an Overfill happens (ala is it a race problem where IB is closing an order while the SL is being updated etc.)?

I would also be interested to find out more about overfills and why they happen.

I received two today and the strategy left orders live after it closed down. I'm running managed orders

There is very little information in the help files.

Thanks

[pbcah]

Hey Bertrand,

Bertrand, I take a good amount of time to detail these queries in clear and unambiguous terms. I would appreciate it if you could take a little more time to read them as I don't seem to be communicating the issues clearly as you have missed some important detail.

I appreciate the fact that you probably have hundreds of emails to process each day and see that you respond in a timely manner and I do appreciate your support, however when-ever I need to send clarification, I loose 24 hours in delays in solving my problem. When you misread my question, I loose another 24 hours asking again and you have another reply to send. I have customers who are waiting and this problems is in your standard code.

Hi Paul, I'm not sure which 'overfill issues' you refer to - working with the unmanaged approach always had the potential implication of having to deal with overfills / inflight executions as they come in - so you would need to monitor this new IOrder property and rectify as needed in your custom code,

(As I've outlined in another thread this is not an atomic operation and we would prefer to submit these 3 orders as one block and will redevelop our order management handling to use Unmanaged orders over the next couple of weeks. This is the other "A" class problem outstanding).

Bertrand, we've implemented our strategy using your managed API. The overfills happen with the managed implementation.

unfortunately we would not have a sample handy for this (if you choose to set IgnoreOverFills to true as I would imagine to not halt the strategy - http://www.ninjatrader.com/support/helpGuides/nt7/ignoreoverfill.htm)

We can't just simply ignore Overfills and want to ensure no overfills can occur in the Managed version. These are stop orders in the market and cannot be ignored as they become filled positions that eventually produce losses.

We will be developing an unmanaged approach to accommodate single transaction bracket orders later however our managed implementation works except for the Overfill issue and the bracket order issue and we will deploy this with a warning about the possibility of a Naked order if the internet connection is lost.

As these events are very infrequent, I have not been able to fully diagnose the exact sequence of events that cause them. Our Strategy employs an EnterShortStop order, plus StopLoss and ProfitTarget bracket orders that are placed and adjusted inside the OnExecute method as recommended. (As I've outlined in another thread this is not an atomic operation and we would prefer to submit these 3 orders as one block and will redevelop our order management handling to use Unmanaged orders over the next couple of weeks. This is the other "A" class problem outstanding).

Once there is a position and the 2 bracket orders are in place, the only thing that is changed is the SL price from time to time when an indicator needs to move the stop. When the overfill error occurred, it appears apparent that it was not necessary to move the SL price and we don't see any related event information that points to this happening.

Can you outline the circumstance where an Overfill happens (ala is it a race problem where IB is closing an order while the SL is being updated etc.)?

Can you also outline or even better send some example code to show how to correctly handle overfill events?

I know I'm probably being a bit of an Australian Terrier here but I need these answers. Can you please send me this information so I can write in corrective actions to stop this happening?

Are there still any issues with Overfills as listed against NT7 14 in the latest NT7 16 release?

There is a forum post that outlines serious problems with NT7 14 that produced overfills. I have searched for it but cant find it however it is there. I can't find any reference to these issues in the release notes.

I would appreciate your urgent assistance.

Regards

Paul

[NinjaTrader_JoshP]

Paul,

1. Overfills are a fact of the matter regardless of if you are using the managed or unmanaged approach. Whether you have a possibility of running into them is all dependent on how your orders are programmed.

2. The way you are submitting orders inherently runs the risk of overfills. Any time you have at least two orders competing to do the same thing you run the risk. In your case you have Stop, Target, and even a reversal order. Having competing orders like that additional reversal order is not recommended for situations you have already noted.

Consider this scenario:

Stop gets filled at exchange
Exchange reports to brokerage
Brokerage is still in the process of reporting back to you on the fill
Reversal order gets filled at exchange before brokerage reports
You now have closed the same position twice and are left with an overfill

There are other scenarios that can cause overfills and they are all caused by the inherent nature of the order structure the code is placing. To prevent overfills you would need to ensure you never have competing orders. To do this you should just either modify a preexisting order to induce a fill when you desire or cancel the preexisting order, wait for cancel confirmation, then submit your new order. There is no way around this.

Furthermore you mentioned that you have EnterShortStop and a stop/target bracket. Please clarify what methods exactly you are using for your stop/target bracket. There are internal order handling rules for managed orders that generally would prevent the above scenario from even happening in the first place.

[pbcah]

Hi Josh,

Thanks for coming back to me.

Paul,

1. Overfills are a fact of the matter regardless of if you are using the managed or unmanaged approach. Whether you have a possibility of running into them is all dependent on how your orders are programmed.

After discussing this with IB, I've found that IB has a setting in their TWS called "Overfill Protection - If checked this dictates that only 1 order in the group will be routed to an echange at a time. This strategy completely removes the possibility of an overfill".

Are you aware of this and have you tested it and are there any side affects that we should know about?

2. The way you are submitting orders inherently runs the risk of overfills. Any time you have at least two orders competing to do the same thing you run the risk. In your case you have Stop, Target, and even a reversal order. Having competing orders like that additional reversal order is not recommended for situations you have already noted.

Consider this scenario:

Stop gets filled at exchange
Exchange reports to brokerage
Brokerage is still in the process of reporting back to you on the fill
Reversal order gets filled at exchange before brokerage reports
You now have closed the same position twice and are left with an overfill

Lets assume we're only going short to simplify the discussion.

We're using IB TWS which provides an OCA bracket order mechanism that has Overfill Protection turned on. With bracket orders, we have an Entry Sell Limit (EL), a Buy Stop Loss (SL) and a Buy profit target Limit order (PT).

The SL, EL and PT orders are separated by reasonable margins (not tight). In IB we send all three orders to the broker at the server with one single transmit command. The entry order is the parent order and the SL and PT orders are child orders. When the parent order is executed, the child orders are automatically placed as associated orders to the parent position. When either order is subsequently filled, the other order is automatically cancelled.

BTW, if you happen to get a partial fill and decide to change the order before it is completely filled, you will be charged a second commission.

There are other scenarios that can cause overfills and they are all caused by the inherent nature of the order structure the code is placing. To prevent overfills you would need to ensure you never have competing orders. To do this you should just either modify a pre-existing order to induce a fill when you desire or cancel the pre-existing order, wait for cancel confirmation, then submit your new order. There is no way around this.

If Ninja can send orders to IB as a block (described above) then execute 1 single submit, then according to IB there is no possibility of an overfill.

Is this your experience?

Furthermore you mentioned that you have EnterShortStop and a stop/target bracket. Please clarify what methods exactly you are using for your stop/target bracket. There are internal order handling rules for managed orders that generally would prevent the above scenario from even happening in the first place.

Please see my description above.

Please understand guys, I'm really focused on helping you eliminate potential for errant orders in the system. Please don't take my observations and information as criticism, I'm only interested in helping you to improve Ninja 7. No offence is intended or implied.

Regards,

Paul

[NinjaTrader_JoshP]

Paul,

We would not know how settings on TWS impact your order behavior. We can only recommend you program your strategy in a manner that avoids the issue to begin with instead of relying on some fallback setting like Overfill Protection on TWS. Essentially what you are saying of what Overfill Protection does is exactly the same as I have recommended which is only submit one order at a time. Should you all of a sudden want the other order, cancel the first, wait for cancel confirmation, then submit the second. This guarantees no overfills are possible.

Your scenario of EL, SL, and PT works fine in NT. Submit your entry. When your entry is filled as determined by OnOrderUpdate() or OnExecution(), fire off your SL and PTs. You can even just use Set() methods and that will be automatically done for you without using OnOrderUpdate() or OnExecution(). The SL and PT are already automatically OCOed when you use Set() methods. If you decide to submit manual SL and PT via Exit() methods you will need to program in your own OCO handling.

[pbcah]

Hi Josh,

Thanks for your reply.

Paul,

We would not know how settings on TWS impact your order behavior. We can only recommend you program your strategy in a manner that avoids the issue to begin with instead of relying on some fallback setting like Overfill Protection on TWS. Essentially what you are saying of what Overfill Protection does is exactly the same as I have recommended which is only submit one order at a time. Should you all of a sudden want the other order, cancel the first, wait for cancel confirmation, then submit the second. This guarantees no overfills are possible.

Josh our implementation only sends one order at a time. New orders are only sent when both orders and position are closed. We send the SL and PT when the OnExecution event arrives and only update the SL during the life-cycle of a position. We have still seen Overfills very infrequently (1 out of 50 orders). Is there any other circumstance that will cause an Overfill?

Your scenario of EL, SL, and PT works fine in NT. Submit your entry. When your entry is filled as determined by OnOrderUpdate() or OnExecution(), fire off your SL and PTs. You can even just use Set() methods and that will be automatically done for you without using OnOrderUpdate() or OnExecution(). The SL and PT are already automatically OCOed when you use Set() methods. If you decide to submit manual SL and PT via Exit() methods you will need to program in your own OCO handling.

The problem I have with the NT implementation is that you are implementing this activity in the client not in the server. If it is implemented in the server the SL and PT orders are placed instantaneously that the position is filled.

When you try to do this handshake in the client, the events are separated by quite a large delay, one for the event to be sent from the broker to the TWS, then from the TWS to NT, then decision time in NT before the SL is sent to TWS then from TWS to the broker then the PT is send from NT to the TWS and then onto the broker. All these delays add up and can mean the market has moved opening the possibility of slippage if the SL is tight and this seems to be what we're seeing and this can be the difference between making a profit and a loss.These losses are unnecessary.

Also, as I've outlined before, as the internet is inherently unreliable, it can be disconnected during this window thereby causing naked positions. (as seen by our testers). Again this will result in more unnecessary losses.

When I raised this before, you indicated I could send the primary and attached orders to IB then transmit them simultaneously using the unmanaged methods. As IB manages all the Partial Fill logic at the server, we should simply use the order and position events to manage our state and update local values.

Do you see anything else we'll need to manage in this approach and can you send a sample implementation of the unmanaged methods?

Regards,

Paul

[NinjaTrader_JoshP]

Paul,

If you are still running into overfills then the first step you would want to do is analyze which orders exactly caused your overfills. Then you can pinpoint courses of action to take to address it.

As far as your concern for OCO running on servers, when you submit OCO paired orders in NT they are indeed natively run on IB servers. You can see here for where OCO behavior is ran: http://www.ninjatrader.com/support/f...ead.php?t=5349

To clarify your partial fill question, there is no "partial fill logic" that is ran anywhere per se. If you have specific logic you want to deal with then you would need to program that yourself regardless of if you are using managed or unmanaged. If you do not program for partial fills then it will just behave like how partial fills normally behave. If it is a partial fill on your entry order then each new fill execution would increase your stop loss/profit target quantity. If it is an exit order that is partial filling, it will amend down the quantity of any OCOed order also attached to the order.

Unfortunately there is nothing I can send you about unmanaged at this point in time besides what is already in the Help Guide.

[pbcah]

Josh,

Paul,
If you are still running into overfills then the first step you would want to do is analyze which orders exactly caused your overfills. Then you can pinpoint courses of action to take to address it.

This happens to 1 out of 50 transactions. I postulate that it may be some kind of race problem due to the delay in placing the bracket orders from the client.

As far as your concern for OCO running on servers, when you submit OCO paired orders in NT they are indeed natively run on IB servers. You can see here for where OCO behavior is ran: http://www.ninjatrader.com/support/f...ead.php?t=5349

Yes, eventually OCO orders (OCA in IB) do get handled in the server when they are eventually placed. The problem with the Ninja Managed implementation is that the delay caused by the initial handshake you impose is causing the 2 problems I have highlighting and we are seeing both problems exhibited in our system tests. This problem is an architectural issue that can be present in client server systems.

I am re-implementing my order management using the unmanaged API and already see from the initial tests that these issues can be avoided in Ninja but not by using the Managed implementation.

Unfortunately there is nothing I can send you about unmanaged at this point in time besides what is already in the Help Guide.

"The Unmanaged approach is reserved for VERY EXPERIENCED programmers." is the opening line in the documentation. I am a VERY EXPERIENCED programmer with 30 years of continuous software engineering experience developing systems from robotics control systems to transatlantic cable diagnostics systems to banking systems. I hope I qualify.

I find the documentation for Unmanaged order management is a little thin on the ground for an aspect that is quite important. I understand you are probably encouraging people to use the managed approach however this is no reason to leave out this documentation. The area I'm specifically referring to is the incoming event management and some kind of discussion about cause and effect apart from the info included with IOrder.

There is no information about which events we should expect over and above the Managed events that are passed through. A discussion about which events would normally be intercepted by the Managed implementation would also be helpful. To completely implement the Unmanaged event management (to be able to run on the space shuttle), I'll need to force various failure scenarios to send events to a test harness so I can write an appropriate response. Do your developers must have some more documentation about the range of events and their cause and action use cases please?


Regards,

Paul

[NinjaTrader_JoshP]

Paul,

There is no "handshake" sequence required. Once you receive filled events on your entry order, just submit your SL and PT right after each other. I mean literally ExitLongStop() followed immediately on the next line of code by ExitLongLimit(). Unless you are using point blank stops/targets you would have no problem submitting these orders and have no problem with any overfills in this manner once submitted. If you are using point blank stops/targets there is nothing that can be done differently besides not submitting orders that close to the inside market.

Please be more specific as to what information you want for Unmanaged. There is no "incoming event management". When you use Unmanaged (or even Managed), we pass to you literally everything passed to us by your brokerage. Whatever your brokerage reports is what you will get.

There are no events over and above Managed approach that you will receive. The difference between Unmanaged vs Managed is simply that NT does not impose any order handling rules to try and prevent invalid orders. Unmanaged simply means you have full flexibility to submit whatever order you want at any point in time. Whatever that means for your underlying account position is up to you to keep in check through your own code and NT will not prevent you from doing any type of order action. Both Managed and Unmanaged receive all order event updates from your brokerage.

General sequence of events is that you would Submit your order. The order would then be Accepted. Then it may go into a Working state depending on the order type. Then it may go into PartFilled. Then it may go Filled. If it gets cancelled it would go PendingCancel to Cancelled. If you submit order changes it would go PendingChange and then back to either Accepted or Working.