ChangeOrder() - waiting on lock

[TradeAbove_Josh]

Hello,

I've now encountered this issue several times, where I reach a frozen strategy state with "waiting on lock owned by Thread <#>" when calling ChangeOrder() on an order (see attached screenshot). Note in the screenshot example that ChangeOrder() is being called from OnOrderUpdate() - though I'm not sure that this fact is relevant to the issue. The strategy is Unmanaged. I'm not doing any custom locking (such as locking on the SC or CbiOrder objects) that would cause this. Seemingly, NT is doing some internal ReaderWriterLockSlim locking (see screenshot) that is hanging up. I wish I could provide some simple test code for reproducing but the issue is rare - though again, I've now encountered it several times during real-time Sim101 strategy testing over the last couple of months. I hope you can shed some light on the locking situation and how I can avoid this.

Thanks,
Josh

[NinjaTrader_Jim]

Hello TradeAbove_Josh,

Thanks for your post.

Changing an order from within OnOrderUpdate may be part of the problem, but I have some additional questions

Is the order you are changing the same order that is updating OnOrderUpdate? If so, I may recommend triggering a delayed action so the order change is made elsewhere and after the OnOrderUpdate processes for that order. I have attached a rough example that starts a timer for a delayed event in OnBarUpdate. A similar approach could be taken.

If this doesn't help, could you provide more detail, for when this order change takes place, and what you have programmed in OnOrderUpdate so we can give further advise?

Ultimately we can best assist with small examples that focus on the specific part of your strategy you are having trouble with, that are written to have that part of code hit the issue you are experiencing. For example, if this is encountered due to rare trading conditions and timing, it may be possible to to trigger similar actions/scenarios that would cause this to occur in order to get the issue more reproducible.

We look forward to assisting.

[TradeAbove_Josh]

Thanks for your response, Jim.

Re: Is the order you are changing the same order that is updating OnOrderUpdate?
It's a good question, but no, the orders are different.

Re: If this doesn't help, could you provide more detail, for when this order change takes place, and what you have programmed in OnOrderUpdate so we can give further advise?
I have an order action queueing system (to mitigate the issue discussed here: https://ninjatrader.com/support/foru...d-freeze-chart) to prevent SubmitOrderUnmanaged(), ChangeOrder() and CancelOrder() calls from happening in rapid succession. So I'm waiting for order1 to reach a "submitted" state (either Submitted, ChangeSubmitted or CancelSubmitted, depending on what action was called on order1) before allowing ChangeOrder() to be called for order2. So the desired OrderState for order1 is detected in OnOrderUpdate(), which then triggers order2's ChangeOrder(). I'm quite confident that this call queuing is only helping and not related to the issue, as it's simply chaining order actions (rather than allowing them to be executed at the same time / almost at the same time) by waiting for a desired order state for the latest submitted/modified order (no thread waits or anything weird).

Can you provide info on what NT object(s) is being locked during ChangeOrder()? At a minimum, I'd like to be able to determine if a lock exists (a.k.a. anticipate an issue) and if so skip the ChangeOrder() call to avoid the lock up and implement a more graceful failure.

Thanks,
Josh

[NinjaTrader_Jim]

Hello Josh,

Our Scripting Support team does not have access to internal platform source code, but I would suspect Account.Orders to be locked if you are seeing a lock message related to ChangeOrder. I don't think locks would be necessary as we do not outline using before calling ChangeOrder in our Help Guide. (The issue may be related to changing within OnOrderUpdate.) You are welcome to try and report your findings with locking Acount.Orders. If you have an example that shows without locks the threading error is seen, and with locks the error is not seen, we could illustrate the issue to Product Management and Development to see if we can add a note to the Help Guide clarifying.

I am not sure if this is directly related to your issue, but I would suggest using ChangeOrder after you see the order has an OrderState of OrderState.Accepted (Stop orders / Stop Limit orders) or OrderState.Working (Limit orders and Stop Limits converted to Limit orders.) I would interpret those states as when an order change is completed.

The thread Kate was working touches on strategies that have numerous updates to orders causing the Chart to lock up because Chart Trader is enabled and tried to update order markers faster than the chart should be updated. If a strategy is extremely resource intensive (lots of actions on each tick, using single tick data series for operations we would otherwise use a larger data series) and exhibits issues freezing charts, We would suggest either running from the Control Center, disabling Chart Trader on the chart, or modifying the strategy so it does not change orders so frequently (price changes may be more effective than changing with each tick.)

If you are getting freezes that are not related to running on a chart with Chart Trader enabled or hidden, I would like to investigate that further, without your queueing system. Could you provide a test strategy if this is the case?

[TradeAbove_Josh]

Thanks, Jim.

Given the challenge to reproduce this (this strategy operates 'On bar close' and the original issue reported above did not occur during a burst of market volatility), I'd love to get Product Management and Development's hunch on what ReaderWriterLockSlim.WaitOnEvent might indicate for the ChangeOrder() lock that is not releasing.

Re: I would suggest using ChangeOrder after you see the order has an OrderState of OrderState.Accepted (Stop orders / Stop Limit orders) or OrderState.Working (Limit orders and Stop Limits converted to Limit orders.)
I am actually already doing this. In the original screenshotted example the order passed to ChangeOrder() is of state OrderState.Working. I've been using OnOrderUpdate()/OnExecutionUpdate() to trigger Unmanaged order actions for quite some time now - the classic situation being submitting stop/target orders when an entry fill comes in. I'm sure this is common practice. Would you be able to confirm with Product Management and Development that order actions (SubmitOrderUnmanaged(), ChangeOrder() and CancelOrder()) should not be called from OnOrderUpdate()/OnExecutionUpdate()?

Re: If you are getting freezes that are not related to running on a chart with Chart Trader enabled or hidden, I would like to investigate that further, without your queueing system. Could you provide a test strategy if this is the case?
I wish I could, but I'm not able to provide a simplified test strategy at this time, given the challenge to reproduce with the regular strategy. If it turns out that the platform does not support SubmitOrderUnmanaged(), ChangeOrder() and CancelOrder() calls from OnOrderUpdate()/OnExecutionUpdate() then that's likely my issue; if I removed the call queueing then I'd be left with a bare-bones 'On bar close' strategy.

[NinjaTrader_Jim]

Hello Josh,

Product Management and Development aren't user facing, and we can't exactly inquire on their hunch without something reproducible. Even if the issue is not 100% reproducible every time, we will need some simplified example code to go down that path, and a procedure to take that will get us there eventually.

I have done quite a bit of managed/unmanaged strategy work using OnOrderUpdate only and also with OnExecutionUpdate. There is nothing stating that you cannot change/cancel/submit an order from OnOrderUpdate. I only had guessed about an attempt to modify an order that was already updating.

OnExecutionUpdate is recommended for stop/target submission because this is when the strategy's Position object is updated. It is also possible (even necessary) to use OnOrderUpdate to trigger order submissions and changes to other orders since this would be the process to handle order errors and rejections with RealtimeErrorHandling

I don't think the order queuing code would be necessary. It seems that was implemented to get around the limitation involving the chart getting overloaded when Chart Trader is enabled or hidden.

I suggest we take a path forward without using your order queueing system and as long as you are:

1. Enabling on a Chart with Chart Trader disabled
-OR-
2. Enabling from the Control Center (without a opening Chart on that symbol with Chart Trader enabled)

I don't expect freezes or locking issues. If we have issues there, we can consider them potential bugs and move forward demonstrating the issue with Quality Assurance so we can get the issue on the Development team's desk.

If you want to work with me more directly over email (in case you do not want to share some of your order handling logic on the forum) you can email me at scriptingsupport [at] ninjatrader [dot] com with the text "Attn Jim 3366164"

[TradeAbove_Josh]

Thanks, Jim.

Re: I don't think the order queuing code would be necessary. It seems that was implemented to get around the limitation involving the chart getting overloaded when Chart Trader is enabled or hidden.
Unfortunately, this submit/change/cancel queueing by using the "submitted" state of the last submit/change/cancel as a trigger to perform the next submit/change/cancel is necessary in my situation because I have clients that require user interaction with their strategies. It has resolved the freezes I was getting before using this approach.

I don't suspect that this issue is related to the Chart Trader-enabled chart overloading, but I'll see if I can better prove this. Either way, a fix is needed for the Chart Trader-enabled chart overloading. "Don't use Chart Trader" is difficult advice for a programming service provider to swallow when NinjaTrader is the preferred platform.

[NinjaTrader_Jim]

Hello Josh,

The cases I have seen where the chart freezes with Chart Trader enabled and a resource intensive strategy applied have only been extreme cases. One case was using single tick Bid, Ask and Last series to submit/change/cancel orders, which can certainly be abusive for how many updates hit the chart to update orders when Chart Trader is enabled. That developer reported they no longer encountered the symptom after updating to Release 25.

I will be happy to give a second opinion to confirm if that is the same case for you, and offer any suggestions if I have any.

The same goes for asking about the locking situation, If I am given some code that shows exactly what you are doing, I would be better equipped to ask for opinions of others that have access to the platform source code.

If you want to keep your strategy logic privileged, please reach out to me over email.

[TradeAbove_Josh]

Thanks, Jim.

I'll continue testing on 8.0.25.0 and reach out if/when I'm able to demonstrate my issue.

[bltdavid]

I'll continue testing on 8.0.15.0 and reach out if/when I'm able to demonstrate my issue.

8.0.15.0?

Is that a typo?

Why are you not testing with the latest release 8.0.25.0?

[TradeAbove_Josh]

Typo, thanks. Edited