The strategy freezes and freeze chart.

[hedgeplay]

Hi Kate,

Like fx wrote in some posts and Josh wrote above I also see frequent deadlocking issues on just using 4 orders at a time where updates to as few as 2 LimitOrders place MilliSeconds, 30 seconds or minutes apart from other action, especially in the simulator. The deadlocks sometimes force a hard crash and reboot of NinjaTrader. When using the simulator this crash often corrupts NinjaTrader.sdf with a hung pending cancel or change order.

Recent Re-test Environment

I tested again roughly a month ago using a new bare-bones install of NT8 8.0.26.0 with zero Dlls from me or any third party installed and did see any improvement in the propensity to deadlock.

In order to test with a with a clean slate I leverage a recent strategy example you posted where I think you and Jim added management of a second order pair to the Unmanaged template. I leveraged those strategy changes forward to managing four orders concurrently.

I seeing deadlocking pretty quickly by submitting just two changeOrders in succession (like move both Buy Limit orders up by 1.5 points). The log shows that most often two successive changeOrders execute successfully with execute times both inside the same millisecond or just a few milliseconds apart and sometimes NT8 locks up when the execution happens a good distance in apart in milliseconds.

I was using test platform with 0.7-1.9 seconds of market data lag (measured at OnDataPoint in a BarType), CQG Continuum data, and a base TCP/IP latency from Chicago of 44ms.

The deadlocking issues persist when the NT8 Custom project is compiled in normal and in Debug modes.

The deadlocking issues persist when the VS 2022 is attached in Debug mode and when not attached.


The point of this post.. a request.

I do not know the scope of SFT-5163. Some text left me with the impression that SFT-1563 might be written to include mention of the faster ~"close some of the distance vs HFT" execution speed we all want and/or possibly managing very large number of orders.

So I wanted to ask that if the ChangeOrder() and CancelOrder() Cbi-Chart-Data pipeline deadlocking issues are being tracked in SFT-5163 as a feature enhancement please independently submit these concerns as a bug.

... As a bug without asking for enhancements to the current expected NinjaTrader feature set so these order management deadlocking bugs are more likely to get attention sooner.

From my perspective:

- I still agree with my earlier comments on this topic area, "Yes PlayBack and Simulation/Testing functionality is important but production order management reliability and features is much more important. So I would not want to see production features or quality hobbled to address these bugs."

- I would be happy with even very thin simple solutions if they improve reliability. Just early spitballing a few example ideas here:

--- While debugging, I have noticed deadlocks occur most often on the submission of the second ChangeOrder(). If it will reduce deadlocking, guidance and a sample of how when one is using Unmanaged Orders one should use the functionality of myAccount.Change(new[] { ... }); to submit all currently known change order requests within a single object array[] in a single call
https://ninjatrader.com/support/help...tml?change.htm

--- An informational-only public object we could monitor that shows when the NT internal code has locks in place for order change management or order related chart sync so we can see when locks are held by NT code but NT managed code process execution is not dependent on the lock being free so incorrect use by developers poses no risk to the execution health of NT provided code.
---E.g. public static object SyncChangeOrderLockIsHeld

--- An "IsLocked" public property bool we can check before submitting a CancelOrder() or ChangeOrder() request.

--- Guidance on where and how we should make more liberal use of "Dispatcher.InvokeAsync" when ChartTrader is active to reduce risk of deadlocking.


Thanks!
Hedge

[jeronymite]

hedgeplay First, may I refer you to this comment I have made many times in the Forum, by way of introducing further comments below: https://ninjatrader.com/support/foru...78#post1201878

The issue with Orders being "stuck" in Cancel pending is not the only issue you seem to be facing, and probably not the most severe, but it does seem to be part of it; so my comments here are really only related to that part, which I have also seen, plus some additional insights from what I have written and tested myself quite separately. I confess I have not read every word of the entire thread on this, so I hope you'll forgive me if I have missed important matters.

Additional comments:

• My comments here come from having developed a very complex AddOn (100K lines) that is capable of submitting 2000+ Orders (yes, 2K+) at the click of a button, most of which are Limit or Stop types of Orders, but some of which will be Market
• I found that Orders become "stuck" in Cancel pending after being Cancelled when an Order object is created (using Account.CreateOrder()) but never actually submitted; this happens whether it is a single Order trade or multiple Orders and regardless of any duration between Orders
• To overcome this, I adopted the approach of only creating Order objects immediately prior to their submission, rather than all Orders that might be generated by a request by a trader, but which may be delayed in being submitted and therefore potentially "stuck" if Cancelled before submission
• This clearly observable (and reliably wrong) behaviour is not the result of any crash or hang; it is simply and absolutely repeatable and demonstrates a shortcoming somewhere in the code; it is thus also not necessarily a "corruption" of the db, although that may be the "under the covers" cause.
• By avoiding creating Orders until they are to be submitted, I completely stopped all instances of Orders becoming "stuck" in Cancel pending after being Cancelled
• My software can submit huge numbers of Orders as fast as the system/software/network allow; however, I can only say that it is my experience that I have seen no deadlocks related to such an "avalanche" of Orders, remembering that my software is an AddOn whereas yours is a Strategy, and the symptoms of your issue seem significantly more pernicious.

This in no way detracts from your experiences -- quite the opposite. I offer it as a way of encouraging your further investigation to see if you can narrow down the exact, hopefully repeatable, circumstances causing the issues you're experiencing.

As a final comment, I am very aware of the potential for the sort of issues you are having, and, having been through many frustrating challenges with finding ways to overcome or fix things, I know that it can be disheartening, frustrating and annoying; but I urge you to keep trying to find an identifiable and repeatable set of circumstances that can produce the symptoms of the issue. We both know that such an approach will make it more likely the issue will be investigated by NinjaTrader Development. And as developers ourselves, we also know that with such a successful analysis, it may present opportunities for workarounds that relieve us of the burden of the issue, and allow space for it to be fixed by the NT developers.

If I can assist with any testing, please feel free to reach out to me.

Thanks.

[hedgeplay]

Hi @jeronymite;

Thanks for the reply.

I very much appreciate your watchtower work in this forum .. always on the lookout to see how you can other people by leaving posts rich in quality content.

The orders referenced in the posts above are all created via strategies using work I Currently all my submissions are made via SubmitOrderUnmanaged() and the local order object is not created until the order's first event in OnOrderUpdate().

Sometimes I do use Account.CreateOrder() so thanks for the tip on the need to be careful with process timing when using Account.CreateOrder().

I don't personally see the deadlocking on submission of blocks of orders, occasionally on CancelOrder() but for me the NT8 backoffice deadlocking related to ChangeOrder() really problematic.

=========

Where I personally see


"I offer it as a way of encouraging your further investigation to see if you can narrow down the exact, hopefully repeatable, circumstances causing the issues you're experiencing."

From the reading in this thread and other posts in the forum:

1. My perception is that issues (bugs) are known, just potentially have not yet been prioritized high enough to be fixed. So my request to Kate above was if not yet done so to submit the bugs independent on the enhancement request also discussed in the thread.

- My perception is that NinjaTrader development team already have the issues at least somewhat characterized and if not the NinjaTrader development team are the only people with enough design and code visibility to could carry out further code level investigation beyond what has been done to date. I don't want clutter the thread with the long list the dev team can see but we can not. The bugs are not really random we just don't have enough visibility to the many moving NT8 backoffice parts to create a narrow instantly repeatable example strategy.

- These seems to be known multithreaded deadlocking issues that have to do with:

1. How and when Ninjatrader.Cbi back-office code interacts with the NinjaTrader Chart
2. When Chart Trader is enabled on a chart
3. When Orders are active
4. Is at least more frequent when PlayBack or the Simulator is involved as well.

2. I think it is highly repeatable using the lighter more common strategy examples fx.practice raised later in the thread, or the simple back to back CancelOrder() multi-order strategy Josh mentions in his post or the Sample strategy approach I mention above laid upon charts with heavy data lag and / or heavy indicator pressures and left to run on auto-trade for a number of hours.

=========

I added a few brainstorming bug fix ideas to my post to Kate above. Is there any improvement or addition you would make to that list?

Hedge

[jeronymite]

Thanks very much, Hedge. I appreciate your kind words.

Your suggestions are good. The priority of fix bugs and then enhance is spot on. I have no additional insights on the deadlock matter for NinjaTrader consideration. You have done well there. I might only suggest that within one's own code, one might judiciously use locking to ensure there is no potential for "overrun" within one's own code that might contribute to potential deadlocks elsewhere. Just a thought if it seems relevant.

There are certainly issues experienced that speak of deadlock potential in the platform, and there is obviously work to be done in areas where that is already identified. One might also posit that a code review in areas with a high potential for deadlocks could lead to pre-emptive "hardening" to try to minimise risk, but that is perhaps more project-oriented rather than task-oriented.

There is probably some common code in the platform for Managed, Unmanaged and AddOn methods that manipulate Orders. Managed methods seem to be relatively stable in this sense, so one wonders if there is already code that could/should be leveraged to better effect in the "less managed" methods to reduce risk of deadlocks. Pure speculation.

In some circumstances, I have found situations that sometimes manifest as an apparent deadlock, but which, given sufficient time, resolve to continue execution. These are almost certainly performance related, and I have written on some of these elsewhere (https://ninjatrader.com/support/foru...ormance-issues).

One important performance design consideration that I found the hard way relates to work performed on threads, and specifically the UI thread. Every button/menu click is handled on the UI thread, and therefore any consequential actions, unless explicitly "off-loaded", happen on the UI thread. For example, symptomatically I see things as described in the post I mention above, which have the hallmarks of this type of design flaw. Freezing UI elements is often quite likely to be a symptom of not recognising and catering for this behaviour, particularly when the expected behaviour is known to be intensive or subject to arbitrary "wait" periods that are not implemented well. Whilst I can't say this is the case in any specific instance, when there is enough "disruption" to the UI, this may be a likely cause. Quite simply, any UI-initiated action that has the potential to be intensive should be recognised at initiation and handed off to its own thread for the main actions so as to maintain the responsiveness of the UI. Although not directly related to the problems you've identified, this type of issue would seem to explain quite a few of the "hangs" I tend to see, particularly if waiting long enough sees the hang "resolve".

I hope you'll find joy in resolving the issues you've encountered soon.

Thanks.

[TradeAbove_Josh]

I am curious are you seeing this a lot during live production or via simulation? I see this mostly during use of Simulation.

Mostly in simulation, yes, but that may be because I run far more sim than live.

What I have done to try to limit the issue: reduce dependencies on Locks and hobble my use of NinjaTrader Chart UI functionality.

• Run the strategies only via the control center, only use the charts to monitor but not to execute the strategies
• Add a 50 MilliSecond delay between consecutive Change or Cancel orders.
• Not yet sure it helps but when possible send a Account.CancelAllOrders() rather than individual cancels so I am only personally initiating one locking event against the huge Account and Orders objects rather than four in succession.
• Have not tried this yet but it is the next step. Why send four Cancel requests in sequence? Again so my code is the source of fewer lock requests all orders to cancel a single list and send just one Cancel Request via Account.Cancel(new[] { stopOrder, targetOrder .. etc }).
• This is such a problematic and risk generating bug I gave up running strategies through charts and the use of ChartTrader to monitor strategy execution and now have switched to pairing key strategies with an indicator that mimics all order entries and exits on the chart. I monitor strategy execution via the many wonderful windows of the Control Center. Now my strategy order submissions do still have Control Center UI interdependencies but no longer have any Chart UI interdependencies.

Edit: I am adding links to these additional posts from above in case someone finds this thread and is looking for a collection of mitigation tips.

Thanks very much for detailing what you've tried. As you referenced, I have implemented order action queueing as a bandaid solution that only turned out to reduce but not resolve the deadlocks. My next attempt might be to write a mirroring indicator that provides strategy visuals while running from the Control Center, as you suggest.

Hi Kate,

The point of this post.. a request.

I do not know the scope of SFT-5163. Some text left me with the impression that SFT-1563 might be written to include mention of the faster ~"close some of the distance vs HFT" execution speed we all want and/or possibly managing very large number of orders.

So I wanted to ask that if the ChangeOrder() and CancelOrder() Cbi-Chart-Data pipeline deadlocking issues are being tracked in SFT-5163 as a feature enhancement please independently submit these concerns as a bug.

... As a bug without asking for enhancements to the current expected NinjaTrader feature set so these order management deadlocking bugs are more likely to get attention sooner.

From my perspective:

- I still agree with my earlier comments on this topic area, "Yes PlayBack and Simulation/Testing functionality is important but production order management reliability and features is much more important. So I would not want to see production features or quality hobbled to address these bugs."

- I would be happy with even very thin simple solutions if they improve reliability. Just early spitballing a few example ideas here:

--- While debugging, I have noticed deadlocks occur most often on the submission of the second ChangeOrder(). If it will reduce deadlocking, guidance and a sample of how when one is using Unmanaged Orders one should use the functionality of myAccount.Change(new[] { ... }); to submit all currently known change order requests within a single object array[] in a single call
https://ninjatrader.com/support/help...tml?change.htm

--- An informational-only public object we could monitor that shows when the NT internal code has locks in place for order change management or order related chart sync so we can see when locks are held by NT code but NT managed code process execution is not dependent on the lock being free so incorrect use by developers poses no risk to the execution health of NT provided code.
---E.g. public static object SyncChangeOrderLockIsHeld

--- An "IsLocked" public property bool we can check before submitting a CancelOrder() or ChangeOrder() request.

--- Guidance on where and how we should make more liberal use of "Dispatcher.InvokeAsync" when ChartTrader is active to reduce risk of deadlocking.


Thanks!
Hedge

This is an excellent acknowledgement of the issue we are facing and some efforts that I know we would all love to see from the NT development team. NinjaTrader_Kate