How do I prevent deadlocks with MarketData, FundamentalData events?

[carnitron]

Sometimes when I add or remove an Update event from these objects, the thread on which I am doing this hangs.

It happens frequently enough when I shut down my AddOn with FundamentalData that I have turned that subscription off entirely. But I just saw a case with MarketData now too, though this is the first time for that in many hundreds of hours of runs.

Long story short, is there an NT8 object I am supposed to be locking? Like when I do something like this:

MarketData marketData = new(instrument);
// Saw a hang on a line like this:
marketData.Update += OnMarketData;

or this:

// Seen many hangs on a line like this:
_fundamentalData.Update -= OnFundamentalData;

I have tried using locking on my own static object to ensure none of my threads somehow step on each other down in NT8 code, but that had no effect. This might have worked, if all MarketData objects were sharing a static service internally that was not threadsafe. But the problem with this approach is that it only prevents my own threads from subscribing/unsubscribing at the same time, it does not prevent them from fighting with NT8 threads, nor does it prevent NT8 threads from fighting with each other.

Thanks in advance for any help you can provide.

Regards,
Peter

[NinjaTrader_Gaby]

Hello carnitron,

Thank you for your inquiry.

Please see the Help Guide links below for MarketData and FundamentalData, both pages have code demonstrating how to properly subscribe to updates. You should be using InvokeAsync() to subscribe to the events.

MarketData - https://ninjatrader.com/support/helpguides/nt8/marketdata.htm
FundamentalData - https://ninjatrader.com/support/helpguides/nt8/fundamentaldata.htm

Multithreading Consideration for NinjaScript - https://ninjatrader.com/support/helpGuides/nt8/multi-threading.htm

If you have any further questions, please let us know.

[carnitron]

Hi Gaby, thank you for your help here. I have updated my code to use async dispatch after (re)reading the pages you provided and using dispatching.

It seems to work so far, so theoretically I got it right and issue solved. Time will tell. But if it works, that's a nice win, so thank you.

I have a couple of questions/comments.

1. Amusingly, I actually looked at the MarketData/FundamentalData pages and did a text search for "lock" and "thread" but got no hits. I skimmed the examples and saw the InvokeAsync call and then (wrongly) assumed it was necessary b/c we were in the context of an NTTabPage and this was a UI-focused example. I register for MarketData and FundamentalData down in one of my DLLs on my own threads, so I figured this did not apply.

So one question is: does NT8's Instrument class have UI code associated with it? I.e. it's not just a lower level class that has nothing to do with UI, but actually is a much larger construct? And b/c of these UI bits I need to think about interacting with Instrument as if it were one of NT8's UI objects?

I guess another comment would just be that it would be nice if the NT8 doc surfaced some of the important content it has buried in examples and off on other pages that are not connected to the first place a person would look. This is not the first time this has happened to me. If subscribing and unsubscribing these events is perilous from a multi-threading perspective, that should be highlighted and made very clear in the pages for MarketData and FundamentalData, no?

2. Another threading question. What are the main Dispatchers in NT8 that I need to be aware of and use?

When trying to do things like update NT8 menu items from one of my background threads, I have had success with ControlCenter.Dispatcher.InvokeAsync() and Application.Current.Dispatcher.InvokeAsync(). Do these two techniques actually point to the same Dispatcher? Copilot tells me WPF UI controls share the same Dispatcher and that Application.Current.Dispatcher is the way to get at it, thus implying that these two techniques actually return the same Dispatcher.

On the other hand, the "Multi-Threading Consideration for NinjaScript" page you posted here says there are "multiple main UI threads". Does this mean the best (and maybe only) way to get the "correct" Dispatcher is to use the Dispatcher method on the object I am trying to talk to?

Are there separate Dispatchers even for objects like Instrument? Like if Instrument does NOT actually have UI bits, but is multi-threaded, meaning I need to access Instrument.Dispatcher any time I want to do certain operations with it?

Thanks in advance for any help you can provide here.

Cheers,
Peter

[NinjaTrader_Gaby]

Hello,

The Instrument thread and the UI thread are two separate threads. Subscriptions to data events (like MarketData, MarketDepth, and FundamentalData) from Instruments should be done on the Dispatcher of the [same instrument’s] instrument thread. Once a market data subscription is started, it is assigned an instrument thread randomly and all processes which are driven from realtime data for that instrument will be on the same instrument thread moving forward. The instrument thread will then drive logic.

While this is not mentioned in the pages for MarketData and FundamentalData, it is alluded to on the page for 'Multithreading considerations for NinjaScript':

"Depending on your CPU configuration, the NinjaTrader application will usually consist of multiple main UI threads, where various features like Charts or NinjaScript objects run, along with a number of background worker threads where events such as market data updates will be distributed throughout the product. In principle, an object can only access information related to objects that exist on the same thread. It is possible (and quite likely), that the thread which a NinjaScript object is running will not be the same thread as the event which is calling the object. In cases where you need to access objects on the UI from a NinjaScript objects calling event thread, a dispatcher can be used."

https://ninjatrader.com/support/helpGuides/nt8/multi-threading.htm

For UI events, ControlCenter.Dispatcher.InvokeAsync() should be always used as this is the dispatcher for the UI thread.

Both the UI thread and the Instrument thread are multithreaded. NT spawns them per the number of logical processor cores detected on startup. If you have 4 cores, this means you'll have 4 UI threads and 4 instrument threads for a total of 8 threads.

As mentioned in the above help guide page, it is possible (and quite likely), that the thread which a NinjaScript object is running will not be the same thread as the event which is calling the object. Anytime you are doing something like subscribing to market data events, accessing objects in the UI, etc. you should use a dispatcher in your script.

Please let me know if you have any further questions.

[carnitron]

Hi Gaby. Sorry for the delay in responding here, I had a big deadline yesterday.

Anyhow, thanks for your clarifying comments here. Let me see if I can summarize instruments:

1. There is one Instrument thread for each logical core.

2. Once an Instrument is assigned an Instrument thread, it will always have that thread. You can always talk to an Instrument using its assigned Instrument thread by using Instrument.Dispatcher.InvokeAsync().

3. Subscribing and unsubscribing to and from instrument data has to be done via Dispatch.InvokeAsync().

If I have that correct, I think we are good there.


UI threads are still a bit unclear.

1. You mention that if I have 4 cores then I would have 4 UI threads. And you say that ControlCenter.Dispatcher.InvokeAsync() should always be used to dispatch for the UI thread. But which one? If I have 4 UI threads, which ones are assigned to which UI objects? How do I use ControlCenter.Dispatcher to get to the UI threads not assigned to Control Center?

2. Just to confirm, when I call Dispatcher.CurrentDispatcher in response to a menu click in my AddOn, that is always using ControlCenter's thread?


General questions:

1. Are there any other threads besides the UI and Instruments threads that I need to be aware of and access?

2. For instance, Indicators need to be dispatched to. Which threads do they use? What about MarketAnalyzerColumns?

3. Minor question for instruments. Would it be safe to read between the lines and state: "Reading properties like Name is threadsafe and can be done from any thread"? Instrument's API is pretty small and appears mostly read-only, besides the events.

Thanks for all the help.

Cheers,
Peter​

[NinjaTrader_ChelseaB]

Hello Peter,

I think you've got the basic understanding correct.

Each unique instrument thread will be assigned to a specific logical processor. All subscribers to that instrument will also be running on that thread.

"You mention that if I have 4 cores then I would have 4 UI threads. And you say that ControlCenter.Dispatcher.InvokeAsync() should always be used to dispatch for the UI thread. But which one? If I have 4 UI threads, which ones are assigned to which UI objects? How do I use ControlCenter.Dispatcher to get to the UI threads not assigned to Control Center?"

The dispatcher of the object being accessed should be used. If you are modifying the WPF of a chart, use the ChartControl.Dispatcher. If you are accessing the ControlCenter to add menu items use the ControlCenter.Dispatcher. If you are accessing an instrument use the Instrument.Dispatcher.

Just to confirm, when I call Dispatcher.CurrentDispatcher in response to a menu click in my AddOn, that is always using ControlCenter's thread?

When a menu item is clicked, this will be running in the UI thread. If you needed to access an object on another thread like the Instrument thread when the menu item is clicked, use the dispatcher for that object when working with that object.

"Are there any other threads besides the UI and Instruments threads that I need to be aware of and access?"

There is also an account thread for order updates. (Account.OrderUpdate / Account.ExecutionUpdate / Account.PositionUpdate)
Assigning a handler method to these events generally will not require a Dispatcher.

https://ninjatrader.com/support/helpGuides/nt8/orderupdate.htm

The ControlCenter (and any other window) as well as the ChartControl would both be on the UI thread.

"For instance, Indicators need to be dispatched to. Which threads do they use? What about MarketAnalyzerColumns?"

Indicators being accessed by other scripts would be on the same thread, the UI thread. Indicators are called as methods. For example Print(SMA(7)[0]);.

The MarketAnalyzerColumns scripts will be generated on the UI thread, but the market updates will be from the Instrument thread.

"Minor question for instruments. Would it be safe to read between the lines and state: "Reading properties like Name is threadsafe and can be done from any thread"? Instrument's API is pretty small and appears mostly read-only, besides the events."

This depends. As an example a button's Foreground / Background property is a property and would need to be accessed with the ChartControl.Dispatcher (or <window>.Dispatcher).
While the Name property is thread safe to access from any thread.

[carnitron]

Chelsea, thank you for such explicit, clear answers to all my questions. I am set. Thank you.

BTW, I feel like what we just covered here should go into the doc somehow, I'm sure other people would find it helpful.

Cheers,
Peter