Lifecycle, State Default, Building List of Indicator Names, Registry Storage Concept?

[balltrader]

I read the article about the lifecycles, and how, when the list of Indicators is requested by user right click on chart, an instance of each Indicator is invoked for the purpose of building the list of Indicators.

Code in the State=State.SetDefaults gets run during that operation, which made me think it is also running a lot of AddPlot statements, etc...

So I am wondering why not use a type of registry storage concept, to hold a list of all active Indicators, to build the list a lot faster without invoking an instance of each indicator just to get it's "name" ? To clarify, I don't mean using the Windows Registry, but storing the list of Indicators (and other custom objects) in a simple List, stored in the SQLite database or even a local file.

The list would only need to be "Rebuilt" after a compile / import operation.

That seems like it would save some time in building the list when configuring Indicators.


Then when an Indicator is selected from the list, only that single Indicator needs to have an instance invoked, so it's properties can be set in the UI, and then cloned / attached to the chart.

I'm sure there is other things happening behind the scenes and you can't reveal the secret sauce of the Platform, etc. I'm just trying to shut down the nagging voice in the back of my head for an hour.

[NinjaTrader_ChrisL]

Hello, thanks for writing in. Unfortunately, we have no public information about this aspect of the platform as this is coming from code that is protected in the binary files of NinjaTrader. There could be many undocumented reasons for the need to load the indicator/strategy list in this fashion.

[QuantKey_Bruce]

There's something worse that you may not realize yet about this particular platform approach.

Suppose you have a strategy that in State.Configure loads a few dozen different data series with AddDataSeries to use them in the strategy and all of the strategy's nested indicators.

Subsequent to compiling that, every time you go into your Strategies list, just pulling up the list seems to be slower, and then when you instantiate the strategy by double-clicking it or clicking it and choosing Add, there is this very long pause... note that you haven't applied it to the chart yet, but because those AddDataSeries statements exist, and it is in State.Configure even though there is no instance of it on the chart (yet), it is doing something there that takes a lot of time before you get control of the UI to even configure the input parameters of the not-yet applied strategy.

[balltrader]

NinjaTrader_ChrisL I should have posted this under the category "Suggestions / Feedback". Can you move it there, and then can you create a mythical Ticket to be reviewed by the Senior Software Engineers?

QuantKey_Bruce That is the scenario / troubled situation I was anticipating...

[NinjaTrader_ChrisL]

Hi balltrader, I will move this post and create a feature request for you.

[jeronymite]

QuantKey_Bruce and balltrader The symptoms you describe are probably related to this, a major performance issue that has been there for a long time, and still most definitely needs to be addressed.

Thanks.

[QuantKey_Bruce]

QuantKey_Bruce and balltrader The symptoms you describe are probably related to this, a major performance issue that has been there for a long time, and still most definitely needs to be addressed.

Thanks.

Note though that the pause I describe happens without disconnecting or reconnecting. It happens every time you open the Strategies dialog (or Indicators, if you have any custom indicators that add a lot of data series in State.Configure). It could be that the connection-related pause and this have something in common internally in that they both involve loading a bunch of data series.

[jeronymite]

... custom indicators that add a lot of data series in State.Configure). It could be that the connection-related pause and this have something in common internally in that they both involve loading a bunch of data series.

Exactly. Adding data series is excruciatingly non-performant!

Thanks.

[QuantKey_Bruce]

Exactly. Adding data series is excruciatingly non-performant!

Yes, this is a slow operation, but the I would also emphasize that the design decision to have the needed data series added in State.Configure, together with the design decision to call State.Configure in places like the Indicators and Strategies dialogs, means that the data series operations are being run in places where they are meaningless. The low-hanging fruit is to not add any data series unless this instance is an actual instantiation on a hosting window like a chart. NinjaTrader could simply set a flag for whether this is a "real" instance that will ever reach State.DataLoaded or not, and if it is not, ignore AddDataSeries. That alone seems like it would speed it up tremendously.

[jeronymite]

QuantKey_Bruce All good things you suggest.

Nonetheless, from bitter experience every day, I can tell you that it is the act of subscribing to data that makes the platform crawl. I do this in an AddOn, so purely under my control when it happens ... and even just subscribing to one Instrument can still freeze the platform.

And this is not some odd Instrument via a convoluted Connection. This is things like EURUSD from NinjaTrader's own servers. Doing several in parallel is inviting the "arctic winter" of frozen windows and other NT functionality for many seconds, even as long as some tens of seconds, with windows being flagged as "Not Responding" and window "frames" turning white! And before there is even a hint of the words "computer performance issue", it is unequivocally not that! And that comes from more than four decades of professional technical work with IT.

It's the subscription process itself, not specifically Strategies or Indicators, although their implementation would tend to aggravate the effect. It can happen with standard NT actions unrelated to Strategies and Indicators. Try opening an FX Correlation or FX Board containing the 28 majors before opening a Connection, then Connect, and you probably could have a nice European holiday before the platform is running "at speed" again.

As noted in my post about it, it needs to be looked at "square on" and addressed to fix it.

Thanks.

[balltrader]

gosh that sounds bad. do you see these freezes in the trace files? have you tried testing the same endpoint to the connection using a websocket or even a tracert to test if you still experience the hang?

[jeronymite]

balltrader Thanks for your ideas. Frankly, after seeing this being utterly reliably reproducible over many years, using different Connections, different Instruments, different Internet connectivity, etc, I simply suffer it "as is" until someone in NinjaTrader takes it seriously and investigates and finally fixes (even improves would be a start) the performance in this respect.

Even in my original post, you can see that NinjaTrader_Jason said:

Unfortunately this behavior can occur when you connect in NinjaTrader. It can become unresponsive during the connection period.

Still, it's "only just two years" since I posted that, so I guess I can look forward to at least a few more years of waiting, based on trends for such things.

Thanks.

[QuantKey_Bruce]

I have the same thing happen - I have some use cases where I open up few dozen charts, each of which loads a bunch of timeframes with AddDataSeries. It is on a very high end machine (64 threads and 128GB of RAM if I recall correctly so it never comes anywhere near saturating its hardware resources). The platform is completely unresponsive when opening this workspace for perhaps 20 minutes each time. You can end process NinjaTrader with task manager but there is nothing else you can do but wait it out and hope nothing goes wrong.

It would appear the UI threads simply aren't getting any time slice while it's doing whatever it is doing during this time.

[jeronymite]

Exactly, QuantKey_Bruce! Exactly! And it's been that way for "waaaaaay toooooo long", and even acknowledged by NinjaTrader Support staff.

It would seem to be related to use of the UI thread (any click will initially use that thread) and unless the work requested by a click is off-loaded to another "worker" thread, the UI thread is monopolised, leading to symptoms that are of the nature we are describing if that work is "substantial". Of course, it could be this or other things or a combination of things.

It's a grievous behaviour, because it is both extremely disruptive, but also quite avoidable. One can only hope that now the "new order" of NinjaTrader software is revealed, and once the immediate concerns on that front are resolved, that serious work will occur on the Desktop platform to bring it up-to-date with its underlying technology, and address such highly non-performant aspects as this, amongst other needs.

Thanks.