Bug in OnMarketData on 8.0.27.1

[bltdavid]

I've managed to come across a bug that consistently
reproduces in 8.0.26.1 and 8.0.27.1.

I did not test the 8.1.1.0 release.

The steps are bit obscure.

I create one chart with just a PriceLine indicator, then
duplicated the chart to a new window.

Then I cycle through connecting/disconnecting through
4 connections ... I found the bug will manifest on the
4th connection every single time (bug could manifest
with fewer connections, or a different connect order,
I dunno).

The bug manifests with an error in the log file,
Indicator 'Price line': Error on calling 'OnMarketData' method on bar -1:
You are accessing an index with a value that is invalid since (yada yada)

I made a video.

https://screencast-o-matic.com/watch/c0ehbTVyW8D

[bltdavid]

Video update complete.

[QuantKey_Bruce]

In the inbuilt indicator's OnMarketData method, you could add this line at the top:

if (CurrentBar < 0 || Values.Count < 3) return;

--

Bruce DeVault
QuantKey LLC
www.quantkey.com

[bltdavid]

Yeah, I could do that -- but I'd lose that change on an upgrade, reinstall, or repair.

Updating that indicator privately is not a real fix, certainly I can't tell friends & family
(much less customers) I help to support to do that ... yikes!

An official fix needs to come from NinjaTrader.

If that's how NT ultimately fixes this issue, well, that's their prerogative, of course.

[QuantKey_Bruce]

Of course. I'm just pointing out what needs to be done more specifically.

--

Bruce DeVault
QuantKey LLC

https://www.quantkey.com

[bltdavid]

I hope engineering takes into consideration that the root cause
should be investigated, and that the solution could be much deeper
than just adding a bit of cosmetic guard code to OnMarketData.

Furthermore, if NT is really going to 'punt' this issue and just take
the route of adding 'guard code' to eliminate the error, they'll need
to investigate every indicator that uses OnMarketData.

These are the ones I know about,

$ grep OnMarketData @*.cs
@BuySellPressure.cs: protected override void OnMarketData(MarketDataEventArgs e)
@BuySellVolume.cs: protected override void OnMarketData(MarketDataEventArgs e)
@NetChangeDisplay.cs: protected override void OnMarketData(Data.MarketDataEventArgs marketDataUpdate)
@PriceLine.cs: protected override void OnMarketData(MarketDataEventArgs e)
@VolumeProfile.cs: protected override void OnMarketData(MarketDataEventArgs e)

And, most likely, the entire 'Order Flow +' suite will also need to be looked at.

[NinjaTrader_BrandonH]

Hello bltdavid,

Thanks for your post.

We are attempting to reproduce the behavior you reported and are researching this matter further.

We will provide more follow up information as soon as we have finished reproducing and researching this topic.

Thanks for your patience; I look forward to assisting further.

[NinjaTrader_BrandonH]

Hello bltdavid,

Thanks for your patience.

I was able to reproduce this behavior in NinjaTrader 8 (8.0.27.1 and 8.1.1.0).

This seems to be caused by connecting to the Playback connection with no Market Replay data downloaded for the instrument, disconnecting from the Playback connection, and then connecting to a live data feed connection.

To prevent this error, you would need to have Market Replay data downloaded when connecting to the Playback connection.

If we download Market Replay data for the instrument, connect to the Playback connection, let the chart load, disconnect from the Playback connection, and connect to a live data feed connection, the error does not appear in the Log tab of the Control Center.

See this demonstration video: https://brandonh-ninjatrader.tinytak...MV8yMTA5MzAxMQ

Please let me know if I may assist further.

[QuantKey_Bruce]

I hope engineering takes into consideration that the root cause
should be investigated, and that the solution could be much deeper
than just adding a bit of cosmetic guard code to OnMarketData.

Errors of this general sort also can be caused by a race condition - we often see things like low-probability exceptions where things happen like OnRender is running, but while it's running, the indicator goes through a state change to Terminated and cleans up its resources, causing OnRender to crash because the resources it was using were cleaned up. The only solution for that is to not undertake state changes until OnRender finishes, but that could cause other issues. Still, it's this sort of thing that is probably the "root cause" in question here. You're disconnecting and reconnecting over and over, causing the indicator to go through state changes and reinitialize all of its things, but in the interim, market data events are being fired off in another thread. With some bad luck, it's entirely possible the state changes will happen concurrently with OnMarketData, and pandemonium ensues. So yes, guard code is not "enough" but it would be better than nothing if they can't stop the possibility of them being concurrent.

--

Bruce DeVault
QuantKey LLC
www.quantkey.com

[bltdavid]

This seems to be caused by connecting to the Playback connection with no Market Replay data downloaded for the instrument, disconnecting from the Playback connection, and then connecting to a live data feed connection.

Not surprised.
This matches some speculation I had later.

To prevent this error, you would need to have Market Replay data downloaded when connecting to the Playback connection.

Hahahhhaa ... yeah, uh, so when this happens you
want customers to load some market replay data
in order to prevent the error? Not gonna happen.

I understand what you're saying, but the onus on
'preventing the error' lies with NinjaTrader, not me.

You gave me nice chuckle, thanks.

Btw,
If you're not going to fix it, you can just say so.

[bltdavid]

I would suspect this is simply a race condition - we often see things like low-probability exceptions where things happen like OnRender is running, but while it's running, the indicator goes through a state change to Terminated and cleans up its resources, causing OnRender to crash because the resources it was using were cleaned up. The only solution for that is to not undertake state changes until OnRender finishes, but that could cause other issues. Still, it's this sort of thing that is probably the "root cause" in question here. You're disconnecting and reconnecting over and over, causing the indicator to go through state changes and reinitialize all of its things, but in the interim, market data events are being fired off in another thread. With some bad luck, it's entirely possible the state changes will happen concurrently with OnMarketData, and pandemonium ensues. So yes, guard code is not "enough" but it would be better than nothing if they can't stop the possibility of them being concurrent.

Thanks, Bruce -- good insights!

-=o=-

Actually, I kinda doubt there's a race condition here.

All we do is disconnect from Playback and connect to
a live data feed connection, per my discovery.

The reason the error occurred, it seems, per Brandon,
is because no playback data is available -- meaning,
if playback data were available, no error would occur.

In this specific case,
I may be wrong, but there's no resource in contention
to cause a 'race' between threads. It's the fact that the
data itself is simply not there, and (I assume) this lack
of data slips past several layers of internal code checks,
until finally poof! we see OnMarketData is called with
bar -1, and the no playback data condition slips through
as an obscure error to the user.

The final layer of checking for the 'missing playback
data' is the 'missing guard code' in the OnMarketData
routine itself -- I agree this is certainly good enough
to prevent the error message.

-=o=-

Interesting story about the OnRender. Is that history
available in a forum thread somewhere? Got a link?

[NinjaTrader_BrandonH]

Hello bltdavid,

Thanks for your note.

I have made the Development team aware of the behavior so that they may look into it further and possibly implement a fix for it.

Once the Development team gets back to me after they investigate this matter I will reach back out to you with more information.

Thanks for bringing the behavior to our attention.

[QuantKey_Bruce]

Interesting story about the OnRender. Is that history
available in a forum thread somewhere? Got a link?

No, I don't have a link handy - but it's something that's happened. It's one of those sorts of things where if you restart NinjaTrader and let it load your large number of slow-loading charts or whatever and while it's loading you X out of NinjaTrader or disconnect the datafeed and reconnect we'll see a message in the log - not every time but once in a while. It's been that way for some years now so before I read Brandon's reply that he had reproduced your specific issue, I chimed in that something vaguely similar has happened here. In hindsight now, you're right of course - you're not experiencing this particular issue and I should have read Brandon's comment before throwing this in there since it now seems to be an unrelated issue. I'll try to remember to capture some details the next time I see it so I can create a proper thread about that specifically.

--

Bruce DeVault
QuantKey LLC
www.quantkey.com