Multi-instrument data series error

[Kylemcfar57x]

Im got this error for the first time today that caused the platform to freeze up and could only close through task manager.

'SMA' tried to load additional data. All data must first be loaded by the hosting NinjaScript in its configure state. Attempted to load NQ 06-23 Globex: 1 Minute

Would creating the logic for sma and making it part of the indicator solve this instead of calling on the default sma for nt8

Looking for some advice to troubleshoot since i only encountered this issue once and im having trouble recreating it, i load my dataseries like this

else if (State == State.Configure)
{
AddDataSeries(DataSeries0Instrument.ToString(), BarsPeriodType.Minute, DataSeries0Period);
AddDataSeries(DataSeries1Instrument.ToString(), BarsPeriodType.Minute, DataSeries1Period);
AddDataSeries(DataSeries2Instrument.ToString(), BarsPeriodType.Minute, DataSeries2Period);
AddDataSeries(DataSeries3Instrument.ToString(), BarsPeriodType.Minute, DataSeries3Period);
AddDataSeries(DataSeries4Instrument.ToString(), BarsPeriodType.Minute, DataSeries4Period);
AddDataSeries(DataSeries5Instrument.ToString(), BarsPeriodType.Minute, DataSeries5Period);
AddDataSeries(DataSeries6Instrument.ToString(), BarsPeriodType.Minute, DataSeries6Period);
AddDataSeries(DataSeries7Instrument.ToString(), BarsPeriodType.Minute, DataSeries7Period);
AddDataSeries(DataSeries8Instrument.ToString(), BarsPeriodType.Minute, DataSeries8Period);
AddDataSeries(DataSeries9Instrument.ToString(), BarsPeriodType.Minute, DataSeries9Period);
AddDataSeries(DataSeries10Instrument.ToString(), BarsPeriodType.Minute, DataSeries10Period);
AddDataSeries(DataSeries11Instrument.ToString(), BarsPeriodType.Minute, DataSeries11Period);
AddDataSeries(DataSeries12Instrument.ToString(), BarsPeriodType.Minute, DataSeries12Period);
AddDataSeries(DataSeries13Instrument.ToString(), BarsPeriodType.Minute, DataSeries13Period);​
}

and i set my defaults like this

DataSeries0Instrument = Instrument.GetInstrument("AAPL");
DataSeries0Period = 1;
DataSeries1Instrument = Instrument.GetInstrument("MSFT");
DataSeries1Period = 1;
DataSeries2Instrument = Instrument.GetInstrument("AMZN");
DataSeries2Period = 1;
DataSeries3Instrument = Instrument.GetInstrument("NVDA");
DataSeries3Period = 1;
DataSeries4Instrument = Instrument.GetInstrument("TSLA");
DataSeries4Period = 1;
DataSeries5Instrument = Instrument.GetInstrument("BRK_B");
DataSeries5Period = 1;
DataSeries6Instrument = Instrument.GetInstrument("GOOGL");
DataSeries6Period = 1;
DataSeries7Instrument = Instrument.GetInstrument("XOM");
DataSeries7Period = 1;
DataSeries8Instrument = Instrument.GetInstrument("UNH");
DataSeries8Period = 1;
DataSeries9Instrument = Instrument.GetInstrument("META");
DataSeries9Period = 1;
DataSeries10Instrument = Instrument.GetInstrument("ES 06-23");
DataSeries10Period = 1;
DataSeries11Instrument = Instrument.GetInstrument("NQ 06-23");
DataSeries11Period = 1;
DataSeries12Instrument = Instrument.GetInstrument("YM 06-23");
DataSeries12Period = 1;
DataSeries13Instrument = Instrument.GetInstrument("ES 06-23");
DataSeries13Period = 1;

and i use this type converter for the properties window

[XmlIgnore]
[TypeConverter(typeof(NinjaTrader.Gui.Tools.Instrum entSearchTextConverter))]
[Display(Name="Data Series 1 Instrument", GroupName = "OverWatch Data Series", Order = 0)]
public Instrument DataSeries0Instrument
{ get; set; }

[NinjaTrader_ChelseaB]

Hello Kylemcfar57x,

Unfortunately, this code is unsupported as this violates the warning in the help guide about using variables for AddDataSeries(). Values supplied to AddDataSeries() must be hard coded and cannot be variables.

From the help guide:
"Arguments supplied to AddDataSeries() should be hardcoded and NOT dependent on run-time variables which cannot be reliably obtained during State.Configure (e.g., Instrument, Bars, or user input). Attempting to add a data series dynamically is NOT guaranteed and therefore should be avoided. Trying to load bars dynamically may result in an error similar to: Unable to load bars series. Your NinjaScript may be trying to use an additional data series dynamically in an unsupported manner."

https://ninjatrader.com/support/helpGuides/nt8/adddataseries.htm

"Tips:
4. For the instrument name parameter null could be passed in, resulting in the primary data series instrument being used."

https://ninjatrader.com/support/helpGuides/nt8/adddataseries.htm

That said, the error is likely stating that either this is a hosted indicator that was called by host script such as another indicator or strategy that has not added the same AddDataSeries() calls, or is calling an indicator that has AddDataSeries() and is not calling the same AddDataSeries() calls as the hosted indicator.

From the help guide:
"Should your script be the host for other scripts that are creating indicators and series dependent resources in State.DataLoaded, please make sure that the host is doing the same AddDataSeries() calls as those hosted scripts would. For further reference, please also review the 2nd example below and the 'Adding additional Bars Objects to NinjaScript' section in Multi-Time Frame & Instruments"

https://ninjatrader.com/support/helpGuides/nt8/adddataseries.htm

"Note: To maximize data loading performance, any NinjaScript object (indicator or strategy as host) which references a multi-series indicator which calls AddDataSeries must include it's own calls to AddDataSeries(). For example, if the code above was included in an indicator, and that indicator was referenced in a NinjaScript strategy, then the hosting strategy will need to include the same calls to AddDataSeries(). When the strategy adds the additional Bars objects, the calls to AddDataSeries() within the indicator will be ignored. If the Bars objects are not added by the strategy in such cases, and error will be thrown in the Log tab of the Control Center that would read - "A hosted indicator tried to load additional data. All data must first be loaded by the hosting NinjaScript in its configure state.""
https://ninjatrader.com/support/help...ditionalBarsOb jectToninjascript

[Kylemcfar57x]

So there is no possible workaround for a variable data series, So my only alternative would be to Distribute Updated code everytime the Futures contracts are updated?

[QuantKey_Bruce]

You could look up the current futures contract before you AddDataSeries it (assuming you know which master instrument you're talking about and that is not changing each time). The issue with dynamic AddDataSeries has mostly to do with Strategy Analyzer, which relies on loading the data series in advance of the iterations of the optimizer so that the data series can be shared and does not have to be loaded again for each iteration, so it can't vary. But, the current contract would not vary within one optimizer run (or if it does, I suppose, that's a long run.)

[Kylemcfar57x]

I narrowed down the issue to a threading issue, I hardcoded a multi-data series that has the same deadlock behavior, I also made a single data series that uses the same rendering logic, the single data series version has no problems. originally i was getting a write-read error with the lists i use to store the data so i used, lock (lockObject) where changes are made to the lists, i suspected this was why it was crashing without throwing an exception, i changed the lock (lockObject) to

protected override void OnBarUpdate()
{
    try
    {
        if (CurrentBars[0] < BarsRequiredToPlot || CurrentBars[1] < BarsRequiredToPlot || CurrentBars[2] < BarsRequiredToPlot || CurrentBars[3] < BarsRequiredToPlot)
            return;

        rwLock.EnterReadLock();
        try
        {
            if (BarsInProgress == 0)
            {

            }
            else if (BarsInProgress == 1)
            {

            }
        }
        finally
        {
            rwLock.ExitReadLock();
        }

        rwLock.EnterWriteLock();
        try
        {
            if (BarsInProgress == 0)
            {
                // Update the lists here
            }
            else if (BarsInProgress == 1)
            {
                // Update the lists here
            }
            // Add more else if statements for other bars in progress


        }
        finally
        {
            rwLock.ExitWriteLock();
        }
    }
    catch (Exception ex)
    {
        // Handle exception here
    }
}​

and i did the same for onrender()

now getting this exception

Unhandled exception: Not enough quota is available to process this command Unhandled exception: Write lock may not be acquired with read lock held. This pattern is prone to deadlocks. Please ensure that read locks are released before taking a write lock. If an upgrade is necessary, use an upgrade lock in place of the read lock.

[QuantKey_Bruce]

I would try to reimagine this problem where you don't need locking at all, or you don't need anything as fancy. Maybe there is a way to offload these tasks so you can handle them in another thread that only has a singleton instance so it doesn't need any locking internally? Or, you can use a simple lock(objecthere) for a very brief time instead of upgradable locks?

I have learned the hard way to be extremely sparing with locks at all, and to use them only in very tight, tiny pieces of code where I am 100% confident they cannot be reentrant or cause a deadlock. It's harder than one might think to anticipate every possible way that can happen, especially in a complex multithreaded environment like NinjaTrader running on a machine with a couple of dozen cores where the thread pools are large and concurrency is near constant under heavy load.

The idea of locking all of OnBarUpdate or something (not saying you're doing this, but it's something I tried back a long time ago) I would never even think about today because I know in advance now that's a disaster waiting to happen.

I usually try very hard to write code that has zero locks, or if it has any locks they're on very tight, small pieces of code that I'm 100% confident cannot be stopped and will exit in a deterministic length of time.

[Kylemcfar57x]

ThankYou for the insight, I removed all the locks and im encountering the same issue, now im kind of dumbfounded, I can only close NT through Task Manager im getting this error

Unhandled exception: Write lock may not be acquired with read lock held. This pattern is prone to deadlocks. Please ensure that read locks are released before taking a write lock. If an upgrade is necessary, use an upgrade lock in place of the read lock.

https://forum.ninjatrader.com/forum/ninjatrader-8/indicator-development/95866-chart-rendering-upgradeable-lock-error?_gl=1*1tbew29*_gcl_aw*R0NMLjE2NzUxOTMyMjkuQ2 p3S0NBaUFsZU9lQmhCZEVpd0FmZ21YZndQbEdnTDc3Yk9BYTIz VDJYbGxUenRGUlotdVhaZlVOUkxkbDhQa1g2bEVBQ2tYZDlhNU 9Sb0NjNGNRQXZEX0J3RQ..&t=97057

i glanced over your post here do you think it could be how im disposing my rendering objects or related to how im storing values on a list, i use a loop to iterate through the data to produce the squares

// Format the difference value as a string with a "+" symbol for positive values
        string diffText = diffInTicks >= 0 ? string.Format("+{0:F0}", diffInTicks) : string.Format("{0:F0}", diffInTicks);

        using (var textBrush = new SharpDX.Direct2D1.SolidColorBrush(RenderTarget, new SharpDX.Color(textR, textG, textB)))
        using (var textFormat = new TextFormat(Core.Globals.DirectWriteFactory, "Arial", SharpDX.DirectWrite.FontWeight.Normal, SharpDX.DirectWrite.FontStyle.Normal, fontSize))
        using (var textLayout = new TextLayout(Core.Globals.DirectWriteFactory, "  " + diffText + "  ", textFormat, 200, 20))
                {
                    float textWidth = textLayout.Metrics.Width;
                    float xOffset = (squareSize - textWidth) / 2;
                    // Adjust the Y position for the text to display it under the instrument text
                    float yPosDiffText = yPosText + 14;
                    RenderTarget.DrawTextLayout(new SharpDX.Vector2(xPos + xOffset, yPosDiffText), textLayout, textBrush, SharpDX.Direct2D1.DrawTextOptions.None);
                    textBrush.Dispose(); // dispose of the textBrush object
                    textFormat.Dispose(); // dispose of the textFormat object
                    textLayout.Dispose(); // dispose of the textLayout object
                }​

[QuantKey_Bruce]

It is not necessary to dispose of those three objects in the using() clause because the using disposes of them at the close of its code block... not sure that is the only issue but that is an issue in that you are disposing of them twice.

Further, if you're doing this inside of a loop you should consider making all those things in advance if you can, or whichever ones in advance if you can so you're not iteratively creating and destroying them on each bar or something like that.

[Kylemcfar57x]

yea i thought i didn't need the dispose bc i was using "using" but i threw them in just incase I also moved any manipulation of list data to onbarupdate() ran it through playback several times without incident but then encountered a Unhandled exception: Not enough quota is available to process this command, i was watching ram usage through the playback no increase as it ran through playback, i increased my virtual disk usage limit on my pc and im currently running it again, i will remove the unnecessary dispose() and see if it resolves any problems, currently after changing the virtual disk usage I haven't had any issues but im still going to remove the dispose() to be sure

Appreciate your input it has been helpful

[QuantKey_Bruce]

After you've removed the extra disposes there, I would restart NT8 before you continue testing. Once memory gets all tangled up, it doesn't easily untangle.

[Kylemcfar57x]

removing the disposes reduced the cpu usage by %50 on max speed for playback but i was still getting the read-write lock, so i changed all my lists to arrays to clear up any potential problems with adding and clearing to the lists causing the exception, seems to be working for now, im on the 20th or so iteration of playback with no issue previously i could never get past the 2nd, I may be a bit optimistic to say its fixed but im hopeful lol