DOM Data vs OnMarketDepth divergent data

[zundradaniel]

The DOM and data persisted via OnMarketDepth seems to get widly out of sync when working with column addons and using the OnRender function.

This code is using the PullStack code as a baseline.

The only things I've changed are

1. Instead of rendering Tuple.Item2 (which stores the calculation for the difference between the previous and current depth) I'm rendering Item1 which is simply the current depth.
2. Color the background red if the data stored in the askPriceDepthMap or bidPriceDepthMap does not match the row.BidVoume or row.AskVolume respectively

All of the thread saftey mechanisms are in place as they are in the OnMarketData and OnMarketDepth functions so this isn't a threading issue.

If this column code is giving inaccurate output then so does the PullStack column (only it's not as noticible sinced the value presented is a rapidly changing calculated value)

I'm assuming the OnMarketDepth is the source of truth here. However, this is troubling since the dictionary objects mentioned above are utilizing OnMarketDepth i.e. under the covers the maps should be the nearest source of truth. However, what's presented ot the user as a baseline on a naked DOM is divergenent from the data stored in the data structures. In other words, it seems like when using the DOM we're not actually seeing liqudity as it stands at times.

Lastly, I also want to note, that when the Ninjascript code is refreshed or the column addon loaded initially, the data starts out matching. Then over time, the longer it runs the more out of sync it
becomes.

.

[NinjaTrader_Jesse]

Hello zundradaniel,

If you have used the pulling stacking column as a starting point that won't directly match the values that you see in the dom so that would be expected. That has logic to reset its values so its not a direct representation of the data in the same way. You can see an example of how that display works in the link below.

https://ninjatrader.com/support/helpGuides/nt8/index.html?using_superdom_columns.htm#UnderstandingthePullingStackingColumn

[zundradaniel]

Hi NinjaTrader_Jesse, thanks for the reply.

I'm certain the reset logic isn't the issue. I should have mentioned in my previous post that this was removed as well. I only used the Pull/Stack as a baseline so that you guys know I'm starting from a "trusted" piece of code, but I hit this issue when writing my own liquidity based column addon. Essentially what I did was take the Pull/Stack and simplfy it down to simple data persistence in OnMarketData/Depth, the UI presentation and changed what was being presented to show the Depth vs The Pull/Stack calculation. I've attached the code used below:

I've also screen recorded this in action which can be found here: https://www.youtube.com/watch?v=ZsBmZC4hOTE

namespace NinjaTrader.NinjaScript.SuperDomColumns
{
    public class LiqudityState3 : SuperDomColumn
    {
        private                Dictionary<double, Tuple<long, long>>    askPriceDepthMap;
        private                Dictionary<double, Tuple<long, long>>    bidPriceDepthMap;
        private readonly    object                                    collectionSync                = new object();
        private                FontFamily                                fontFamily;
        private                FontStyle                                fontStyle;
        private                FontWeight                                fontWeight;
        private                Pen                                        gridPen;            
        private                double                                    halfPenWidth;
        private                bool                                    heightUpdateNeeded;
        private                double                                    previousAsk                    = double.MinValue;
        private                double                                    previousBid                    = double.MinValue;
        private                Timer                                    resetTimer;
        private                double                                    textHeight;
        private                Typeface                                typeFace;

        protected override void OnMarketData(MarketDataEventArgs marketData)
        {
            if (marketData.MarketDataType == MarketDataType.Last)
            {
                lock (collectionSync)
                {
                    Tuple<long, long> depthTuple;
                    if (askPriceDepthMap.TryGetValue(marketData.Price, out depthTuple))
                        askPriceDepthMap[marketData.Price] = Tuple.Create(depthTuple.Item1 - marketData.Volume, depthTuple.Item2 - marketData.Volume);
                    if (bidPriceDepthMap.TryGetValue(marketData.Price, out depthTuple))
                        bidPriceDepthMap[marketData.Price] = Tuple.Create(depthTuple.Item1 - marketData.Volume, depthTuple.Item2 - marketData.Volume);
                }
            }
        }

        protected override void OnMarketDepth(MarketDepthEventArgs marketDepth)
        {
            if (marketDepth.Position >= SuperDom.DepthLevels)
                return;

            lock (collectionSync)
                if (marketDepth.MarketDataType == MarketDataType.Ask)
                {
                    if (marketDepth.Operation == Cbi.Operation.Add)
                        askPriceDepthMap[marketDepth.Price] = Tuple.Create(marketDepth.Volume, 0L);
                    else if (marketDepth.Operation == Cbi.Operation.Update)
                    {
                        Tuple<long, long> depthTuple;
                        if (askPriceDepthMap.TryGetValue(marketDepth.Price, out depthTuple))
                            askPriceDepthMap[marketDepth.Price] = Tuple.Create(depthTuple.Item1, marketDepth.Volume - depthTuple.Item1);
                        else
                            askPriceDepthMap[marketDepth.Price] = Tuple.Create(marketDepth.Volume, 0L);
                    }
                }
                else if (marketDepth.MarketDataType == MarketDataType.Bid)
                {
                    if (marketDepth.Operation == Cbi.Operation.Add)
                        bidPriceDepthMap[marketDepth.Price] = Tuple.Create(marketDepth.Volume, 0L);
                    else if (marketDepth.Operation == Cbi.Operation.Update)
                    {
                        Tuple<long, long> depthTuple;
                        if (bidPriceDepthMap.TryGetValue(marketDepth.Price, out depthTuple))
                            bidPriceDepthMap[marketDepth.Price] = Tuple.Create(depthTuple.Item1, marketDepth.Volume - depthTuple.Item1);
                        else
                            bidPriceDepthMap[marketDepth.Price] = Tuple.Create(marketDepth.Volume, 0L);
                    }
                }
        }

        protected override void OnRender(DrawingContext dc, double renderWidth)
        {
            // This may be true if the UI for a column hasn't been loaded yet (e.g., restoring multiple tabs from workspace won't load each tab until it's clicked by the user)
            if (gridPen == null)
            {
                if (UiWrapper != null && PresentationSource.FromVisual(UiWrapper) != null)
                {
                    Matrix m            = PresentationSource.FromVisual(UiWrapper).CompositionTarget.TransformToDevice;
                    double dpiFactor    = 1 / m.M11;
                    gridPen                = new Pen(Application.Current.TryFindResource("BorderThinBrush") as Brush,  1 * dpiFactor);
                    halfPenWidth        = gridPen.Thickness * 0.5;
                }
            }

            if (fontFamily != SuperDom.Font.Family
                || (SuperDom.Font.Italic && fontStyle != FontStyles.Italic)
                || (!SuperDom.Font.Italic && fontStyle == FontStyles.Italic)
                || (SuperDom.Font.Bold && fontWeight != FontWeights.Bold)
                || (!SuperDom.Font.Bold && fontWeight == FontWeights.Bold))
            {
                // Only update this if something has changed
                fontFamily            = SuperDom.Font.Family;
                fontStyle            = SuperDom.Font.Italic ? FontStyles.Italic : FontStyles.Normal;
                fontWeight            = SuperDom.Font.Bold ? FontWeights.Bold : FontWeights.Normal;
                typeFace            = new Typeface(fontFamily, fontStyle, fontWeight, FontStretches.Normal);
                heightUpdateNeeded    = true;
            }
            double verticalOffset    = -gridPen.Thickness;
            double pixelsPerDip        = VisualTreeHelper.GetDpi(UiWrapper).PixelsPerDip;

            lock (SuperDom.Rows)
                foreach (PriceRow row in SuperDom.Rows)
                {
                    if (renderWidth - halfPenWidth >= 0)
                    {
                        Rect rect = new Rect(-halfPenWidth, verticalOffset, renderWidth - halfPenWidth, SuperDom.ActualRowHeight);

                        GuidelineSet guidelines = new GuidelineSet();
                        guidelines.GuidelinesX.Add(rect.Left    + halfPenWidth);
                        guidelines.GuidelinesX.Add(rect.Right    + halfPenWidth);
                        guidelines.GuidelinesY.Add(rect.Top        + halfPenWidth);
                        guidelines.GuidelinesY.Add(rect.Bottom    + halfPenWidth);

                        dc.PushGuidelineSet(guidelines);

                        if (DisplayType == PullingStackingDisplayType.BidAsk)
                        {
                            Rect bidRect = new Rect(-halfPenWidth, verticalOffset, (renderWidth / 2) - halfPenWidth, SuperDom.ActualRowHeight);
                            Rect askRect = new Rect((renderWidth / 2) - halfPenWidth, verticalOffset, (renderWidth / 2) - halfPenWidth, SuperDom.ActualRowHeight);
                            
                            Brush bidBackColor = Brushes.Transparent;
                            Brush askBackColor = Brushes.Transparent;                            
                            
                            Tuple<long, long> bidVolume;
                            Tuple<long, long> askVolume;
                            
                            if (bidPriceDepthMap.TryGetValue(row.Price, out bidVolume) && row.BidVolume > 0)
                            {
                                if (bidVolume.Item1 != row.BidVolume)
                                {
                                    bidBackColor = Brushes.Red;
                                }
                            }
                            
                            if (askPriceDepthMap.TryGetValue(row.Price, out askVolume) && row.AskVolume > 0)
                            {
                                if (askVolume.Item1 != row.AskVolume)
                                {
                                    askBackColor = Brushes.Red;
                                }
                            }
                            
                            dc.DrawRectangle(bidBackColor, null, bidRect);
                            dc.DrawRectangle(askBackColor, null, askRect);
                        }
        

                        dc.DrawLine(gridPen, new Point(-gridPen.Thickness, rect.Bottom), new Point(renderWidth - halfPenWidth, rect.Bottom));
                        dc.DrawLine(gridPen, new Point(rect.Right, verticalOffset), new Point(rect.Right, rect.Bottom));

                        // Write bid/ask pulling/stacking values
                        if (SuperDom.IsConnected
                            && !SuperDom.IsReloading
                            && State == State.Active)
                        {
                            lock (collectionSync)
                            {
                                if (DisplayType == PullingStackingDisplayType.BidAsk || DisplayType == PullingStackingDisplayType.Bid)
                                {
                                    Tuple<long, long> bidVolume;
                                    if (bidPriceDepthMap.TryGetValue(row.Price, out bidVolume) && row.BidVolume > 0)
                                    {
                                        fontFamily    = SuperDom.Font.Family;
                                        typeFace    = new Typeface(fontFamily, SuperDom.Font.Italic ? FontStyles.Italic : FontStyles.Normal, SuperDom.Font.Bold ? FontWeights.Bold : FontWeights.Normal, FontStretches.Normal);

                                        if (renderWidth - 6 > 0)
                                        {
                                            if (row.BidVolume != bidVolume.Item1)
                                            {
                                                Print($"[Bid] DOM and MarketDepth values do not match: Price: {row.Price}, OnMarketDepthMap: {bidVolume.Item1}, Row Depth: {row.AskVolume}: Diff: {bidVolume.Item1 - row.BidVolume}");
                                            }
                                            
                                            FormattedText bidText = new FormattedText(bidVolume.Item1.ToString(Core.Globals.GeneralOptions.CurrentCulture), Core.Globals.GeneralOptions.CurrentCulture, FlowDirection.LeftToRight, typeFace, SuperDom.Font.Size, BidForeColor, pixelsPerDip) { MaxLineCount = 1, MaxTextWidth = (renderWidth / 2) - 6, Trimming = TextTrimming.CharacterEllipsis };
                                            // Getting the text height is expensive, so only update it if something's changed
                                            if (heightUpdateNeeded)
                                            {
                                                textHeight = bidText.Height;
                                                heightUpdateNeeded = false;
                                            }

                                            dc.DrawText(bidText, new Point(0 + 4, verticalOffset + (SuperDom.ActualRowHeight - textHeight) / 2));
                                        }
                                    }
                                }

                                if (DisplayType == PullingStackingDisplayType.BidAsk || DisplayType == PullingStackingDisplayType.Ask)
                                {
                                    Tuple<long, long> askVolume;
                                    if (askPriceDepthMap.TryGetValue(row.Price, out askVolume) && row.AskVolume > 0)
                                    {
                                        fontFamily    = SuperDom.Font.Family;
                                        typeFace    = new Typeface(fontFamily, SuperDom.Font.Italic ? FontStyles.Italic : FontStyles.Normal, SuperDom.Font.Bold ? FontWeights.Bold : FontWeights.Normal, FontStretches.Normal);

                                        if (renderWidth - 6 > 0)
                                        {
                                            if (row.AskVolume != askVolume.Item1)
                                            {
                                                Print($"[Ask] DOM and MarketDepth values do not match: Price: {row.Price}, OnMarketDepthMap: {askVolume.Item1}, Row Depth: {row.AskVolume}: Diff: {askVolume.Item1 - row.AskVolume}");
                                            }
                                            FormattedText askText = new FormattedText(askVolume.Item1.ToString(Core.Globals.GeneralOptions.CurrentCulture), Core.Globals.GeneralOptions.CurrentCulture, FlowDirection.LeftToRight, typeFace, SuperDom.Font.Size, AskForeColor, pixelsPerDip) { MaxLineCount = 1, MaxTextWidth = (renderWidth / 2) - 6, Trimming = TextTrimming.CharacterEllipsis };
                                            // Getting the text height is expensive, so only update it if something's changed
                                            if (heightUpdateNeeded)
                                            {
                                                textHeight = askText.Height;
                                                heightUpdateNeeded = false;
                                            }

                                            dc.DrawText(askText, new Point(renderWidth / 2 + 4, verticalOffset + (SuperDom.ActualRowHeight - textHeight) / 2));
                                        }
                                    }
                                }
                            }
                        }

                        dc.Pop();
                        verticalOffset += SuperDom.ActualRowHeight;
                    }
                }
        }
    }
}​

[NinjaTrader_Jesse]

Hello zundradaniel,

It is not clear what you are trying to achieve, are you trying to use the same values that you see in the dom column? If so you should not start with the pulling/stacking column as that does not represent that information directly.

[zundradaniel]

NinjaTrader_Jesse Forget pulling/stacking for a moment haha. As I mentioned, that was just a baseline to demonstrate the issue with all the "pulling/stacking" logic pulled out. My issue is the data that comes through and is displayed on the DOM is different from the data that comes through raw via OnMarketDepth.

OnMarketDepth is as close as the NT API allows us to get to the "raw" data and hence is the source of truth at any given time. So if the DOM says something different it's not representing that truth source but rather is presenting a delayed (sometimes signficantly delayed) view of the state of liquditity.

For example, if OnMarketDepth has aggreated 500 on the inside bid, but the DOM shows 20 that's an issue right? In the market, at that moment there are 500 lots on the inside bid, but I think there is 20 because that's what the DOM is telling is sitting there.

[zundradaniel]

NinjaTrader_Jesse After digging into this some more, I've found that the DOM and the OnMarketDepth data to in fact match. The issue is in the OnRender call inside of Column addons. It looks like that call is delayed enough for the data to get out of sync given the thread lock present. I don't think there is anything that can be done to fix this in the current architecture as the only real fix would be to use the data populated in the maps in the same thread as in the OnMarketDepth call. This can be done with indicators, but not on Column addons. It does mean that any code written as a column addon (such as the pull/stack) will always have the possibility of being slightly out of sync. Knowing this, I will personally stop relying on column addons and stick to indciators and the DOM itself.

[NinjaTrader_Jesse]

Hello zundradaniel,

To use OnmarketDepth you need to utilize all events including the update and remove events. You can see a sample of working with level 2 data here:

https://developer.ninjatrader.com/docs/desktop/creating_your_own_level_ii_data_book_accessing_market_depth

[zundradaniel]

NinjaTrader_Jesse it isn't examples, I need. I'm very familiar with the NT code as I've been writing indicators for 5 years and have been a professional software engineer for 3x that. This is an archiectural issue as I mentioned above. Thanks for considering, it, but this is a fundamental flaw in the way column addons work.