Market Depth Map data structure and OnRender() filters. MarketDepthMap, Elasticsearch

[Gorkhaan]

Hello NinjaSupport!

I coded 2 indicators. One loads order book level2 data from Elasticsearch to Ninjatrader, the other saves the data from Ninjatrader to Elasticsearch.
Data is sent every OnBarUpdate on a 1 second chart. I only send the delta (changed levels).


My issue is about OnRender performance rendering RectangleF-s using this data.
Many times it just locks up, freezes. Sometimes it recovers. It's nowhere near as responsive as your official MarketDepthMap indicator.


Current data structure

private DateTime[] startTimes;
private DateTime[] endTimes;
private double[] shiftedStartPrices;  // Shifted with TickSize / 2
private double[] shiftedEndPrices;    // Shifted with TickSize / 2
private double[] volumes;                 // Stored to filter with it on OnRender stage
private double[] calculatedOpacity;   // I use fixed values, like minVolume=100, maxVolume=750. It calculates a ratio and stores it here
private SharpDX.RectangleF[] rectangles; // I create rectangles once on OnDataLoaded stage, these must be modified in OnRender (X, Y, Width, Height)

OnRender Code

        protected override void OnRender(ChartControl chartControl, ChartScale chartScale)
        {
            if (
                RenderTarget == null || RenderTarget.IsDisposed ||
                dxBrush1 == null || dxBrush1.IsDisposed ||
                dxBrush2 == null || dxBrush2.IsDisposed ||
                dxBrush3 == null || dxBrush3.IsDisposed ||
                dxBrush4 == null || dxBrush4.IsDisposed ||
                dxBrush5 == null || dxBrush5.IsDisposed ||
                dxBrush6 == null || dxBrush6.IsDisposed ||
                dxBrush7 == null || dxBrush7.IsDisposed ||
                dxBrush8 == null || dxBrush8.IsDisposed ||
                dxBrush9 == null || dxBrush9.IsDisposed ||
                dxBrush10 == null || dxBrush10.IsDisposed
            ) {
                base.OnRender(chartControl, chartScale);
                return;
            }

            if (!IsInHitTest) {

                // I have played with Antialiasing performance was the same. Didn't help at all.
                previousAntialiasMode = RenderTarget.AntialiasMode;
                RenderTarget.AntialiasMode = SharpDX.Direct2D1.AntialiasMode.Aliased;

                //
                // Here is the main part. These arrays are HUGE. These arrays are "connected" with their indexes
                // meaning startTimes[555] is the same document from elasticsearch which has shiftedEndPrices[555], etc.
                // So index means a specific document. A typical day generates around 450000 documents (using 5 seconds OnBarUpdate runs. I recon 1 second updates can generate more delta documents)
                // I fill these arrays at OnDataLoaded stage, so it only runs once.
                //
                // This part is about filtering the data to render. Only render RectangleF-s where the chart window is visible.
                // This definitely helped a lot, but it still feels sluggish compared to your official MarketDepthMap indicator
                // FSVOnRender means FilterSmallVolumeOnRender. It's still sluggish with or without this
                for (int i=0; i<shiftedStartPrices.Length; i++) {
                    if (
                        (!FSVOnRender || volumes[i] < FixedVolumeMin) ||
                        startTimes[i] < chartControl.FirstTimePainted ||
                        endTimes[i] > chartControl.LastTimePainted ||
                        shiftedEndPrices[i] < chartScale.MinValue ||
                        shiftedStartPrices[i] > chartScale.MaxValue
                    ) {
                        continue;
                    }

                    SharpDX.Vector2 startPoint = new SharpDX.Vector2(chartControl.GetXByTime(startTimes[i]), chartScale.GetYByValue(shiftedStartPrices[i]));
                    SharpDX.Vector2 endPoint = new SharpDX.Vector2(chartControl.GetXByTime(endTimes[i]), chartScale.GetYByValue(shiftedEndPrices[i]));
                    float width = endPoint.X - startPoint.X;
                    float height = endPoint.Y - startPoint.Y;

                    rectangles[i].X = startPoint.X;
                    rectangles[i].Y = startPoint.Y;
                    rectangles[i].Width = width;
                    rectangles[i].Height = height;

                    // DXBrushSelector selects a brush reference based on the opacity (calculated at OnDataLoaded stage)
                    RenderTarget.FillRectangle(rectangles[i],
                        DXBrushSelector(
                            calculatedOpacity[i],
                            ref dxBrush1,
                            ref dxBrush2,
                            ref dxBrush3,
                            ref dxBrush4,
                            ref dxBrush5,
                            ref dxBrush6,
                            ref dxBrush7,
                            ref dxBrush8,
                            ref dxBrush9,
                            ref dxBrush10
                        )
                    );
                }
                RenderTarget.AntialiasMode = previousAntialiasMode;
            }
            base.OnRender(chartControl, chartScale);
        }


        public override void OnRenderTargetChanged()
        {
            if (RenderTarget == null || RenderTarget.IsDisposed) {
                return;
            }
            if (RenderTarget != null)
            {
                try {
                    if (dxBrush1 != null) { dxBrush1.Dispose(); }
                    if (dxBrush2 != null) { dxBrush2.Dispose(); }
                    if (dxBrush3 != null) { dxBrush3.Dispose(); }
                    if (dxBrush4 != null) { dxBrush4.Dispose(); }
                    if (dxBrush5 != null) { dxBrush5.Dispose(); }
                    if (dxBrush6 != null) { dxBrush6.Dispose(); }
                    if (dxBrush7 != null) { dxBrush7.Dispose(); }
                    if (dxBrush8 != null) { dxBrush8.Dispose(); }
                    if (dxBrush9 != null) { dxBrush9.Dispose(); }
                    if (dxBrush10 != null) { dxBrush10.Dispose(); }
                    dxBrush1 = MDStroke1.Brush.ToDxBrush(RenderTarget);
                    dxBrush2 = MDStroke2.Brush.ToDxBrush(RenderTarget);
                    dxBrush3 = MDStroke3.Brush.ToDxBrush(RenderTarget);
                    dxBrush4 = MDStroke4.Brush.ToDxBrush(RenderTarget);
                    dxBrush5 = MDStroke5.Brush.ToDxBrush(RenderTarget);
                    dxBrush6 = MDStroke6.Brush.ToDxBrush(RenderTarget);
                    dxBrush7 = MDStroke7.Brush.ToDxBrush(RenderTarget);
                    dxBrush8 = MDStroke8.Brush.ToDxBrush(RenderTarget);
                    dxBrush9 = MDStroke9.Brush.ToDxBrush(RenderTarget);
                    dxBrush10 = MDStroke10.Brush.ToDxBrush(RenderTarget);
                } catch (Exception e) { }
            }
        }


        // Based on a ratio, calculated from Indicator parameters: FixedMinVolume and FixedMaxVolume
        public SharpDX.Direct2D1.Brush DXBrushSelector(
            double opacity,
            ref SharpDX.Direct2D1.Brush b1,
            ref SharpDX.Direct2D1.Brush b2,
            ref SharpDX.Direct2D1.Brush b3,
            ref SharpDX.Direct2D1.Brush b4,
            ref SharpDX.Direct2D1.Brush b5,
            ref SharpDX.Direct2D1.Brush b6,
            ref SharpDX.Direct2D1.Brush b7,
            ref SharpDX.Direct2D1.Brush b8,
            ref SharpDX.Direct2D1.Brush b9,
            ref SharpDX.Direct2D1.Brush b10
        ) {
            if (opacity <= 0.1) {
                return b1;
            } else if (0.1 < opacity && opacity <= 0.2) {
                return b2;
            } else if (0.2 < opacity && opacity <= 0.3) {
                return b3;
            } else if (0.3 < opacity && opacity <= 0.4) {
                return b4;
            } else if (0.4 < opacity && opacity <= 0.5) {
                return b5;
            } else if (0.5 < opacity && opacity <= 0.6) {
                return b6;
            } else if (0.6 < opacity && opacity <= 0.7) {
                return b7;
            } else if (0.7 < opacity && opacity <= 0.8) {
                return b8;
            } else if (0.8 < opacity && opacity <= 0.9) {
                return b9;
            } else if (0.9 < opacity && opacity <= 1) {
                return b10;
            } else if (opacity > 1) {
                return b10;
            }
            return b1;
        }

So my questions are:

1. Am I wasting resources somewhere in OnRender? Like the brush selector?
2. Is it okay to filter data to render like that in OnRender? Is there a better way to do it?
   1. I tried to filter in OnRenderTargetChanged, but the performance was even worse.
3. Would you kindly share the (semantic?) code regarding to the data storage method (like my arrays) and rendering method from your official MarketDepthMap source code?
4. Any other recommendations is appreciated.

I've attached a screenshot.

[NinjaTrader_Kate]

Hello Gorkhaan,

Thank you for your post.

Unfortunately, we cannot provide any internal source code. We generally wouldn't recommend doing any calculation in OnRender, but it seems from your comments that adding the filtering there is helping somewhat by reducing the amount of changes being made.

One of my colleagues suggested you might try instantiating the rectangles in OnRenderTarget changed and reusing them from there.

If you test this indicator from our publicly available User App Share, do you see similar sluggishness? This one does quite a bit of heavy rendering:

https://ninjatraderecosystem.com/user-app-share-download/dvaluearea/

Also, the ReuseDxBrushesLoopDictionaryExample from this post would be another good one to test:

https://ninjatrader.com/support/forum/forum/ninjatrader-8/indicator-development/98639-performance-optimization#post789606

Do you see the same slowdown with either of the above two scripts?

Thanks in advance; I look forward to assisting you further.

The NinjaTrader Ecosystem website is for educational and informational purposes only and should not be considered a solicitation to buy or sell a futures contract or make any other type of investment decision. The add-ons listed on this website are not to be considered a recommendation and it is the reader's responsibility to evaluate any product, service, or company. NinjaTrader Ecosystem LLC is not responsible for the accuracy or content of any product, service or company linked to on this website.

[Gorkhaan]

Thank you Kate!
I'll have a look and reply with my findings.

/G

[Gorkhaan]

Well I have spend some time on it. I think I have optimized it as much as I can, as it is now.
To speed up the rendering, the only way is to decrease the data to render.

Elasticsearch has an awesome feature called Rollup jobs. I managed to decrease my data points using rollup indicies.

https://www.elastic.co/guide/en/kibana/current/data-rollups.html

With this it's quite close to the built-in MarketDepthMap indicator's performance.

[TCJVQTZtVk]

Just stumble onto this and I'm late to the game so apologize if this is out of context... ElasticSearch is pretty slow, every look at Redis? I've used Redis to manage full order book using scored sorted sets.