Large memory use

[ETFVoyageur]

I have a custom indicator that is taking a lot of memory when I use it in a Market Analyzer, so I took a look at why that might be. I have isolated it to a single line of code causing the problem, and am looking for advice on how to deal with that. My Market Analyzer setup is constant for all tests, except for the length of the linear regression calculation. The setup is:

• Running a fresh instance of 64-bit NinjaTrader for each test
• Market Analyzer has 4 columns: Description, Instrument, SMA(1), and the indicator being tested. The first three have little effect on performance.
• Market Analyzer has 1600 rows
• Data is daily
• Market Analyzer | Properties | # of bars to look back = 512
• Indicator setting in "Columns" dialog: # of bars to look back = 512
• Indicator setting in "Columns" dialog: Maximum bars look back = 256
• I ran tests with regression periods of 252, 126, and 63

I reduced OnBarUpdate() to the following -- I do not know how to make the code much simpler:

        protected override void OnBarUpdate()
        {
            if (CurrentBar<2) return;
            //IndicatorLine.Set(LinRegSlope(Math.Min(CurrentBar,period1Bars))[0]);
            //IndicatorLine.Set(50);
            return;
        }

I restarted NinjaTrader for each test, so there would be no residual memory artifacts from a previous run. I uncommented the simpler setting line ( IndicatorLine.Set(50) ) for the first trial. I set the indicator period length to 252, although it is not used in this case. The result ran quite quickly, did not consume much memory, and served to illustrate that the basic setup is not the problem, nor are ancillary things such as Initialize() and OnStartup():

• The run took 0:15 (15 seconds) and the working set grew to 0.7GB

Next I reversed the commenting for the remaining tests, making the linear regression call operational. Here are the results of three runs, restarting NinjaTrader between runs:

• Period=252 time=5:29 working set=10.3GB
• Period=126 time=1:59 working set=5.6GB
• Period=63 time=1:02 working set=3.1GB

-- There would have been the same number of calls to linear regression in all tests -- what varied was the length of the linear regression being calculated. The longer the linear regression, the more memory got consumed.

-- In each case, the working set started out small, and increased steadily as the code ran. In other words, memory was being consumed as the code executed. My code is not allocating any memory as it executes -- so it must be the library code. It looks as if each call to LinRegSlope() consumes an amount of memory that varies with the length of the regression.

In case anyone is wondering, the machine is an i7 quad core 3.4GHz with 16GB RAM, running Windows 8.1, so I doubt that the machine itself is causing problems. For example, the tests never got into paging.

OK -- how should I do the task at hand without growing to such a ridiculous working set,?

--EV

[NinjaTrader_CalH]

Hello EV,

Thank you for your post.

Market Analyzer has 1600 rows

Is this the correct amount? This is really high as most data providers have a 500 symbol limit.

Would you mind sharing the entire script that you are running these tests on, so that I can test this on my end?

[ETFVoyageur]

Hello EV,

Thank you for your post.


Is this the correct amount? This is really high as most data providers have a 500 symbol limit.

Would you mind sharing the entire script that you are running these tests on, so that I can test this on my end?

Yes, that is the correct amount -- I am using Kinetick EOD for the data and they seem happy. It is a list of many ETFs so I can rank them on a custom indicator of my own.

Script -- I'll get back to you. I'm not worried as much about privacy as I am about practicality. I'll try to get something together for you.

--EV

[ETFVoyageur]

Cal,

I have put together a very simplified example:

• Keep things simple for you
• Ensure the test is easily reproducible
• Verify that I am not getting hit by something neither of us thought of

I started by using the wizard to create a new indicator, and then made minimal modifications to it -- mainly adding the OnBarUpdate() code and removing things like comments (to keep the test case text minimal).

My test case:

• Start a new instance of 64-bit NinjaTrader
• Workspace has only a Control Center
• Create a new Market Analyzer
  • Two columns: Instrument, MyTestIndicator
  • Market Analyzer Properties
    • # of bars to look back: 512
    
    
  • Indicator
    • Type: Day
    • # of bars to look back: 512
    • Maximum bars look back: TwoHundredFiftySix
    • (Indicator) Number of bars: 252
    
    
  
  
• Use Market Analyzer's "Add Instrument List" to load my list of ETFs (about 1600)
  • 4:16 minutes until the values were populated
  • 10.1 GB working set when the values were populated
  • These values were when populating from the local cache -- there was no connection to any data provider
  
  

Time and working set vary in a minor way from run to run. Time is always over 4 minutes, and working set always climbs steadily to over 10 GB. The computer is an i7 quad core 3.4 GHz with 16GB RAM, running Windows 8.1 (with all patches). I doubt the computer is the problem, and I have never seen it paging.

How would you like me to provide this example? Just let me know ...

• I have pasted the indicator code below
• Would you also like my ETF list?
• Would you also like my configured Market Analyzer?

--EV

#region Using declarations
using System;
using System.ComponentModel;
using System.Diagnostics;
using System.Drawing;
using System.Drawing.Drawing2D;
using System.Xml.Serialization;
using NinjaTrader.Cbi;
using NinjaTrader.Data;
using NinjaTrader.Gui.Chart;
#endregion

// This namespace holds all indicators and is required. Do not change it.
namespace NinjaTrader.Indicator
{
    [Description("This is to test the huge memory consumed by LinRegSlope()")]
    public class MyTestIndicator : Indicator
    {
        protected override void Initialize()
        {
            CalculateOnBarClose    = false;
            Overlay                = false;
        }
        protected override void OnBarUpdate()
        {
            if (CurrentBar<2) return;
            IndicatorLine.Set(LinRegSlope(Math.Min(CurrentBar,period1Bars))[0]);
            //IndicatorLine.Set(50);
        }

        [Browsable(false)]    // this line prevents the data series from being displayed in the indicator properties dialog, do not remove
        [XmlIgnore()]        // this line ensures that the indicator can be saved/recovered as part of a chart template, do not remove
        public DataSeries IndicatorLine
        {
            get { return Values[0]; }
        }
    
        private int period1Bars = 21;
        [Description("Number of bars used to calculate the linear regression slope")]
        [Gui.Design.DisplayName ("Number of bars")]
        public int Period1Bars
        {
            get { return period1Bars; }
            set { period1Bars = Math.Max(1, value); }
        }
    }
}

#region NinjaScript generated code. Neither change nor remove.
// This namespace holds all indicators and is required. Do not change it.
namespace NinjaTrader.Indicator
{
    public partial class Indicator : IndicatorBase
    {
        private MyTestIndicator[] cacheMyTestIndicator = null;

        private static MyTestIndicator checkMyTestIndicator = new MyTestIndicator();

        /// <summary>
        /// This is to test the huge memory consumed by LinRegSlope()
        /// </summary>
        /// <returns></returns>
        public MyTestIndicator MyTestIndicator()
        {
            return MyTestIndicator(Input);
        }

        /// <summary>
        /// This is to test the huge memory consumed by LinRegSlope()
        /// </summary>
        /// <returns></returns>
        public MyTestIndicator MyTestIndicator(Data.IDataSeries input)
        {
            if (cacheMyTestIndicator != null)
                for (int idx = 0; idx < cacheMyTestIndicator.Length; idx++)
                    if (cacheMyTestIndicator[idx].EqualsInput(input))
                        return cacheMyTestIndicator[idx];

            lock (checkMyTestIndicator)
            {
                if (cacheMyTestIndicator != null)
                    for (int idx = 0; idx < cacheMyTestIndicator.Length; idx++)
                        if (cacheMyTestIndicator[idx].EqualsInput(input))
                            return cacheMyTestIndicator[idx];

                MyTestIndicator indicator = new MyTestIndicator();
                indicator.BarsRequired = BarsRequired;
                indicator.CalculateOnBarClose = CalculateOnBarClose;
#if NT7
                indicator.ForceMaximumBarsLookBack256 = ForceMaximumBarsLookBack256;
                indicator.MaximumBarsLookBack = MaximumBarsLookBack;
#endif
                indicator.Input = input;
                Indicators.Add(indicator);
                indicator.SetUp();

                MyTestIndicator[] tmp = new MyTestIndicator[cacheMyTestIndicator == null ? 1 : cacheMyTestIndicator.Length + 1];
                if (cacheMyTestIndicator != null)
                    cacheMyTestIndicator.CopyTo(tmp, 0);
                tmp[tmp.Length - 1] = indicator;
                cacheMyTestIndicator = tmp;
                return indicator;
            }
        }
    }
}

// This namespace holds all market analyzer column definitions and is required. Do not change it.
namespace NinjaTrader.MarketAnalyzer
{
    public partial class Column : ColumnBase
    {
        /// <summary>
        /// This is to test the huge memory consumed by LinRegSlope()
        /// </summary>
        /// <returns></returns>
        [Gui.Design.WizardCondition("Indicator")]
        public Indicator.MyTestIndicator MyTestIndicator()
        {
            return _indicator.MyTestIndicator(Input);
        }

        /// <summary>
        /// This is to test the huge memory consumed by LinRegSlope()
        /// </summary>
        /// <returns></returns>
        public Indicator.MyTestIndicator MyTestIndicator(Data.IDataSeries input)
        {
            return _indicator.MyTestIndicator(input);
        }
    }
}

// This namespace holds all strategies and is required. Do not change it.
namespace NinjaTrader.Strategy
{
    public partial class Strategy : StrategyBase
    {
        /// <summary>
        /// This is to test the huge memory consumed by LinRegSlope()
        /// </summary>
        /// <returns></returns>
        [Gui.Design.WizardCondition("Indicator")]
        public Indicator.MyTestIndicator MyTestIndicator()
        {
            return _indicator.MyTestIndicator(Input);
        }

        /// <summary>
        /// This is to test the huge memory consumed by LinRegSlope()
        /// </summary>
        /// <returns></returns>
        public Indicator.MyTestIndicator MyTestIndicator(Data.IDataSeries input)
        {
            if (InInitialize && input == null)
                throw new ArgumentException("You only can access an indicator with the default input/bar series from within the 'Initialize()' method");

            return _indicator.MyTestIndicator(input);
        }
    }
}
#endregion

[NinjaTrader_CalH]

EV,

Thank you for the report.

When add an instrument to the Market Analyzer, it will create a new instance of that indicator for that instrument. This effectively gets multiplied by 1600 instruments that you are trying to use. Additionally, since you are calling another indicator your script, there is another instance of that LinReg being created as well to run the calculations for your return. You could look at this as 3200 indicators running.

I ran this with the S&P 500 list and was over 3GB after 2/3 of the instruments loaded data.

Your script is fine but trying to run it on 1600 instruments is where the RAM usage is coming into play. Run this on the DOW 30 and you won't see such a huge drain.

[ETFVoyageur]

Cal,

The first thing is that reducing the number of instruments is not an option -- the point is to be able to rank all of them. What I need is to know how to control all of that huge memory use.

It is clear that the big memory use is not from my indicator itself -- switch which set line is used and you can see that. Setting to a constant value, instead of calling LinRegSlope() does not lead to big memory use. Hence the big memory use is coming from calling LinRegSlope().

That leads to the obvious question: why aren't the instances of LinRegSlope() getting garbage-collected once they are no longer in use? I am not retaining any reference to them, so they should be garbage-collected and memory should not grow so big.

• Is this a NinjaTrader bug (holding on to some reference to them)?
• Is there some way I can force them to get garbage-collected?

It seems to me that the bottom line is:

• The LinRegSlope objects will need to get created as needed. That will cost CPU cycles. I can accept that.
• Each of those objects will require some memory while it is running. I can also accept that.
• None of those objects requires any memory once it has returned its value to me, though. What I cannot accept is that such memory is never freed. I need to know what to do to free that memory.

--EV

[NinjaTrader_CalH]

EV,

This is not a bug but rather waiting for the Garbage Collection to come into play which is handled by the .NET framework.
You can force a Collection but it is not recommended
http://msdn.microsoft.com/en-us/libr...(v=vs.85).aspx

[ETFVoyageur]

EV,

This is not a bug but rather waiting for the Garbage Collection to come into play which is handled by the .NET framework.
You can force a Collection but it is not recommended
http://msdn.microsoft.com/en-us/libr...(v=vs.85).aspx

Thanks. I'll look into that. As is, the problem is that the garbage collection NEVER occurs. Not that it is delayed, but that in never happens.

--EV

[NinjaTrader_CalH]

Another consideration is the Operating System will run GC once it determines you actually need that memory.

[ETFVoyageur]

Another consideration is the Operating System will run GC once it determines you actually need that memory.

I know that is the theory, but it does not seem to happen. Standby memory went to near-zero and the size was still huge.

I tried calling GC.Collect() after every bar, and it kept the memory size down ... but (of course) was way too slow. Just done as a proof that there was memory to be collected. I am working on a better place to collect it now.

--EV

[ETFVoyageur]

I'm worried that there is no garbage to collect -- that something is hanging on to a reference to each of the LinRegSlope objects. I have modified OnBarUpdate() as indicated below. I added a couple of print statements, and one garbage collector call per instrument (after the last bar). The result took longer to run, but did not decrease the working set.

Result for the S&P500 with the GC call commented out (but the print statements retained) was 1:24 minutes, ending up with a working set of 3.4GB

Result with the GC call live was 5:15 minutes, ending up with a working set of 3.4GB

Somehow the storage used by the LinRegSlope calls is never going away, even if I call GC after the last bar of each instrument! Something is wrong with this picture.

--EV

        protected override void OnBarUpdate()
        {
            if (CurrentBar==0) {
                Print("");
                Print("DEBUG: Instrument=" + Instrument.FullName);
            }
            if (CurrentBar<2) return;
            IndicatorLine.Set(LinRegSlope(Math.Min(CurrentBar,period1Bars))[0]);
            if (CurrentBar == (Count-1)) { Print("\tOnBarUpdate, last bar, calling garbage collection"); GC.Collect(); }
            //IndicatorLine.Set(50);
        }

[koganam]

I'm worried that there is no garbage to collect -- that something is hanging on to a reference to each of the LinRegSlope objects. I have modified OnBarUpdate() as indicated below. I added a couple of print statements, and one garbage collector call per instrument (after the last bar). The result took longer to run, but did not decrease the working set.

Result for the S&P500 with the GC call commented out (but the print statements retained) was 1:24 minutes, ending up with a working set of 3.4GB

Result with the GC call live was 5:15 minutes, ending up with a working set of 3.4GB

Somehow the storage used by the LinRegSlope calls is never going away, even if I call GC after the last bar of each instrument! Something is wrong with this picture.

--EV

        protected override void OnBarUpdate()
        {
            if (CurrentBar==0) {
                Print("");
                Print("DEBUG: Instrument=" + Instrument.FullName);
            }
            if (CurrentBar<2) return;
            IndicatorLine.Set(LinRegSlope(Math.Min(CurrentBar,period1Bars))[0]);
            if (CurrentBar == (Count-1)) { Print("\tOnBarUpdate, last bar, calling garbage collection"); GC.Collect(); }
            //IndicatorLine.Set(50);
        }

You are probably going to have to rewrite the LinRegSlope indicator to use a more efficient algorithm.

The shipping version is using a for loop and also calling another indicator, SUM, (with its attendant initialization and other operational overhead).

[ETFVoyageur]

But why isn't LinRegSlope memory getting freed once the indicator has served its purpose? Who is permanently hanging onto a reference to it?

--EV

[koganam]

But why isn't LinRegSlope memory getting freed once the indicator has served its purpose? Who is permanently hanging onto a reference to it?

--EV

Have you tried using a named instance of the called indicator, so that NT does not have to go through its multiple looping code to find the indicator instances? (Take a gander at the magic wrapper code, and you will see what I mean).

[ETFVoyageur]

One thing I am thinking of is that an indicator can tell whether it is being used in a chart or otherwise (LastBarIndexPainted==0 for non-charting).

Some indicators need the full calculation in any case, such as ones that keep a running EMA of their result. Most, though, do not need to calculate until the last bar (or perhaps enough before that to apply a smoothing function). In most cases, that would save a lot of calculation.

--EV