What's the purpose of MaximumBarsLookBack?

[egan857]

Hi, I have a strategy that eats up a lot of memory, and memory use constantly grows all day long.

While searching through various threads on memory usage, I came across MaximumBarsLookBack and I don't understand how it's used.

When I create a new strategy, MaximumBarsLookBack is set to TwoHundredFiftySix and Days To Load is set to 20 in the Edit Strategy window. My code does not specify a value for MaximumBarsLookBack, so presumably it's using the value set in the UI, or 256.

I have a loop in the code that iterates through the most recent 3000 1-minute bars and does some calculations. These calculations are only done once, and then I set a flag to keep the loop from running again.

If MaximumBarsLookBack is set to 256, then how can it be looking at 3000 bars of historical data? I'm almost positive that it's really looking back 3000 bars.

After reading a little about MaximumBarsLookBack, it seems that I could set MaximumBarsLookBack to Infinite when the program is initiated, then after my one-time loop set it down to 256. In fact, I could set it as low as 61 if that would further improve performance.

So, my questions are:
1. If MaximumBarsLookBack is set to TwoHundredFiftySix on the Edit Strategy screen and not defined in the code, but the code loops back to view 3000 bars of data, then is MaximumBarsLookBack being overridden, or is it not used in this situation?

2. Would it help performance to set MaximumBarsLookBack to Infinite in the code and then set it to TwoHundredFiftySix after the one-time loop?

3. Can I set the MaximumBarsLookBack to another number less than 256?

[NinjaTrader_Brett]

Hello,

Thanks for the forum post.

Curious how many trades is this strategy placing or are you writing anything to the log or trace file? As the log must be loaded in the log tab of t he control center and if you are making a lot of log entries this is the number one suspicion for me for a strategy that eats up more memory through the day.

As far as the loop this is fine although any looping is generally not a good thing as its not very effecient. Its normally better to run the code on the bars from CurrentBar = 0 to the Current Bar when the strategy/indicator first loads it calculated on all these bars from bar 0 to start. Then you keep some variable moving forward and only adjust values moving forward as you go our SMA indicator is an example of this style setup. But as you say you only loop back once therefor I see no issue with this however you would not be able to use maximum bars loop back 256.

The maximum bars loop back is a ring loop. Which means that if you request data back behind 256 bars you will actually be access the first valuer of the current bar again and loop it goes. You would not be able to use maximum bars look back 256 and request bars beyond that and get accurate results.

Let me know if any further questions.

-Brett

[egan857]

Hi Brett,

This strategy trades anywhere between 100 and 1500 trades a day, plus it maintains an active exit order for the duration of the trade which updates every time the price moves more than a few cents. Yesterday was quiet and it only made 65 trades and was still eating up memory like crazy. I used to do a lot of Print statements, but now I send all that information to text files (using multex and StreamWriter). That slowed down the memory usage, but the real improvement came when I changed all the loops to run only when necessary (once a minute or every 3 minutes for some small loops, and once for the huge loops (1000 and 3000 iterations).

I run strategies for 100 symbols twice, once for the Sim101 account and once for my IB account, so I have 200 programs running. The log files typically run about 2 - 5 GB per day. I have one workspace and once chart that only displays a few indicators (ParabolicSAR, VolumeUpDown, ADX), plus ChartTrader. For a while I thought the charts were eating up memory, but the problem persists even after I delete the charts. I also just turned off Save Chart Data As Historical in the Control Center options, also with no improvement.

I'm not sure I completely understood your answer about MaximumBarsLookBack. I only loop more than 256 times twice, once to look back 3000 bars on a 1-minute chart and another time to look back 1000 bars on a 3-minute chart, and both of those loops are only performed once. I try whenever possible to save, for example, highs and lows for the last x bars and then when the bar shifts I adjust the numbers by dropping off bar x+1. It sounds like MaximumBarsLookBack is not going to affect my program one way or the other.

Here's a clue as to what's going on. When I start the programs early in the day, they take up less space than when I shut down and restart the programs later in the day. At 9:30, the programs take about 700MB; at 11:30, they start off around 1.4GB; at 13:30, they start off around 2GB, and then they grow from that starting point. For example, I just started the programs 30 minutes ago. They started at 1.49GB, then 1.76GB, 2.02GB, and 2.34GB at the next three 10-minute intervals. It's not going to take long before I run out of RAM and have to restart the program.

[NinjaTrader_Brett]

Egan,

Sounds like its the log file that is causing issues.

The log file has to be fully loaded into memory in the log tab and if the log file it 2 gigs then that's 2 gigs of memory in use by NT right there.

Unfortunately, each time you take an order action all order state will be written to the log and the log loaded on each restart of NT in the background.

Do you see anything in the log other then order actions and is it just order actions taking up the majority of log lines?

-Brett

[egan857]

Oh. Yeah, I looked at yesterday's log and even though I had less than 100 trades I had 11,000 of the 14,000 lines in the log file related to order entries, and over 10,000 of those were for order activity to exit the trades. Looks like I'm micromanaging the exits. I do want to continue to modify the exit orders as the price changes, but I just modified the code to drastically reduce the number of adjustments. It's late in the day, so we really won't know until late tomorrow whether this reduces the size of the log file and the memory usage. I'll reply on Thursday with the results. Wish me luck. Thanks for your help!

[egan857]

Hold the phone. I made a mistake. Yesterday's log file was 3.6MB, not 3.6GB. The trace file is over 14MB, and I'm looking over it now. I have less confidence that the recent change will drastically impact memory usage.

[NinjaTrader_Brett]

Alright, well lets see what the difference is and we will go from there. Specifically its not the trace file we load it is the log file though so please insure you are checking the correct log folder.

The last thing we want to check is when does the memory start to go up. Is it gradual over time or bursts of memory usage such as a loop running?

What are you at memory wise after the changes?

-Brett

[egan857]

Yes, I was checking the logs, and yesterday's log (log.20120813.txt) is the one discussed in earlier messages (3.6MB, vast majority of the lines relate to exit order modifications). BTW, I never made any sense out of the trace files.

Memory goes up consistently from the very beginning, and it seems to burst a little on 1-minute boundaries. My primary data series is 1-minute charts and I do have some calculations that only happen immediately after the first print of the minute, so this makes sense. The memory increase is at a slightly increasing arithmetic rate, so for example it'll start at 1GB and go to 2GB after 1 hour and 3.1GB after 2 hours, so the differential in the second hour is slightly more than in the first hour (those aren't actual numbers; typically I run out of memory about 3-4 hours into the day, and when I restart it starts off at a higher number and maxes out again before the end of the day). I have a 64-bit machine with 8GB RAM, but with all the system overhead it leaves me about 6.2GB for Ninja before memory is maxed out.

[koganam]

Yes, I was checking the logs, and yesterday's log (log.20120813.txt) is the one discussed in earlier messages (3.6MB, vast majority of the lines relate to exit order modifications). BTW, I never made any sense out of the trace files.

Memory goes up consistently from the very beginning, and it seems to burst a little on 1-minute boundaries. My primary data series is 1-minute charts and I do have some calculations that only happen immediately after the first print of the minute, so this makes sense. The memory increase is at a slightly increasing arithmetic rate, so for example it'll start at 1GB and go to 2GB after 1 hour and 3.1GB after 2 hours, so the differential in the second hour is slightly more than in the first hour (those aren't actual numbers; typically I run out of memory about 3-4 hours into the day, and when I restart it starts off at a higher number and maxes out again before the end of the day). I have a 64-bit machine with 8GB RAM, but with all the system overhead it leaves me about 6.2GB for Ninja before memory is maxed out.

It sounds more like a memory leak to me. You are probably creating a lot of objects that you are not Dispose()ing after use, and they are just sitting in memory chewing it up. Of course, we cannot really tell, at least not without seeing the code.

[sledge]

Yes, I was checking the logs, and yesterday's log (log.20120813.txt) is the one discussed in earlier messages (3.6MB, vast majority of the lines relate to exit order modifications). BTW, I never made any sense out of the trace files.

Memory goes up consistently from the very beginning, and it seems to burst a little on 1-minute boundaries. My primary data series is 1-minute charts and I do have some calculations that only happen immediately after the first print of the minute, so this makes sense. The memory increase is at a slightly increasing arithmetic rate, so for example it'll start at 1GB and go to 2GB after 1 hour and 3.1GB after 2 hours, so the differential in the second hour is slightly more than in the first hour (those aren't actual numbers; typically I run out of memory about 3-4 hours into the day, and when I restart it starts off at a higher number and maxes out again before the end of the day). I have a 64-bit machine with 8GB RAM, but with all the system overhead it leaves me about 6.2GB for Ninja before memory is maxed out.

Can you set your windows pagefile size higher? Or do you experiencing excessive thrashing? (or is that all automagically calculated in win7 and beyond, unlike XP days when you had an option).

[NinjaTrader_Brett]

If the log is not the culprit and everything else has been checked then most likely Koganams response is correct at this point.

Now we would need to analyze the code to see if we can dispose() or optimize the code so it does not create objects that then persist in memory.

-Brett

[egan857]

Thank you, Koganam, Sledge, and Brett for your suggestions.

Sledge, I don't believe Windows pagesize is configurable in Windows 7.

Koganam and Brett, I'll be happy to share my code, but I warn you that there's a lot to slog through. It is well documented, though.

Today's log file results aren't going to be definitive since I was doing so much testing.

What I don't understand is why Disposing is a good thing. When I run Dispose(), my program disables. There doesn't seem to be any way to dispose specific objects, and it seems like the only thing I want to dispose is the historical bar data that has been loaded into memory during my loops. I don't keep large arrays of intermediate data, just a couple of variables, almost all of which are used when deciding whether to buy or sell a stock.

Let's take a sample situation. I loop 1000 times through the 3-minute bars to count the number of candles with large wicks. I also loop 150 times through the 3-minute bars to find cusp points (highest high and lowest low plus/minus 4 bars). I store one variable for the wick count and 4 5-element arrays to capture the 5 most recent cusp points. After the initial setup, I never need to look at those old historical bars again. Then, every 3 minutes, I look at the highs and lows of the 5 most recent bars (not with a FOR loop, though) and possibly update my four 5-element arrays (also not with a FOR loop).

In this example, if we disposed of all the historical bars a few minutes after program initiation, we'd free up a lot of space. Every 3 minutes, I'd pull up information for the most recent 5 bars, and 4 of them were already in memory from 3 minutes ago (right?). I don't want to dispose of my wick count variable, and I definitely don't want to dispose my 4 arrays. So is there a way to selectively dispose just the old bar information?

BTW, every hour I manually run garbage collection (GC.Collect(); GC.WaitForPendingFinalizers(). In general, the garbage collection cleans up about 2MB per program, or 400MB for all 200 programs. It takes a couple of minutes and one of my 8 CPUs is totally maxed out, making trading difficult. But, after about 5-10 minutes, memory has increased fast enough that I'm back where I started.

[Radical]

Sledge, I don't believe Windows pagesize is configurable in Windows 7.

I'm not sure how this would affect your performance, but check out this video: http://www.youtube.com/watch?v=DWCVhk5gTps

[egan857]

Radical, you are absolutely right and I should have known better since I saw that video a couple of months ago and modified my settings.

Sledge, I apologize. Windows 7 does allow you to change the virtual memory size and mine is set to min 8GB and max 16GB.

[egan857]

OK, so I've had a couple of days to do some testing, and still not enough progress.

I was able to reduce the log file by one-third by changing how often I adjust my exit limit order prices, but that had absolutely no effect on performance. The log file is about 2-3MB per day (down from 3-5MB), and I really don't see how this can translate into 6GB of RAM in half a day.

I just started playing around with the OnTermination section. Until now, I did not have one because I thought OnTermination was called when the program terminated (silly me). It appears to execute on every tick, when it is done going through all of the code once. This is something that really should be addressed in the documentation. Anywho, I added an OnTermination section and stuck Dispose() in there. It had a very minor improvement on memory usage, but now the memory increases constantly, not in one-minute bursts. I also tested putting in garbage collection (GC.Collect(): GC.WaitForPendingFinalizers() ; ), but that ate up memory faster and increased the CPU usage. I also tried both in the OnTermination section, but that didn't help either.

So, if I have a memory leak, how do I get rid of it? Dispose() obviously isn't the answer; it shuts the program down if I run it in the OnMarketData method, and it doesn't help much in OnTermination. Can I get a better result by disposing of something explicitly (i.e., CurrentBars[0].Dispose() )? I'm getting desperate here - my program's finally starting to get good results, and this is the only thing keeping me from running it all day long or getting additional symbols in my Kinetick feed.