Possible memory problem

[ChuckAllen]

I am running an optimization that will take about 6-8 hours to complete. This is on a quad core Dell.
Please see NT-Bug32.Jpg.
Normally, the CPU usage will be 25-30% based on Norton...Look at the right side of the screen dump...Notice the CPU Usage is right on at 30%...this probably includes a little extra for the screen dump. This optimization (when I ran it last time), showed 2 Gig of memory available (Norton's PM free) the entire time. This time, the reading started around 2 Gig and has moved down to about 1 Gig. Also, looking at the screen dump, All NT Screens Seem Dead. During all previous optimizations, the charts and everything updated perfectly. There has been no update for about an hour now.

I am using the latest version. I updated a few days ago.
My other programs seem to work fine.

I will wait until morning to see if NT really did the optimization & will update again.

[NinjaTrader_Jason]

Hello ChuckAllen,

I will await the outcome of this morning to check if the optimization finished.

If not, can you please post your log and trace files for today and yesterday. Please include the time stamp when you started the optimization.

[ChuckAllen]

Att: Josh...High Priority!

These problems have to do with the Memory issue below.
I'm sorry, when the link to the forum is not included in the email, I try to
just reply to the email.


Can you please see if someone knows how to clear up the memory
usage for NT? It is using almost all PC resources. The optimization info
is missing. I don't know how to prevent loading whatever is taking up all
my PCs memory.

I have rebooted the PC & tried loading NT 2 more times. It still takes
a ton of memory & everything is drastically slower.


Thanks.

[NinjaTrader_Jason]

Hello ChuckAllen,

Can you please tell me if you have a 32 or 64 bit OS.

Am I correct it is using almost all PC resources if you only open the Control Center?

In addition, can you please tell me the size of the database. Please go to (My) Documents-->NinjaTrader 7-->db and tell me the size of the NinjaTrader(sdf) file.

[ChuckAllen]

Att Josh:

I have started the upload to SendSpace.Com as requested.
It still has about 20 minutes to go.
It contains the Log & Trace files for the memory issue.

I could not find your post/ticket number in the forum, so I used the [email protected] and my email address [email protected]. In the description along with something like Log and Trace files for my post around Yesterday @ 10:20 PM. That was the only ID I could find.

[ChuckAllen]

Att Josh I put your comments here, I did not see your post anywhere.

Long tests can take memory. If you run out of memory during that test then unfortunately you will simply need to decrease your optimization length. Looks to me that you are hitting the computer architectural limitations of memory for a 32-bit application. You will need to consider a 64-bit computer with significantly more RAM should you wish to do such extensive testing. On a 32-bit machine I suggest you decrease your testing dates and decrease your testing permutations to allow your results to be within the ~1.5GB limitation of applications on 32-bit computers.
Saving test results will not save detailed reports of the results. It saves the summary results, but not the individual information. This is expected behavior to save memory.
If you are saving a workspace with such an extensive Strategy Analyzer chart it will recover with heavy memory usage. Long tests yields extremely long charts which can take a lot of memory to pull up every single bar.
For your question about parameter settings, please go to the Settings tab if you want to see the individual settings in individual rows.
In general, memory management is handled by .NET's garbage collector. This is not controlled by NT, but by the Windows .NET framework. From the looks of your screenshot though, the memory would definitely not be released with the Strategy Analyzer still open as it is still using the information contained in the memory. After you run a backtest/optimization, all of that is stored in memory and will maintain there while you still have the Strategy Analyzer opened and reviewing those results.
Sincerely,
Josh Peng
NinjaTrader Customer Service

[ChuckAllen]

Reply to Jason

32 bit, Dell Quad Core, 4 Gig memory, 4 Gig page/virtual memory, XP Pro.

Yes, just openning the Control Center. I ONLY have 1 Chart open...nothing else.
The NT load takes about 1 minute and I can see the PM Free from Norton go down dramatically when NT loads.

Database Sizes:

NinjaTrader.Mdb 2,672 KB
NinjaTrader.Old.Mdb 121,192 KB
NinjaTrader.Old.sdf 4,924 KB
NinjaTrader.sdf 5,408 KB


Thanks.

[NinjaTrader_JoshP]

ChuckAllen,

From your logs I can see they are heavily flooded with error messages that likely come from an indicator processing on the chart or was processed on the chart beforehand. This will indeed cause NT to load slowly as it needs to recover nearly 600MB of error messages.

7/7/2010 2:22:17 PM|3|4|03/04 03:50PM:NBarsUp3, TTT
7/7/2010 2:22:17 PM|3|4|03/04 03:50PM:NBarsUp3, TTT
7/7/2010 2:22:17 PM|3|4|03/04 03:50PM:NBarsUp3, TTT
7/7/2010 2:22:17 PM|3|4|03/04 03:50PM:NBarsUp3, TTT
7/7/2010 2:22:17 PM|3|4|03/09 08:00AM:NBarsUp3, TTT
7/7/2010 2:22:17 PM|3|4|03/09 08:00AM:NBarsUp3, TTT

I am seeing about 2.5 million of these log entries. Each of these entries would be trying to populate into the Control Center's log tab. What I suggest is you remove whatever NinjaScript file is logging these messages from running, close NT while saving this new workspace, then going into Documents\NinjaTrader 7\log and delete the huge log files so NT will not try to load them into the Log tab.

[ChuckAllen]

Reply to Josh

I'm sorry about the messages. I did not realize that I can delete the log files.

One of the other suggestions I have not submitted yet has to do with these messages. There is no easy way to keep my messages separate and then delete just them. All of the log levels seem to be related to NTs debugging.

*** It would be nice to be able to have an Info or Customer log level so we can keep our logs separate. Then I could sort & hopefully delete them...

*** It would be nice to be able to mark & delete messages from the log files.

*** I could not have imagined in my wildest dreams that the logs would have been loaded. Why not just load them from the file when requested. This is a tremendous waste of memory when we don't realize what is happening behind the scenes.

Is there another way/mechanism to write information that you Don't load? For example: in a couple scripts, I write the Bar, Indicator, and trade info.
Do I have to write my own logger?

If so, is there a way to get the optimization info during a run & write it to my own log file for analysis later?

[ChuckAllen]

Deleting the Log File Messages worked

Josh,

When I deleted the Log files & started NT again, it works much better.

I will submit the Log File ideas in a separate post.

[NinjaTrader_JoshP]

I suggest you use Print(). Log() should only be used for critical information you wish to convey and not diagnostics information. Please see my response in your other thread.