Announcement

Collapse
No announcement yet.

Partner 728x90

Collapse

Log/Trace file memory hog bug

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    Log/Trace file memory hog bug

    B18, TWS 900.6

    Sometimes NT7 uses less than 100mb memory, and suddenly it uses 1.5gb. It looks like the large amount of memory used is simply due to repetitive logging in these two files:

    /trace/trace.yyyymmdd
    /log/log.yyyymmdd


    On "good days" the log file may be 2-7kb. And on bad days about 700mb. Same with the trace file. To me it looks like the extreme memory use is simply the fact that NT7 keeps these big files open for logging. Hence, it seems like a completely unnecessary problem, and simple thing to fix. I assume one either needs to:

    1. Put some logic to avoid the repetitive logging, or
    2. Put a file size limit to start on a "new file", when the size reaches some limit.

    I'm not sure which logging parameter causes this, as it may be different ones. But by starting a new log file at a specified file size of say 20mb would avoid any potential error that causes this behavior to eat up all system resources and hence seems like the best solution. There is no need for an log file of 700mb, because I as a user will never be able to or want to send that to you for error-tracing anyway.

    It's completely useless and unacceptable that the program priorities log files over system resources. Specially a trading program. Also the program itself has no use for these log files, at least I assume it's just pure text-dump for debugging purpose. (?)

    The files are now filled with this type of message, but I believe I've seen other repetitive log messages also before:
    Code:
    2010-07-09 00:00:12:087 ERROR: IB.Globals.MessageLoop msg='12': Index was out of range. Must be non-negative and less than the size of the collection.
    Parameter name: index
    2010-07-09 00:00:12:087 ERROR: IB.Globals.MessageLoop msg='12': Index was out of range. Must be non-negative and less than the size of the collection.
    Parameter name: index
    In any case, my suggestion should be easy to implement and effective. At least it seems so to me.. (?)
    Attached Files
    Last edited by bretddog; 07-14-2010, 12:12 AM.

    #2
    Your understanding is incorrect:
    - trace info is written to files and does not have noticeable RAM footprint
    - log info is written to files (which does not have any noticeable RAM footprint) and to the log tab. However, only log 1000 (default) entries would be loaded on NT startup

    also: not sure why you feel a log file could have a size of 20MB or larger (which would be very untypical). I could see this being the case as some NinjaScript indicators/strategies would write to the logs excessively, which then should be redesigned anyway.

    Comment


      #3
      So do you mean that even though both my log file and trace file is approx 700mb, this has nothing to do with the fact that it simultaniously has a memory footprint of about 1.5gb, exactly the size of these files?

      I don't run any Ninjascript strategies or custom indicators. I only have a 5 charts with cci and an ema.

      I don't mean that a log file "should" have a size of 20mb. Obviously it should be much smaller. But 20mb would be a sensible "cut-off" size, in case the log file size indeed was causing exessive memory footprint.. as it seems to do.

      Of course what is causing the large trace- and log- files I don't know. But my point is that it happens, and the code should prevent this in some way.

      But if you mean that having these files in the size of several hundred mb should not cause any memory hog.. then ok, I can understand that.. Then there must be instead a more serious underlying problem (which also is the cause of the large files) which we should try to trace.
      Last edited by bretddog; 07-14-2010, 06:11 AM.

      Comment


        #4
        Logs and trace file size in itself is irrelevant regarding RAM footprint. However, log file size of 700MB indicates that something else might be going wrong. Are your log files that big?

        Comment


          #5
          Ok.
          Yes indeed they get this big quite often, but not every day. You can see from the image on my first post. And the memory footprint is just exactly the size of of the log plus trace file, plus 70mb. Like just now as I type, it's 800mb RAM footprint, with 360mb log file and 370mb trace file.

          I have one trace file of about 3mb that would be good size to send you if you want.

          Comment


            #6
            Let's no confuse file size and RAM footprint of the NinjaTrader.exe process. A member of our support team will follow up shortly.

            Comment


              #7
              Hello bretddog,

              Could you please send me an email at support[at]ninjatrader[dot]com including your phone number. I would like to give you a call to setup remote assistance, so I can check your log files.

              Please put 'Attn Jason' in the subject header and reference this forum thread.

              Comment


                #8
                I'm not confusing it. I'm just pointing out that the RAM hog and exessive file sizes co-exist. They are both present, and match both in time and size. Just pointing out this observation.

                Jason,
                Could you possibly do this in other way? I don't want to disturb my trading hours. And I can also not open the files of this size. its too big to even open in notepad. But I do have smaller files too (which are still bigger than expected).

                Or else i can send you my contact details when I'm done for the day.

                Comment


                  #9
                  Ok, can you please send me files via www.sendspace.com. Please zip a 700 MB log file and check if it is less than 300 MB and send it over. If the .zip file is larger than 300 MB, please use another log file that is less big.

                  Send it to support[at]ninjatrader[dot]com and put 'Attn Jason' in the description field.

                  If this does not work for you, please let me know what time you expect to be done for today. Please note that I am located in the Netherlands and will not be online the entire 'US afternoon'. I can then ask a colleague to give you a call.

                  Comment


                    #10
                    Ok,
                    Sent you the files now.

                    Comment


                      #11
                      Hello bretddog,

                      The log file contains the following information many many times.

                      MessageLoop msg='12': Specified argument was out of the range of valid values.
                      Parameter name: Index was out of range. Must be non-negative and less than the size of the collection.

                      We recently support TWS version 906 - am I correct the issue occurs using this TWS version as well? You can download and install it from the following link.


                      Can you please tell me if you have any NinjaScript indicators or strategies running that would use L2 data?

                      Am I correct this only occurs when you connect to an IB paper account? Does it occur when you connect to an IB live account?

                      Comment


                        #12
                        I run NT7 B18 on two computers:

                        1. TWS 904.9 on Live Account
                        2. TWS 900.6 (Downloaded from your website) on Demo Account

                        The log files I sent you are from (1). But I have exactly same issue with both setups. The other computer (2) has Norwegian XP installed, and the error message repeating is hence:

                        29.06.2010 06:50:21|3|4|IB.Globals.MessageLoop msg='12': Indeksen var utenfor området. Den må være ikke-negativ og mindre enn størrelsen på samlingen.

                        I don't know if I will have time to test 900.6 live for this issue, as it doesn't happen all the time, and it disrupts my trading. But I can't see why it should be different.

                        Comment


                          #13
                          Helo bretddog,

                          Please note that I refer to TWS version 906 and not 900.6. Can you please download and install TWS version 906 from the IB connection guide below.


                          This is currently the supported TWS version with NinjaTrader.

                          In addition, can you please let me know if you have any indicators or strategies running that use level II data.

                          Comment


                            #14
                            Ok, I was not aware you have updated to 906. I will do that for tomorrow. Will take some days until I know if it makes any difference, as it doesn't happen every day.

                            I'm not running any stratgies. And indicators only the standard/included ema and cci.

                            Comment


                              #15
                              Ok sounds good. Please let us know if the issue recurs and inform us what steps you performed in the software just before it recurred.

                              Comment

                              Latest Posts

                              Collapse

                              Topics Statistics Last Post
                              Started by Geovanny Suaza, 02-11-2026, 06:32 PM
                              0 responses
                              595 views
                              0 likes
                              Last Post Geovanny Suaza  
                              Started by Geovanny Suaza, 02-11-2026, 05:51 PM
                              0 responses
                              343 views
                              1 like
                              Last Post Geovanny Suaza  
                              Started by Mindset, 02-09-2026, 11:44 AM
                              0 responses
                              103 views
                              0 likes
                              Last Post Mindset
                              by Mindset
                               
                              Started by Geovanny Suaza, 02-02-2026, 12:30 PM
                              0 responses
                              556 views
                              1 like
                              Last Post Geovanny Suaza  
                              Started by RFrosty, 01-28-2026, 06:49 PM
                              0 responses
                              554 views
                              1 like
                              Last Post RFrosty
                              by RFrosty
                               
                              Working...
                              X