Market Replay Debugging

[NinjaTrader_JoshP]

symphys,

When you bring up task manager is NT still using CPU and memory is still changing? When running on 500x it can be possible that your strategy's calculations are intensive enough and that there is simply too many ticks queued to be process that it slows down while everything is queued until it can fully process the whole queue before being able to proceed. Basically what I mean in these instances is that your computer is maxing out the calculations as fast as it can, but 500x is simply too fast for your computer to process all of the ticks. You can either try decreasing the speed, switching CalculateOnBarClose = false to true, or perhaps look into less CPU intensive strategy calculations.

[NinjaTrader_Austin]

I have some setup that needs to know the day of the data against which the script has started, so I use Time[0].

Dave, for this purpose, you could always just run your start-up procedure during the very first bar:

OnBarUpdate()
{
  if (CurrentBar == 0)
  {
    // do one time calculations, information gathering, check time of bar, etc
  }
}

[symphys]

NT.exe is using 20-40% of processing time, 120Mb RAM so plenty of resources are left.
When crashed NT processing time is 0 and memory doesn't change.
I guess the crash problem is somehow .net related because if I don't open the chart window it won't crash that often...

[dave1992]

symphys,

When you bring up task manager is NT still using CPU and memory is still changing? When running on 500x it can be possible that your strategy's calculations are intensive enough and that there is simply too many ticks queued to be process that it slows down while everything is queued until it can fully process the whole queue before being able to proceed. Basically what I mean in these instances is that your computer is maxing out the calculations as fast as it can, but 500x is simply too fast for your computer to process all of the ticks. You can either try decreasing the speed, switching CalculateOnBarClose = false to true, or perhaps look into less CPU intensive strategy calculations.

He has the same symptoms as me - it's likely he has the same cause as me.

[symphys]

That's why I posted in this thread.
First question would be if that happens only when a strategy is enabled? I didn't try that...

[NinjaTrader_JoshP]

symphys,

Please provide the strategy, replay file in use, and exact steps to reproduce. Thank you.

As far as your question goes, I do not know as none of our test machines have been able to replicate.

[dave1992]

Just so you know, I got a hang today that looks the same, while trading live - not market replay. That lends even more weight to the suggestion it might be some sort of deadlock, since it appears to happen much less frequently while trading live. I do predict that one day you're going to have to fix this. It's a shame you've completely ignored the evidence I've provided in this thread, and appear to be insisting that one of your users writes a reproducable scenario for you. (Incidentally, if it wasn't reliably reproducable, would you then just ignore it for ever? This hang appears as if it may have been going on for years now)

thanks
Dave

[nailz420]

I also get a random hang in market replay while running it at high speed(100x - 500x) running on multiple strategies with secondary periods(Range 1 or Tick 1). If you cannot pinpoint the exact source of this hang, at least give us tools to be able to help you identify it. Perhaps an ability to activate some kind of extra granular debugging output or add some code to check if the market replay has become unresponsive or a way for the user to signal NT that it became unresponsive to generate a stack dump or some other type of report.

Thanks,
Tim.

[NinjaTrader_JoshP]

Dave,

No evidence has ever been ignored. We have been trying to get a reproducible scenario from you guys so we can analyze the exact situation you guys are facing. Even without a reproducible scenario we would still need the requested files. Without them we have absolutely zero chance in analyzing let alone addressing the issue.

Since no one has sent in any of the requested files, this means we are only able to test with what we have on our end. We have already done so to no avail. The whole point of users sending in requested files is so we can find commonalities which can help isolate the issue. It gives us a point of reference. With zero information you are asking us to do a task equivalent to finding a needle in a haystack without hands. With the requested files you will be equipping us with hands to at least have a chance to feel around the haystack and find the needle.

I will again request the files needed. Strategy, replay files for instrument in question, and steps used to run the strategy on the replay (e.g. strategy parameters, 1min? tick chart?, etc). Your cooperation is appreciated.

Tim,

We need the user scenario to be able to analyze anything. This means sending in the strategy in question and the replay files in use. Without the user scenario we can only guess on our end.

For your particular case though, running on a 1 tick series for sure can be slow, even appear to hang, depending on your computer specs and strategy complexity. A 1 tick/range series is extremely intensive on CPU and memory. Running on 500x can quickly stack up the work queue for your CPU and as data keeps flowing at 500x, the queue is not being cleared out fast enough, so you can end up with a lot of backlogged work needing to be done which can give the impression of a hang while your CPU churns away to try and clear the queue. Obviously this will take some time especially if you keep adding more onto the queue which takes more memory, more CPU thus depriving the resources for the computer to process whats already in there.

For you, you should be look into putting less load on your computer. This could mean running less strategies, running at a slower speed, or even tweaking your strategy to not use a 1 tick series which would be the most significant performance boost.

[dave1992]

No evidence has ever been ignored

OK, good, so what were the conclusions from the evidence I presented in post #14

I already told you why my strategy won't work for you, and I'm unprepared to spend a lot of time rewriting it for the small chance that I can make it 100% reproducible for you. That's just a poor use of time. A much better use of time would be for one of your developers to spend a few mins looking at my stack trace to see if there are clues there. I'm not saying I won't try and write you a strategy that does this, just lets do the easy bits first eh? Incidentally, how complex is your main testing strategy? Limits, stops, cancels, 1 tick series etc? Or just MA crossovers...

With zero information you are asking us to do a task equivalent to finding a needle in a haystack without hands

It's not zero information though is it? I put time and effort into compiling the info I gave. At least just respond and say it was useless.

For your particular case though, running on a 1 tick series for sure can be slow, even appear to hang, depending on your computer specs and strategy complexity. A 1 tick/range series is extremely intensive on CPU and memory.

I have to say I find this performance argument quite discredited (and massively overused) In a good 90% of the hangs/freezes I've seen on this forum there is zero evidence that it's performance related.

[symphys]

Sadly I can't send in my strategy code. It's profitable :-)))

[NinjaTrader_Ray]

We understand that sending us proprietary scripts is not always an option. We also agree that spending time writing a script with the small chance of providing a 100% reproducible scenario is a poor use of your time. We have reviewed the stack trace and unfortunately it does not make this witch hunt any easier for us. You are right that eventually we will have to fix this and we are hoping that eventually is sooner than later. Our testing scripts vary and we custom code new scripts based on information that is provided from our users.

If possible, please provide any information that you can as per Josh’s prior post. We run the application through a battery of tests to see if we can reproduce this. This likely will happen on Monday.

Thanks for your patience and understanding.

[nailz420]

Attached please find a type of strategy that hangs in replay mode. I reproduced the hang while running it for 7/26/2010, ES 09-10, 3 minute period, 12 tick stoploss, 3 tick profit target, 500x speed.

Regards,
Tim

[NinjaTrader_Bertrand]

Tim, thanks for reporting - I'll try here shortly on my end - can you please contact me directly at support at ninjatrader dot com in the meantime?

Thanks for your support.

[NinjaTrader_Bertrand]

Edit: Tim, I could not reproduce it here on our latest custom build designed to address the previous reported MR issues - please contact me at our support email so I can follow up directly - thanks.