Market Replay Debugging

[dave1992]

There are plenty of messages on here about Ninja hanging/freezing or closing with the messages - program must close, or encountered a user defined breakpoint (meaning what exactly?)

Some or possibly many of these are associated with Market Replay. Today, every time I use market replay it fails somehow. The first few times were the user breakpoint when I tried to enable all my strategies at once. The last few times it hangs after replaying a few hours worth of data.

We've been through all this - reinstall ninja, delete workspaces etc. It's not easily reproducable. Hangs rarely are. But that doesn't mean that there isn't a serious reliablity problem with market replay. It is NOT performance related, unless it's the poor performance of Ninja somehow. Please understand that. CPU is 1%, 12GB RAM is 40% used.

There's nothing in the logs or trace files. We've been there too. I've suggested using dump files, but simply been told no. I've suggested using proper debug data in beta software - no reply.

Market replay is extremely important to the serious trader - partly because you can't backtest with bid/ask and in many cases you'd be stupid to ignore the effects of bid/ask.

There were some problems fixed in this area recently, so I know it's been looked at and you know a lot more about what causes lock ups than me. I'm on Beta 18. These files are all downloaded from Ninja covering the last 12 days. 6E, 6B etc

How do you propose to fix stuff like this? It's not easily reproducable, so other methods such as proper debug information should surely be looked at.

In summary, Market replay is unreliable, has been since Beta 1, and I don't see how we're moving forward on making it robust. I've spent so many hours on it (typing each strategy in again EVERY time after it fails, etc) and it's really shaking my belief in Ninja as a viable testing platform.

thanks
Dave

[NinjaTrader_RyanM1]

Hello Dave1992,

Thank you for your observations and comments. I will have someone reply to you on Monday.

[NinjaTrader_Bertrand]

Hi Dave, thanks for the feedback and your testing - is there any common denominator that would be in play when your 'freezes' occur? Could it be perhaps custom script related? I spent a good amount of time replaying strategies on the currency futures and US index futures this weekend on a common, reasonably equipped Dell laptop and did not run into an issue or crash while doing so.

Can you please try our SampleMACrossOver on your desired charts, bar types and timeframes to see if this would freeze / hang as well?

Are you working in a 32 or 64 bit environment as this occurs?

Thanks for the support.

[dave1992]

Hi Bertrand,

Thanks for the reply. No, this probably wouldn't happen with simple ma crossover, yes, it probably is custom related. This thread is about how we find out what's causing it. I can only do so much with print statements, reinstalls and speculation. How do we debug this properly is my real point.

Dave

[dave1992]

Furthermore, I've regularly had today the User Defined Breakpoint error. Not just on market replay. It's a pop up box which says Ninja has encountered a user defined breakpoint and must close. It happens very often when startign or restarting a custom strategy.

Nothing in the trace/logs, no significant CPU usage. Nothing. How do we debug that too?

Happens on both 32 and 64 bit versions, with Zen Fire/Mirus connection. Oh, and it happened on a completely fresh install.

The only real thing in common is my custom strategy. I'm happy that that could be the cause, but how do I debug it. Prints aren't going to work, since the strategy usually isn't actually started when it happens.

[NinjaTrader_Bertrand]

Dave, for a start to investigate this please forward us your trace / logs on to support at ninjatrader dot com or using the Help > Mail to Support feature, please direct them to my Attn and mention our thread here. I just want to take look through to see if anything peaks out that could give us a clue. For the custom strategy, you could start by commenting out parts / methods of the code and then rechecking if that would step by step improve the stability, I'm happy to run it here on my end, too if that's an option for you.

[dave1992]

Thanks.

I've sent the trace files. Commenting parts of code out is not really viable since it's a large strategy, the error is fairly random and the whole process would take too long and not be specific enough. This is not a simple code error, which I could find in minutes. Even if it could be narrowed down it doesn't tell us what the problem is or how to fix it.

You really need to seriously address my questions about debugging.

[NinjaTrader_Bertrand]

Thanks Dave, just replied back - you would need to debug your custom code then by running a isoluation approach to find the part that raises this issue on your end, you start with the basic working set of code and expand complexity until you hit the point where it would break.

[dave1992]

Bertrand,

With respect, I do understand how software testing works, I've been doing it for over twenty years. I already explained why that was not viable approach.

Please please please take seriously my comments about debugging. Why can't I just provide a dump file? Wouldn't that then take your developers 5 mins? Or give me some means of debugging more sophisticated than print or trial and error and I'll do it.

thanks
Dave

[NinjaTrader_Bertrand]

Dave, for more advanced debugging you could turn to Visual Studio then to handle it more effiently -

http://www.ninjatrader-support.com/HelpGuideV6/VisualStudio.html

The dumb files created would not contain info related to your custom scripts so they are likely not useful.

If you see the 'program must close...' there should be an entry in the trace file then for this day and if you see this happen, please forward it to us for review.

Please consider how many people are using the Market Replay capabilties daily in beta testing, we've addressed each reproducible case moving through the Beta and we currently do not see an general issue here judging from feedback received as well as our testing with custom strategies such as our default SampleMACrossOver on a continuous basis - please debug your code with the tools outlined to be able to isolate out where the breaking points are so you could address them.

[dave1992]

I shouldn't have to buy Visual Studio to do this, and you don't support the express version.

Dump files usually tell you what a process was doing when it failed - at least which module was being executed, and a stack trace. Maybe they don't with Ninja and .net - can your developers confirm they're useless in this case? What if the error is not in custom scripts, then do they contain 'useful info'?

I told you what the error said - user defined breakpoint etc, there's nothing in the trace file - you've seen it. See before each session start? Nothing.

Can I suggest instead of testing continually with sample ma crossover, which is little like the real world, you write a proper startegy that does lots of things - gets some config from a file, has multiple time frames etc, and test that continually.

Or perhaps a more efficient way - maybe your developers could tell us some of the scenarios that don't work well in a strategy.

[NinjaTrader_Austin]

Dave, just because we don't support the express version of VS doesn't mean it won't work. As you've noticed you can only go so far with the Print() debugging and commenting out certain code sections.

[dave1992]

Right, using visual studio I've worked out one of these problems.

I had no idea that the first bar wasn't available in OnStartUp. 99% of the time if you try it, it is available. I have some setup that needs to know the day of the data against which the script has started, so I use Time[0]. As I said, 99% of the time Time[0] is available and correct. Sometimes it isn't. A search through this forum showed a post that says don't use bars in OnStartUp.

So, a couple of suggestions - perhaps that could be put in the documentation for OnStartUp. Secondly, I don't know what exceptions are checked in that area, but it would be nice if the strat failed with 'error: failed to call OnStartUp' ... etc.

I'll try and find the market replay hang now, although I suppose there's a chance that might not happen with all the debug code running.

Dave

[dave1992]

OK, I've got the stack trace for the market replay hang. I've attached a file containing the trace of each thread that was doing anything.

If I had to have a guess, I'd say it's a database or resource deadlock. That certainly fits in with the symptoms

Dave

[symphys]

I do not have problems with Market Replay with normal speed. But when running my strategy with 500x speed NT locks up frequently. There is no error or whatever, the windows just do not react/refresh anymore.
I need to use the Taskman to kill NT.exe in this case...