NT7b18 crashed without reason, complete freezing

[NinjaTrader_Ray]

Hi All,

It would seem that 4 of 5 of you were all running NinjaScript Strategies. Next steps are:

- Please make sure you upgrade to B19
- Next time you experience the the freeze please confirm that NinjaTrader is unresponsive (does not respond to any mouse action, can't move windows etc...)

a) Your CPU is pegged consuming high %
b) Your CPU usage is really low

Thanks in advance.

[azuul]

Strategies were running during the freeze
Real Time
Working orders were open - in Sim

@ Powermoney : do you mean that strategies were still being executed (=running?) by NT while the GUI was frozen? (that would explain my experienced discrepancy in positions)


The GUI was 100% frozen in place, unresponsive to any action like a windows background picture (Different from other freezes like during optimization with print statements in the code - then NT would freeze but become grey when I try to move an NT window and eventually windows asked if I want to close the application. This was not the case here.)
Windows 7 and TWS were running smoothly at any time. Not performance related in this case.

[powermoney]

No - To clarify, strategies were running until the point of the freeze. An order placed by the strategy moments before the freeze may not have been properly filled. After the freeze, the strategies were not executing and (NinjaTrader) had lost control of the current working orders. They remain in place as is - unmanaged. Positions have to be closed manually after restarting NinjaTrader.

[powermoney]

Hi All,

It would seem that 4 of 5 of you were all running NinjaScript Strategies. Next steps are:

- Please make sure you upgrade to B19
- Next time you experience the the freeze please confirm that NinjaTrader is unresponsive (does not respond to any mouse action, can't move windows etc...)

a) Your CPU is pegged consuming high %
b) Your CPU usage is really low

Thanks in advance.

I upgraded to B19.

The freeze occurred today at 12:01:44 PM.
This is a different computer than previous.
There were only 2 charts open and 1 strategy running at the time.
The CPU usage is normal - around 10%
The Ninjatrader.exe process CPU % was 2-3%
The Ninjatrader.exe process was using 215704k memory

Looking at the trace files, this and previous freezes occurred shortly after a trade - and the last entry in the trace files were:

Yesterday - Machine 1 w/ B18:
2010-07-22 08:30:07:619 (Esignal) Cbi.ExecutionEventArgs.Process: Execution='c0ce2e4205f741b08e528d7fc1527c1a' Instrument='CL ##-##' Account='Sim101' Exchange=Default Price=76.79 Quantity=1 Market position=Short Operation=Insert Order='68f25acd1bc541f9a8b6d398b0620033' Time='7/22/2010 8:30:07 AM'

Today = Machine 2 w/ B19:
2010-07-23 12:01:44:020 (esignal) Cbi.ExecutionEventArgs.Process: Execution='e382bd5db5b242e5bc6b9ae506252512' Instrument='YM ##-##' Account='Sim101' Exchange=Default Price=10307 Quantity=2 Market position=Long Operation=Insert Order='487e63e194364a2c9c8f2a5d6f70b6b5' Time='7/23/2010 12:01:43 PM'

Powermoney

[NinjaTrader_Ray]

powermoney,
Could you send us an email with contact information? We would like to follow up via telephone.

[dave1992]

Of all the freezes I've seen -

- all were while running multi time frame strategies in the strat tab, but I don't use charts particularly anyway.
- all happened during market replay, except for one which was live
- I'm not sure if all the replay ones had a position open when the freeze happened, but I think it's possible
- the live freeze did have two open positions - since my positions usually last 1-5 mins, and are 5-20 trades per day, it is possible that the freeze is more likely soon after a trade, or while a position is open.
- all had no clues in the logs/trace
- all had low cpu/mem
- they may be more likely at higher replay speeds
- I don't think I've seen one while running just one strat, but I don't do that often anyway.

I downloaded B19, and had one complete 20 day replay run through of 5 strats at 500x speed without a freeze. In my experience that would usually have frozen at some point. (By the way, do multiple running strats get to take advantage of multi core, or is it just backtesting?)

regards
Dave

[NinjaTrader_Ray]

Of all the freezes I've seen -

- all were while running multi time frame strategies in the strat tab, but I don't use charts particularly anyway.
- all happened during market replay, except for one which was live
- I'm not sure if all the replay ones had a position open when the freeze happened, but I think it's possible
- the live freeze did have two open positions - since my positions usually last 1-5 mins, and are 5-20 trades per day, it is possible that the freeze is more likely soon after a trade, or while a position is open.
- all had no clues in the logs/trace
- all had low cpu/mem
- they may be more likely at higher replay speeds
- I don't think I've seen one while running just one strat, but I don't do that often anyway.

I downloaded B19, and had one complete 20 day replay run through of 5 strats at 500x speed without a freeze. In my experience that would usually have frozen at some point. (By the way, do multiple running strats get to take advantage of multi core, or is it just backtesting?)

regards
Dave

Thanks. Multiple running strategies will not take advantage of multiple cores. Please stay tuned, we will likely provide you a custom installer since we have identified something that might cause a dead lock in Market Replay.

[azuul]

Version 7.0.0.19

Complete Freeze

I was simulating live 4 strategies (from chart) - single timeframe each - IB is Dataprovider
CPU idle 1-3%
Memory consumption of NT about 250MB that time, not changing during freeze.
Nothing special in the logs and trace, last entry in log prior to freeze was an execution.

Very likely same issue as during first freeze with 7.0.0.18.

Could the freeze be related to latency issues causing a deadlock ?
Had these issues only with IB Data so far.

Sent trace and log (freeze was around 1:39 PM)

[powermoney]

azuul, what instruments were you trading/simulating when NT froze?

[azuul]

only Forex: EUR/USD

Freeze occured rather shortly after starting the strategies, since then I am running the same setup/strategies without problems.
I was only connected to IB

[NinjaTrader_Ray]

We believe we have figured out what the issue is. Its a deadlock situation. We are currently internally testing this and if it looks good, we will provide a custom build for those of you have experienced this issue. Thanks for your patience and we will update this thread once we are ready.

[dave1992]

We believe we have figured out what the issue is. Its a deadlock situation. We are currently internally testing this and if it looks good, we will provide a custom build for those of you have experienced this issue. Thanks for your patience and we will update this thread once we are ready.

This is good news. Thanks.

Could we learn some lessons from this whole issue? This was written on this forum over two weeks ago:

"If I had to have a guess, I'd say it's a database or resource deadlock. That certainly fits in with the symptoms"

It was completely unacknowledged, and since then we've had to argue against numerous posts saying it's your cpu/mem etc. It was very frustrating.

thanks

[NinjaTrader_Ray]

This is good news. Thanks.

Could we learn some lessons from this whole issue? This was written on this forum over two weeks ago:

"If I had to have a guess, I'd say it's a database or resource deadlock. That certainly fits in with the symptoms"

It was completely unacknowledged, and since then we've had to argue against numerous posts saying it's your cpu/mem etc. It was very frustrating.

thanks

Sorry for the frustration, for sure it was not intended. We will try to do a better job in the future.

[NinjaTrader_Ray]

We have a custom build that we would like to provide to those of you in this thread that have experienced the freezing.

Please send an email to our support team, attention Josh and reference this post and provide your forum ID. We will provide further instructions.

[paulg]

Ray, do you still provide a custom version --or the latest beta 22 fixed those issues?

And you guys, dave1982, azuul and powermoney are you experiencing no more freezes?