Bug - Indicator Kills Ninja

[reach4thelasers]

The attached indicator( FatalIndicator.cs) affectionately nicknamed "Kill the Ninja" deliberately triggers a StackOverflowException.

This is Fatal to Ninjatrader 8 and the whole application terminates.

[NinjaTrader_ZacharyG]

Hello reach4thelasers,

Thank you for your inquiry.

I am unable to reproduce the crash on my end. However, it is expected for a core of your CPU to max out upon running this indicator as the KillTheNinja() method continuously calls itself and is stuck in an infinite loop.

What version of NT8 Beta are you running? You can check for this under Help -> About in the Control Center.

What steps have you done to cause the StackOverflowException and crash of the platform?

I look forward to assisting you further.

[reach4thelasers]

I accidentally had a method call itself and debugging took me to a SO Exception.

Version 8.0.0.5 (64 bit standard) - beta

[koganam]

The attached indicator( FatalIndicator.cs) affectionately nicknamed "Kill the Ninja" deliberately triggers a StackOverflowException.

This is Fatal to Ninjatrader 8 and the whole application terminates.

An infinite loop, deliberately coded or otherwise, should cause most programs to either hang or crash, depending on whether they are placing any parameters on the stack. So what exactly is this demonstrating, other than the normal?

[reach4thelasers]

Zach I am using 8.0.0.5 64-bit (Standard) - BETA
Windows 10 Professional 64 Bit

The two attached screenshots show what happens when I add that indicator to a chart. Its a fatal error which crashes the entire Ninjatrader Application.

Koganam, we may have to disagree on our approach to writing robust software. I don't believe its in any way acceptable Ninjatrader to be susceptible to dodgy code written by third party developers which is able to crash the entire appliaction - this is a financial trading platform, not Mine Sweeper!

If a third party indicator throws an exception, catch it, report it and kill the thread don't let it kill the whole platform. If a third party indicator goes into a while(true) {} infinite loop it should be detected and the thread killed - these things should have a heart-beat which tells Ninja they are still alive and the loss of a heart beat should result in termination.

Windows works exactly the same way "appliation xxx has stopped responding" - it lost its heartbeat. How bad would it be if a non-responding application killed the whole operating system? The same applies to ninja and its third party apps.

[koganam]

Zach I am using 8.0.0.5 64-bit (Standard) - BETA
Windows 10 Professional 64 Bit

The two attached screenshots show what happens when I add that indicator to a chart. Its a fatal error which crashes the entire Ninjatrader Application.

Koganam, we may have to disagree on our approach to writing robust software. I don't believe its in any way acceptable Ninjatrader to be susceptible to dodgy code written by third party developers which is able to crash the entire appliaction - this is a financial trading platform, not Mine Sweeper!

If a third party indicator throws an exception, catch it, report it and kill the thread don't let it kill the whole platform. If a third party indicator goes into a while(true) {} infinite loop it should be detected and the thread killed - these things should have a heart-beat which tells Ninja they are still alive and the loss of a heart beat should result in termination.

Windows works exactly the same way "appliation xxx has stopped responding" - it lost its heartbeat. How bad would it be if a non-responding application killed the whole operating system? The same applies to ninja and its third party apps.

I agree with you if something throws an exception in its operation, sure. On the other hand, an infinite loop is either deliberate, as you have done here, or it is the result of a mistake or just sloppy programming. An infinite loop is a different kettle of fish: it is locking up the entire resources of either a thread or the application. An infinite loop should be resolved by the programmer, not a dependence on some outside process to kill it.

Is it possible for an application to create and use its own messaging loop, so that it can preemptively kill its threads/processes? Certainly, The bigger question is: "Should an application really protect programmers from their mistakes?" (Not saying you made a mistake; I know you deliberately coded this to demonstrate your point). The point is taken.

My real question is whether it is a reasonable expectation that NT must go to extraordinary measures to protect programmers from themselves. If a programmer's application crashes the trading platform, it is up to the programmer to fix his code, or lose his customers; not for NT to devote resources to programming measures to kill the sloppy code. NT set out to create a platform, not an operating system.

Sure NT is more elaborate than a game like Minesweeper. Still that does not mean that is needs to become a sub/quasi-operating system.

So evidently, yes, we do differ. I do not think that an application needs to protect errant programmers and their code, and you do. I have to agree with you that our approaches are different. I try to write code that does not crash the platform. If my code crashes the platform, it is my responsibility to resolve the problem in my code. Actually, if I write bad code, I want it to crash the platform as feedback, so that I can correct it. I would rather not ship errant code, than ship the code, hoping that the platform will handle my mistakes. I can only speak for myself.

[reach4thelasers]

It wasn't an infinite loop it was a stack overflow caused by a recursive function. This does not loop forever locking up resources, it throws an exception. Languages such as .NET are compiled to Machine Code in real time during execution. Because of this the compiler lacks a form of optimisation called 'Tail Call Recursion'. Which would allow it to avoid allocating a new StackFrame when a method call comes as the final operation of a method. Since the result of the method call is not required by the calling method but will rather be returned immediately to the previous method, a new stack-frame is not required. Since the .NET languages are compiled "just in time" they lack this form of look-ahead optimisation and new stackframes are allocated for every recursive call and the stack which is limited to 1 Megabyte in .NET overflows and throws an exception.

Threads in a .NET execution environment whilst sharing the heap with the host process and all other threads each have their own stack. There is no reason for a StackOverflow in a strategy - or any other exception to result in a complete crash of the Ninjatrader application.

And should Ninjatrader protect programmers from themselves? Don't know... Should Ninjatrader protect its traders from possible unforeseen errors in a third-party strategy or indicator? Absolutely! Say I buy one of your indicators and leave running... it has an unforeseen bug which gets triggered by an unforeseeable set of circumstances. The bug crashes the strategy and it crashes the whole Ninjatrader app. None of my simulated stop orders get Fired and I lose a ton of money.

In a hosted execution environment such as Ninjatrader the host should be unaffected by third party issues.

[Tiixo]

It seems that Microsoft disagrees with you.
Directly from the MSDN Documentation of the StackOverflowException class.

In the .NET Framework 1.0 and 1.1, you could catch a StackOverflowException object (for example, to recover from unbounded recursion). Starting with the .NET Framework 2.0, you can’t catch a StackOverflowException object with a try/catch block, and the corresponding process is terminated by default. Consequently, you should write your code to detect and prevent a stack overflow.

So even if Ninjatrader tries to detect a StackOverflowException it would not be able to.

[reach4thelasers]

No you can actually enable StackOverflow handling for managed execution environments that run third party code

[koganam]

It wasn't an infinite loop it was a stack overflow caused by a recursive function. This does not loop forever locking up resources, it throws an exception. Languages such as .NET are compiled to Machine Code in real time during execution. Because of this the compiler lacks a form of optimisation called 'Tail Call Recursion'. Which would allow it to avoid allocating a new StackFrame when a method call comes as the final operation of a method. Since the result of the method call is not required by the calling method but will rather be returned immediately to the previous method, a new stack-frame is not required. Since the .NET languages are compiled "just in time" they lack this form of look-ahead optimisation and new stackframes are allocated for every recursive call and the stack which is limited to 1 Megabyte in .NET overflows and throws an exception.

Threads in a .NET execution environment whilst sharing the heap with the host process and all other threads each have their own stack. There is no reason for a StackOverflow in a strategy - or any other exception to result in a complete crash of the Ninjatrader application.

And should Ninjatrader protect programmers from themselves? Don't know... Should Ninjatrader protect its traders from possible unforeseen errors in a third-party strategy or indicator? Absolutely! Say I buy one of your indicators and leave running... it has an unforeseen bug which gets triggered by an unforeseeable set of circumstances. The bug crashes the strategy and it crashes the whole Ninjatrader app. None of my simulated stop orders get Fired and I lose a ton of money.

In a hosted execution environment such as Ninjatrader the host should be unaffected by third party issues.

A recursive loop without a terminating condition is by definition an infinite loop. Where does your recursion terminate in what you wrote?

private void KillTheNinja(){
			KillTheNinja();
		}

Putting aside that, given your evident skill set, I cannot see how you would ever need to buy any of our indicators , if my indicator/strategy did what you describe, I would expect to lose any customer who bought that indicator/strategy. It is up to customers to test the code that they buy, and make sure that it is suitable for their purposes and works correctly. It is up to programmers to, as MSDN says:

... Starting with the .NET Framework 2.0, you can’t catch a StackOverflowException object with a try/catch block, and the corresponding process is terminated by default. Consequently, you should write your code to detect and prevent a stack overflow.

(emphasis mine).

[reach4thelasers]

I'm really not here to get into a petty **** fight - just trying to get my work done and help Ninjatrader iron out their issues. And I certainly won't be buying any of your indicators if you are of the belief that Quality Assurance is down to the customer and not the developer. Thank you and goodbye.

[koganam]

I'm really not here to get into a petty **** fight - just trying to get my work done and help Ninjatrader iron out their issues. And I certainly won't be buying any of your indicators if you are of the belief that Quality Assurance is down to the customer and not the developer. Thank you and goodbye.

Hm. Getting testy if everybody does not agree with us are we?

Quality Assurance is up to the developer, not the customer, and certainly not the host! You have great skills. Use them, instead of expecting someone else to catch your mistakes for you. Where do you get such a sense of infinite entitlement? This is a discussion forum. Going off in a huff because everybody does not agree with you is childish.

You may choose to respond or not. I shall not see it. Fortunately, this forum allows me to block people like you. My company signature is deliberately turned off, because this is personal, and my opinion alone.

[NinjaTrader_Brett]

Guys,

Great flexibility comes great responsibility

Just to clear some confusion and get back on the right track. There is no 'hosted' container where third party code runs in NinjaTrader. This is just simple raw c# code that runs along side NinjaTrader's c# code.

Sure we use C# features as we can such a try/catch blocks to catch exceptions from third party code being run (Along with exceptions in our own code). Exception handling is a complex topic where some exceptions can be safely caught and others cannot.

In the end its c# code that's running which is both a blessing and a curse. It allows customer to complete freedom to build whatever they want but it also allows them to write code which will bring NinjaTrader to its knees.

[koganam]

Guys,

Great flexibility comes great responsibility

Just to clear some confusion and get back on the right track. There is no 'hosted' container where third party code runs in NinjaTrader. This is just simple raw c# code that runs along side NinjaTrader's c# code.

Brett, maybe it is just a semantic difference, but I kind of disagree with you here. Inasmuch as NT indicators, strategies, etc, cannot run unless called by NT in its own context, NT is acting as a host application to run what are essentially extensions of itself. That is why improperly written extensions can bring NT to its knees.

And regardless, I maintain that we who choose to program for NT should not expect NT to take care of our programming mistakes. Programmers should take responsibility for quality of their code, not depend on some other entity to do it for them.

It is an issue that I see continually with many of those that never had to program without the cushion of a managed environment. "Somebody else should take care of catching my mistakes, and if they do not, it is their fault, not mine" is a statement that is almost never heard from those who had learned to program in any language before, and up to C, or even C++. Ever heard that statement from a Linux programmer?

[NinjaTrader_Brett]

Its fine to disagree but just to clarify I'm not talking semantics or opinions here but technical facts. In which case getting on the same page with terms used is the first step.

Technically there just isn't a viable approach with the current state of C# and .Net framework provided from Microsoft to build an 'isolated host environment' for C# code written by third parties to execute in vs NinjaTrader code. This separation of execution context can be thought of similarly to 'processes' at the OS level.

We've looked into utilizing similar C# concepts such as App Domains and Processes but they simply have to high of cost in cross communication to be effective and as such are not an option for NinjaTrader.

In this technical definition NinjaTrader is not 'hosting' third party code in its own 'isolated execution environment' and the execution context is shared by both NinjaTrader and the Third Party Code. Having an 'isolated execution environment' is what would be required to achieve what 'reach4thelasers' is suggesting and for the reasons above is not technically possible at this time.