NtDirect.dll memory leak when calling OrderStatus method?

[JQuant]

I emailed this message to NinjaTrader support team but I want to share it here for the benefit of wider community. In particular, it's very relevant to a related thread:

http://www.ninjatrader.com/support/forum/showthread.php?t=49974&highlight=NtDirect

I have noticed and verified with an additional testing code that repeatedly calling NtDirect.dll method 'OrderStatus' will keep increasing memory usage of my application.

Once I identified this issue I ran the following snippet. I place an order, get it's orderId and then call 'OrderStatus' 5,000,000 times (five million times) in a for loop as here:

for(int i = 0; i < 5000000;i++)
{
OrderStatus(orderId);
}
While this loop is executing, I keep watching TaskManager's memory usage of my application and it keeps increasing fairly fast (from 20MB to 200MB until this loop is done, the loop is done in about a minute). Once done, the memory stays at 200MB.
Anyone using this interface and doing automated trading will run into this issue once they start tracking status of their order as during an open trade one has to keep calling this method frequently (I do it every 20 milliseconds) to be able to get 'real-time' updates on their order status.

[Atomic]

I thought I responded to this, but it must have been your message, so if you see this repeated in a message, that's why.

I solved the memory leak problem for the most part by reducing the string length in the NTOrderStatus function as follows:

{ ** Copyright (c) 2005, NinjaTrader LLC [email protected] }
DefineDLLFunc:  "NtDirect.dll", int, "Connected", int;
DefineDLLFunc:  "NtDirect.dll", lpstr, "OrderStatus", lpstr;
DefineDLLFunc:  "NtDirect.dll", int, "SetAllocReturnString", int;
DefineDLLFunc:  "NtDirect.dll", int, "SetMaxReturnStringLength", int;

inputs: OrderId(string);

SetAllocReturnString(1);				{ alloc memory for returned string, TS will free that memory 	}
SetMaxReturnStringLength(13);			{ limit the max length of return string to 500 					}

if Connected(0) = 0 then 
NTOrderStatus = OrderStatus(OrderId);

I created another version which shortens the string length to 2 and it ran ok for me, someone else had a problem so I've quit using it for the time being. Here it is in case you're interested:

{ ** Copyright (c) 2005, NinjaTrader LLC [email protected] }
DefineDLLFunc:  "NtDirect.dll", int, "Connected", int;
DefineDLLFunc:  "NtDirect.dll", lpstr, "OrderStatus", lpstr;
DefineDLLFunc:  "NtDirect.dll", int, "SetAllocReturnString", int;
DefineDLLFunc:  "NtDirect.dll", int, "SetMaxReturnStringLength", int;

inputs: 
	OrderId(string);
	//NTOS(stringref);

SetAllocReturnString(1);				{ alloc memory for returned string, TS will free that memory 	}
SetMaxReturnStringLength(2);			{ limit the max length of return string to 500 					}

if Connected(0) = 0 then begin
	//NTOrderStatusA = "OK";
	
NTOrderStatusA = "None";
if OrderStatus(OrderID) = "Wo" then NTOrderStatusA = "Working" else
	if OrderStatus(OrderID) = "Fi" then NTOrderStatusA = "Filled" else
		if OrderStatus(OrderID) = "Pa" then NTOrderStatusA = "PartFilled" else
			if OrderStatus(OrderID) = "Ca" then NTOrderStatusA = "Cancelled" else
				if OrderStatus(OrderID) = "Pe" then NTOrderStatusA = "Pending";


end;

[JQuant]

I don't use it in TS, I use it as a classic dll import in my custom .net application. Why should a method that returns a finite number (10 exactly) of fixed strings introduce any memory issues? The dll could have preallocated those 10 strings and just return one of them. Obviously I don't see inside it so I don't know what is going on but I do know that this needs to be fixed somehow. Any programming experts suspecting what could be going on under the hood? None of this is happening on calls to other methods of NtDirect.dll that DON'T return strings. Those methods I can call millions of times and it's all fine. The two methods 'OrderStatus' and 'Orders' that return strings make memory usage go up on every call to them.

[JQuant]

So I finally had it confirmed by a NinjaTrader developer that there indeed is a memory issue with repeatedly calling NtDirect.dll's 'OrderStatus' method. They suggested to use the not officially supported .net 'NinjaTrader.Client.dll' that exposes the same methods as the unmanaged (C/C++ based) officially supported 'NtDirect.dll'.

I appreciate that recommendation but I was wondering if there are other developers of automated strategies that ran into this issue when using Ninja's plugin as 'third party dll' (not running NinjaScript...). What workaround did you come up with (other than directly using NinjaTrader.Client.dll that is not officially supported and thus can change on any update of Ninja)?

[Atomic]

So I finally had it confirmed by a NinjaTrader developer that there indeed is a memory issue with repeatedly calling NtDirect.dll's 'OrderStatus' method. They suggested to use the not officially supported .net 'NinjaTrader.Client.dll' that exposes the same methods as the unmanaged (C/C++ based) officially supported 'NtDirect.dll'.

I appreciate that recommendation but I was wondering if there are other developers of automated strategies that ran into this issue when using Ninja's plugin as 'third party dll' (not running NinjaScript...). What workaround did you come up with (other than directly using NinjaTrader.Client.dll that is not officially supported and thus can change on any update of Ninja)?

Thanks for the info. I think I mentioned that my workaround was to shorten the return string length in the NTOrderStatus function in EasyLanguage. This has solved the problem for me for the most part along with periodic restarts of TradeStation. However, I'm curious about the NinjaTrader.Client.dll. How would I acquire and install it and does it work with TradeStation, if you know.

[JQuant]

I don't use it in TS but my own custom application. I am not sure if I could limit the length of the returned string (it still wouldn't solve the problem as I need to call this method many millions of times during the course of a week). You can use NinjaTrader.Client.dll in your own custom .net code, not sure how non-programmers could make use of it though.

[Rocket130713]

Can someone from Ninja development please set the "SetMaxReturnStringLength" to 13 permanently instead of 500 in the "canned" out-of-the-box NTOrderStatus function i.e. the function we get when we download ninja from the ninja website, as we were getting bad memory leaks due to the value set to 500 by ninja development team. the value should be set to 13 out of the box (there is no string length greater than 13 i.e. "PendingCancel" string is the longest here, it does not need to be set at 500 by ninja company). Could ninja please fix this problem. Thanks in advance. Regards, Rocket.

[NinjaTrader_PatrickH]

Hello Rocket130713,

Thank you for your post.

I will bring this to development's attention.