NtDirect Memory Utilization

[Atomic]

Sorry, I just now saw your post. Orchart is charting part of TradeStation. In any case, whatever the problem was I believe I have taken care of it by installing a couple of utilities which I will link to here in case anyone is interested.
The first (4Gb patch) allows a 32 bit app like TradeStation to utilize 4Gb of ram if you have a 64 bit system.
http://ntcore.com/4gb_patch.php

The second semi-continuously frees up memory which some programs may have taken up but not returned to the system.
http://majorgeeks.com/CleanMem_d5993.html

[NinjaTrader_Brett]

Very good thanks for the information. Wish you happy trading.

-Brett

[Atomic]

Something is continuing to happen. When I run process explorer, the amount of private bytes used by the orchart process just keep clocking up. A short time ago it was up to 1.5 Gb. I disabled the three things that are using the NtDirect dll and the private bytes dropped down about 0.4 Gb. I did not close anything other than disabling the three ELD's that were communicating with NT platform using the NtDirect. As soon as I turn them back on, the memory use starts to clock up. The amount of memory use in and of itself is not such a concern, but I know that eventually this will cause TS platform to crash. Please advise.

[NinjaTrader_Brett]

Hello,

Thanks for the follow up.

You have custom code written for NTDirect.dll? I suspect that your custom code is opening the DLL and not cleaning up its resources or is doing some repetitive action over and over that is causing the memory to increase? Any idea what in your custom code would cause this? What specific strategy do you enable that causes the memory to increase?

Also when is the last time you reimported the NTDirect.dll?

Thank You.

-Brett

[Atomic]

Hello,

Thanks for the follow up.

You have custom code written for NTDirect.dll? I suspect that your custom code is opening the DLL and not cleaning up its resources or is doing some repetitive action over and over that is causing the memory to increase? Any idea what in your custom code would cause this? What specific strategy do you enable that causes the memory to increase?

Also when is the last time you reimported the NTDirect.dll?

Thank You.

-Brett

Yes, but it is only EasyLanguage, which is just using the NtDirect dll functions to send orders from TS to NT and to check order status back from NT. I'm also using the external data feed to get market data to NT. I have no way mange memory within EasyLanguage. I feel that the problem lies with frequent use of the NTOrderStatus and NTFilled functions. I've never noticed the problem using just the external data feed before and I've used it quite a long time. It's only been recently that I have been doing more intensive order management using the NtDirect dll method. I guess my question is, where does the memory management (assuming that's what's needed) have to take place. In the programming of the dll or would it be something with TradeStation. I have some ability to regulate the frequency of the function calls, so I will try to slow those down in the meantime.

Thanks for your help.

[NinjaTrader_Brett]

Hello,

Do you have the ability to create a test copy of this code and strategy and remove the NTFilled call as a first step. Does this remove the problem? Basically we would need to remove items one by one until we identify the problem area then we can analyze it further.

-Brett

[Atomic]

Hello,

Do you have the ability to create a test copy of this code and strategy and remove the NTFilled call as a first step. Does this remove the problem? Basically we would need to remove items one by one until we identify the problem area then we can analyze it further.

-Brett

I have created some test code which I am running now. When I get it to the point where I can demonstrate the problem in a controlled fashion, I'll send you the code. To some extent, it seems to depend on the higher data rates you get during market hours. I've been running it in the overnight market for the past few hours and I'm getting some growth in the Private Byte number, but not as much as during the day. I may have something by day end to send you.

Tom

[NinjaTrader_Brett]

Sounds great.

When we test samples, make sure you restart NinjaTrader to insure no issues exist prior to you testing a sample and then thinking the sample also has issue.

-Brett

[Atomic]

Here is a little piece of code you can run to demonstrate the memory leak:
var:
intrabarpersist CurrentNtOrderID("");



once (getappinfo(airealtimecalc) = 1) begin
CurrentNTOrderID = "sell"+numtostr(barnumber,0);
value1 = NTCommand("PLACE","Sim101","sell",100,"Limit",clos e,0,"Day","",CurrentNTOrderID,"","");
print(formattime("hh:mm:ss tt",computerdatetime)," ",CurrentNTOrderID);
end;

while (NTOrderStatus(CurrentNTOrderID) = "Working" or NTOrderStatus(CurrentNTOrderID) = "PartFilled") and value1 < 20000 begin
print(formattime("hh:mm:ss tt",computerdatetime)," ",CurrentNTOrderID," ",NTOrderStatus(CurrentNTOrderID)," ",value1," ",NTFilled(CurrentNTOrderID));
value1 = value1 + 1;
end;
print(formattime("hh:mm:ss tt",computerdatetime)," ",CurrentNTOrderID," ",NTOrderStatus(CurrentNTOrderID)," ",value1," ",NTFilled(CurrentNTOrderID));

value1 = 0;

I run this on a 1 minute @ES chart. You may want to vary the number of contracts depending on how fast the market is. It will place a limit order on the first live tick. Once you get a return order status, the loop will generate a large number of function calls. When this begins, have Process Explorer open and watch the private bytes for Orchart.exe. They will just keep clocking up until TradeStation crashes. I can clear the private bytes in Orchart.exe only if I close all of the ELD's that have been communicating with the dll.

I hope this will be of some use.

Thanks.
Tom

[NinjaTrader_Brett]

Hello,

Thanks for taking the time to post this.

Unfortunately with this looping like that I actually would expect memory to increase. This type of looping is not something that can just be blanket tested like that would be the equivalent of testing a standard road car out to see how it drives out in the amazon jungle. Its a scenario where it is expected to fail.

Would not expect any issue in a real market scenario here with that. Could you provide any more specifics as to the scenario you are in that are running into issue with? Would recommend not making so many calls will have extreamly high overhead as you have found. Might make sure to space these calls out with some method.

-Brett

[Atomic]

Hello,

Thanks for taking the time to post this.

Unfortunately with this looping like that I actually would expect memory to increase. This type of looping is not something that can just be blanket tested like that would be the equivalent of testing a standard road car out to see how it drives out in the amazon jungle. Its a scenario where it is expected to fail.

Would not expect any issue in a real market scenario here with that. Could you provide any more specifics as to the scenario you are in that are running into issue with? Would recommend not making so many calls will have extreamly high overhead as you have found. Might make sure to space these calls out with some method.

-Brett

This is what I have done for now. All the function calls for NTFilled and NTOrderStatus have been put on a timer to limit their frequency. This will probably make it manageable. I guess it will have to. Thanks for you help.

Tom

[NinjaTrader_Brett]

Right,

Thanks for taking the time to post that.

Biggest thing is just reducing the checks as you have seen since this adds extreme overhead.

-Brett

[Atomic]

I have one more question, if you don't mind. I'm able to control the function calls so I don't approach the memory limit during a trading day. What I see is that the private bytes for orchart will go from 0.5Gb up to about 1.5Gb. then at the end of the day, I can reset this value by closing all of the ELD's that use the dll. It seems like there should be a simpler way to clear out the allocated memory which is not being used. If you know of anything I'd like to hear about it. Thanks.

Tom

[NinjaTrader_Brett]

Hello,

I am unaware of another method to clear this out, I would expect though if you didn't restart it would be no harm no foul though. Dont think you would just continue to see RAM usage go up day after day.

-Brett