NT 7 B17 Freezing during optimization

[azuul]

When running some optimization it happens often lately that NT freezes up completely and I have to shut it down. In that case it is enough to click any NT Window - then the windows box appears that the program stopped working and I can simply choose to wait or close it.

This seem to happen basically when the RAM is full (can happen very quickly after a few minutes)
I am Running NT on a Win 7 64 Bit with 12Gb RAM.
The 32 Bit Version locks up earlier, since it does not use the 12GB RAM, the 64 Bit Locks up after about 11.x GB are in use according to Task Manager.

It is enough to have about 4-5 few parameters in addition with the Dataseries to be optimized to fill up the RAM very quickly. If RAM is full, CPU activity drops from average 45% to 14% (4 core cpu, 3.3GH) - again same behavior as many many month ago, I had the feeling this was fixed in earlier versions!!!

There is often no elapsed time or "time remaining" estimation to be seen in the lower right corner of the control center, only a green bar ( so hard to say when NT is locked up completely)
Sometimes during lockup, there appear 1-4 "additional Windows" in the Win Task Bar, which I cannot access. They seem to be triggered by NT somehow.

No NT Window can be moved during the freeze. Sometimes it helps to wait anyways, even if Windoes tells that the program is unresponsive - and NT does finish its job. Normally I do not have the patience to wait few hours for nothing, since there is no indication wether NT is still working at all.

I tried to optimize a strategy overnight, it locked up after several minutes - in this case there was shown a time: whey does the time show up sometimes and sometimes not? Is this a bug? At least elapsed times should be shown?

time elapsed was 3.xx minutes and remaining of 7.xx minutes - NT froze and stayed for about 8 hours until I closed NT this morning.

Not sure what is causing this lockup, I have not reinstalled NT for a few weeks, After startup it took about 1 GB in RAM immmediatelly, so i saved a clean workspace, far less RAM usage at startup but still same freezing problems.

The RAM problems seemed to be gone in 7.15 and 7.16, at least I did not have these issues with those versions. I hope this has to do with my installation or Data and not with the fact that you made changes to 7.17B that brought up old problems.

My workaround for now is not trying to optimize more than 2-3 Parameters at once...
But sometimes this is not enough!

Any ideas?

[NinjaTrader_Jason]

Hello azuul,

We did not make any changes in this regard as per version 7.0.0.17 as per the link below.

http://www.ninjatrader.com/support/helpGuides/nt7/release_notes.htm

Unfortunately your description indicates the optimization is too heavy on memory. You will need to use less parameters or less historical data to avoid it.

I am not sure why the lower right corner does not reflect remaining/elapsed time each optimization on your end. I suspect it is related to the freezing behavior. Do you notice any similarities between the occurrences?

[azuul]

I reset simulation account and Database - ran a rather simple optimization with an old strategy. The RAM usage was below 2 GB since not too much historical Data was used - I used several parameters - estimated Time was 1 hour 20 min - after 17 Minutes NT froze totally - but interestingly, the task manager still showed a lot of CPU load - after about the 1hour 20 min the cpu load went down to almost 0 - NT still was frozen, had to terminate via task manager.

I have to add that I was running 3-4 strategies at the same time. I still think this should not happen

[NinjaTrader_Austin]

azuul, would you mind posting the trace files from when your NinjaTrader locked up?

You can either post them here, or you can send it in to support at ninjatrader dot com attn 'austin - thread 30896, ticket 335467'.

[azuul]

There is nothing unusual in the trace written, since I close it via task manager or Windows shuts it down. However I found out that NT is still running the optimization process in the background - only the green bar does not show any time or progress - after a while it can suddenly show e.g. 30% or 100% and a time. Somehow only the GUI/controlcenter seems frozen. In this case any open chart also stops updating.
I also observed this scenario when RAM was not filled up and CPU showed not much usage. (just to exclude system performance issues as I suspected earlier.)

When I try to move or access any NT-window during this frozen GUI state, it can happen that all NT-windows become greyish and stay grey, and even after some waiting time and after CPU activity has dropped (indication that optimization has finished), NT stays that way and I have to kill the NT process via task manager.

.

[NinjaTrader_Jason]

Could you please post or send us your trace file. Please include around what time you performed the optimization if you can recall.

[azuul]

I deleted Beta 17 completety - installed Beta 18 - still the GUI seems to freeze during more complicated optimization tasks, as there is no green progress bar, no time elapsed time or time remaining. (timer and progress bar working for fast, simple tasks)
When NT becomes gray after trying to e.g. access the File menu and Windows tells that the program is not responding, it "recovers" and if left alone finishes the optimization eventually.
Trace after closing NT with frozen GUI:

2010-07-01 15:42:23:298 ******************* Session Start (Version 7.0.0.18) *******************
2010-07-01 15:42:23:298 Attaching global exception handlers...
2010-07-01 15:42:23:423 InstallDir='C:\Program Files (x86)\NinjaTrader 7\'
2010-07-01 15:42:23:423 UserDataDir='C:\Users\blue\Documents\NinjaTrader 7\'
2010-07-01 15:42:24:595 Cbi.Globals.MachineId: True/True
2010-07-01 15:42:24:595 MachineId='62531421720AAA9F9DA4F250DA6B2425'
2010-07-01 15:42:24:595 OS='Microsoft Windows NT 6.1.7600.0'/'Win32NT'
2010-07-01 15:42:24:611 Processors: available=8 enabled=8
2010-07-01 15:42:24:611 .NET/CLR Version='2.0.50727.4927'/32bit
2010-07-01 15:42:24:611 Processor='Intel(R) Core(TM) i7 CPU 975 @ 3.33GHz'
2010-07-01 15:42:24:611 PhysicalMemory=12292 MB
2010-07-01 15:42:24:611 Timezone=EST +6 hour(s)
2010-07-01 15:42:24:611 OptimizedCpuPerformance=True
2010-07-01 15:42:24:626 Chart Hot Keys: (removed to shorten this post)
2010-07-01 15:42:24:658 Cbi.Licence.Download5
2010-07-01 15:42:26:205 Cbi.Licence.ProcessLicense10
2010-07-01 15:42:26:205 Cbi.Licence.ProcessLicense13
2010-07-01 15:42:26:205 Cbi.Licence.Download99: 'Type=SimulationOnly State=Verified StartDate=01.01.2005 EndDate=01.12.2099 Collective2 Dtn ESignal External Provider2 Provider3 Provider7 Replay Simulator Yahoo Gain Hosted InteractiveBrokers MBTrading Patsystems Provider1 Provider4 Tda TrackData TradingTechnologies AdvancedStrategyManagement AutomatedTrading BasicEntry Charting SuperDom'
2010-07-01 15:42:26:205 Order entry hot keys disabled
2010-07-01 15:42:26:205 Creating file type watcher...
2010-07-01 15:42:26:205 Retrieving license...
2010-07-01 15:42:26:205 Starting splash...
2010-07-01 15:42:26:376 Migrating ...
2010-07-01 15:42:26:376 Copying custom DLLs...
2010-07-01 15:42:26:376 Running Cbi.Globals.Install()...
2010-07-01 15:42:26:376 WARNING: Session Break (Version 7.0.0.18)
2010-07-01 15:42:26:392 Clearing C:\Users\xxxx\Documents\NinjaTrader 7\bin\Custom\Tmp...
2010-07-01 15:42:26:392 Loading C:\Users\xxxx\Documents\NinjaTrader 7\bin\Custom\NinjaTrader.Custom.dll...
2010-07-01 15:42:26:439 Loading C:\Users\xxxx\Documents\NinjaTrader 7\bin\Custom\NinjaTrader.Vendor.dll...
2010-07-01 15:42:26:439 Initializing database...
2010-07-01 15:42:26:486 (Db) Db.Globals.get_ConnectionString: >Data Source="C:\Users\xxxx\Documents\NinjaTrader 7\db\NinjaTrader.sdf"<
2010-07-01 15:42:26:986 Instantiating serializers...
2010-07-01 15:42:26:986 Deleting temporay files...
2010-07-01 15:42:26:986 Loading server side config file...
2010-07-01 15:42:28:048 Creating user defined methods files...
2010-07-01 15:42:28:048 Loading master instruments...
2010-07-01 15:42:28:345 Loading instruments...
2010-07-01 15:42:28:689 Loading default symbol list...
2010-07-01 15:42:28:720 Loading available connections...
2010-07-01 15:42:29:033 Connecting ATI server...
2010-07-01 15:42:29:033 Cbi.AtiServer.Connect1
2010-07-01 15:42:29:033 Cbi.AtiServer.ConnectNow: Port=36973
2010-07-01 15:42:29:033 Cbi.AtiServer.ConnectNow2
2010-07-01 15:42:29:033 Syncronizing executions...
2010-07-01 15:42:29:064 Registering chart styles...
2010-07-01 15:42:29:251 Registering bars types...
2010-07-01 15:42:29:251 Hiding splash...
2010-07-01 15:42:29:267 (Unknown) Cbi.Connection.Connect0 Disconnected assembly=7.0.0.18 date=01.07.2010
2010-07-01 15:42:29:267 (Unknown) Cbi.Connection.Connect1
2010-07-01 15:42:29:361 (Gain) Cbi.Connection.Connect9
2010-07-01 15:42:29:423 (Gain) Cbi.Connection.ProcessEventArgs.ConnectionStatusEv entArgs Connecting Connecting NoError
2010-07-01 15:42:29:423 (Gain) Cbi.Connection.ConnectNow1: DataConnection=False
2010-07-01 15:42:29:423 (Gain) Cbi.Connection.ConnectNow2
2010-07-01 15:42:29:423 (Gain) Cbi.Connection.ConnectNow.CreateLoader
2010-07-01 15:42:29:423 Server.Client.Connect
2010-07-01 15:42:29:439 Server.Client.Loop0-0: 64.202.118.179/31654
2010-07-01 15:42:29:923 Server.Client.Loop1-0
2010-07-01 15:42:29:923 (Gain) Cbi.Connection.ConnectNow.Connect
2010-07-01 15:42:29:923 (Gain) Adapter.Gain.Connect: user='' mode=Demo multi=False
2010-07-01 15:42:30:283 (Gain) Cbi.Connection.ConnectNow9 ok
2010-07-01 15:42:30:283 (Gain) Cbi.ConnectionStatusEventArgs.Process: Connecting Connecting
2010-07-01 15:42:30:970 (Gain) Adapter.Gain.OnGetRatesServerAuth
2010-07-01 15:42:30:986 (Gain) Adapter.Gain.OnGetRatesServerAuth : C55DF94BCC7FA40B17F18D2396985EA1
2010-07-01 15:42:31:001 (Gain) Adapter.Gain.OnRateServerAuth : running in DEMO mode
2010-07-01 15:42:31:861 (Gain) Adapter.Gain.OnRateServiceDeal DEAL0
2010-07-01 15:42:31:861 (Gain) Adapter.Gain.SetConnectionStatus:
2010-07-01 15:42:31:861 (Gain) Cbi.Connection.ProcessEventArgs.ConnectionStatusEv entArgs Connected Connected NoError
2010-07-01 15:42:31:908 (Gain) Cbi.Connection.ProcessEventArgs.AccountEventArgs: name='Sim101'
2010-07-01 15:42:31:923 (Gain) Cbi.AccountEventArgs.Process: Sim101
2010-07-01 15:42:31:939 (Db) Db.Globals.Recover1: 'Sim101'
2010-07-01 15:42:32:033 WARNING: Unable to recover orders/executions/strategies for instrument(s) 'FDAX 09-10' since provider does not support these instrument types.
2010-07-01 15:42:32:033 (Db) Db.Globals.Recover2: 'Sim101'
2010-07-01 15:42:32:033 (Gain) Cbi.Account.RestorePositionExecutions1: account='Sim101'
2010-07-01 15:42:32:033 (Gain) Cbi.Account.RestorePositionExecutions9: account='Sim101'
2010-07-01 15:42:32:048 (Gain) Cbi.AccountUpdateEventArgs.Process: Account='Sim101' ItemType='BuyingPower' Currency='Unknown' Value=20000000
2010-07-01 15:42:32:048 (Gain) Cbi.AccountUpdateEventArgs.Process: Account='Sim101' ItemType='CashValue' Currency='Unknown' Value=10000000
2010-07-01 15:42:32:048 (Gain) Cbi.AccountUpdateEventArgs.Process: Account='Sim101' ItemType='ExcessEquity' Currency='Unknown' Value=20000000
2010-07-01 15:42:32:048 (Gain) Cbi.AccountUpdateEventArgs.Process: Account='Sim101' ItemType='InitialMargin' Currency='Unknown' Value=0
2010-07-01 15:42:32:064 (Gain) Cbi.AccountUpdateEventArgs.Process: Account='Sim101' ItemType='RealizedProfitLoss' Currency='Unknown' Value=0
2010-07-01 15:42:32:189 (Gain) Cbi.ConnectionStatusEventArgs.Process: Connected Connected
2010-07-01 15:42:32:189 (Gain) defaultTimeToClose=01.07.2010 00:00:00 flattenAll=False
2010-07-01 15:42:32:205 (Unknown) Cbi.Connection.Connect0 Disconnected assembly=7.0.0.18 date=01.07.2010
2010-07-01 15:42:32:205 (z_esignal) Cbi.Connection.Connect9
2010-07-01 15:42:32:205 (z_esignal) Cbi.Connection.ProcessEventArgs.ConnectionStatusEv entArgs Connecting Connecting NoError
2010-07-01 15:42:32:205 (z_esignal) Cbi.Connection.ConnectNow1: DataConnection=False
2010-07-01 15:42:32:205 (z_esignal) Cbi.Connection.ConnectNow2
2010-07-01 15:42:32:205 (z_esignal) Cbi.Connection.ConnectNow.CreateLoader
2010-07-01 15:42:32:205 (z_esignal) Cbi.Connection.ConnectNow.Connect
2010-07-01 15:42:32:220 (z_esignal) ESignal.StandardAdapter.Connect: user='xxxx' mode=Live newsMinutesBack=30 dailyServer='CM*.esignal.Com' internationalTickServer='CM*.esignal.Com' level2Server='' newsServer='CM*.esignal.Com' tickServer='CM*.esignal.Com' proxy='/0//'
2010-07-01 15:42:32:220 (z_esignal) ESignal.StandardAdapter.Connect: winros.exe running
2010-07-01 15:42:32:220 (z_esignal) Cbi.Connection.ConnectNow9 ok
2010-07-01 15:42:32:220 (z_esignal) Cbi.ConnectionStatusEventArgs.Process: Connecting Connecting
2010-07-01 15:42:32:236 (z_esignal) ESignal.StandardAdapter.OnStatus: server=DataManager status=Connected native=0
2010-07-01 15:42:34:126 (z_esignal) ESignal.StandardAdapter.OnStatus: server=NewsServer status=Connected native=0
2010-07-01 15:42:34:189 (z_esignal) ESignal.StandardAdapter.OnStatus: server=TickServer status=Connected native=0
2010-07-01 15:42:34:251 (z_esignal) ESignal.StandardAdapter.OnStatus: server=HistoryServer status=Connected native=0
2010-07-01 15:42:34:892 (z_esignal) ESignal.StandardAdapter.OnStatus: server=IntlTickServer status=Connected native=0
2010-07-01 15:42:34:892 (z_esignal) Cbi.Connection.ProcessEventArgs.ConnectionStatusEv entArgs Connected Connected NoError
2010-07-01 15:42:34:892 (z_esignal) Cbi.ConnectionStatusEventArgs.Process: Connected Connected
2010-07-01 15:42:34:892 (z_esignal) defaultTimeToClose=01.07.2010 00:00:00 flattenAll=False
2010-07-01 15:42:34:908 Setting global sim mode...
2010-07-01 15:42:35:142 Setting ATI...
2010-07-01 15:42:35:142 Starting messages timer...
2010-07-01 15:42:35:142 Starting news timer...
2010-07-01 15:42:35:142 Activating hot keys
2010-07-01 15:42:35:142 ApplicationStart done, took 11,9 secs
2010-07-01 15:42:35:142 Creating CC...
2010-07-01 15:42:37:470 Showing CC...
2010-07-01 15:42:37:533 CC created
2010-07-01 15:42:37:533 Menu item created
2010-07-01 15:42:37:626 Opened workspace 'C:\Users\xxxx\Documents\NinjaTrader 7\workspaces\Untitled1.xml'
2010-07-01 15:42:37:626 Switched workspace 'C:\Users\xxxx\Documents\NinjaTrader 7\workspaces\Untitled1.xml'
2010-07-01 15:44:22:111 (z_esignal) Data.Bars.GetBarsNow: instrument='FDAX 06-10' from='2010-05-28' to='2010-06-16 23:59:59' period=1 Tick splitAdjusted=False dividendAdjusted=False bars=0 session='Eurex Equity Index Futures'
2010-07-01 15:44:25:423 (z_esignal) Data.Bars.GetBarsNow: instrument='FDAX 09-10' from='2010-06-13' to='2010-06-30 23:59:59' period=1 Tick splitAdjusted=False dividendAdjusted=False bars=0 session='Eurex Equity Index Futures'

[NinjaTrader_Jason]

Hello azuul,

Could you please send me the strategy you optimized, so I can test the same optimization on my end. Please include what optimization parameters you used as well (include a screenshot of the optimization settings).

You can send it to support[at]ninjatrader[dot]com. Please reference this forum thread and put 'Attn Jason' in the subject header.

[billr]

For what it's worth, I was experiencing the exact same problem. After exhaustive back and forth help (from Kyle in support) we narrowed it down to a 3rd party application that needed some adjustment from the vendor to correct the problem. Oddly, it took some time after the installation of the 3rd party app before the problem began, so I didn't think it was the problem. Fortunately, the vendor made some adjustments and the problem was solved.

[cgeorgan]

Optimizer freezing up

I've experienced optimizer freeze-ups that are slightly different in this case -

I can often get optimizations to run correctly, but for the last two V7 builds (17, 18) the "graphs" for the PnL curves don't run correctly.

Regardless, what I've seen fairly consistently is that my optimizer will run into a situation where it's "saving" in the status bar...this will become a persistent "saving", and I'll know it's entered a freeze state. If I close the optimizer window, the status will change to "Aborting. Please wait..." This is irrecoverable - NT will need to be shut down at this point to escape this.

Furthermore, if I try to re-open a strategy and run an optimization on it, I'll get an error window informing me that the optimization could not proceed because 'there are no parameters to optimize."

*EDIT: In this situation, shutting down NT will not result in the cessation of the NinjaTrader.exe process, which will continue to run until manually ended, even if the NT Exit procedure appears to go correctly -

*EDIT 2: The situation where the optimizer won't show the correct graph for cumulative profit (on the first attempt to load that tab) can be remedied by choosing one of the other "graphs", such as weekly profit, then cycling back to "cumulative profit."

[NinjaTrader_Jason]

Hello cgeorgan,

Can you please tell me what is displayed incorrectly in the Graph-tab. Could you perhaps post a screenshot that displays the issue.

When NinjaTrader became unresponsive upon optimizing your strategy, how much historical data and optimization parameters were used. Does it make a difference if you use less data and parameters?

Unresponsive behavior can occur upon optimizing a strategy. This usually indicates the optimization is too heavy for your machine. You will need to use less historical data or less optimization parameters.

What strategy did you try to optimize when you received that error message? Does the strategy contain parameters to optimize?

[cgeorgan]

Ah - In terms of the PnL graph not appearing, I guess it just takes a little bit to load...I was waiting only a few seconds and seeing a flat line, but after about 10 seconds the graph populated.

In terms of locking up, I wouldn't say I'm doing too much to it. I'm really only optimizing two fixed parameters on tick data over about 30 days. It very often works - but after repeated use tends to stop working. I'll keep an eye open for what might be causing it...I'm not confident it's a RAM issue, as I'm watching the NinjaTrader process for any spikes (doesn't really happen or go anywhere above 500k in usage).

[NinjaTrader_Jason]

Hello cgeorgan,

I see, good to hear the PnL graph displays as expected.

Sounds good, please let me know if you notice anything that could direct us to the cause.

[azuul]

I found the solution to my problem:

There were Print("text") statements in the code which I did not disable before optimization, what caused several problems:

1. RAM(memory) filled up very quickly until maximum was reached.
2. progress bar not updating and no time remaining showing or "jumping" and halting.
3. NT-windows became unresponsive, NT froze.
4. aborting not working or took very long (even if NT was not frozen yet)

The mean thing is: shorter optimizations with 1 or few dataseries and few parameters will not crash or freeze NT even if there is one or more Print("text") statements.

Maybe this is a bug, I am not sure if you have mentioned this behavior somewhere (although it seems logical and obvious) maybe it should be noted under performance tips or under optimization, that Print("...") statements can cause memory to fill up quickly + lock NT. I assume print could/should also be ignored by the optimizer.

NT runs optimizations now fast and smooth with much less RAM usage compared to earlier.

[NinjaTrader_Jason]

Hello azuul,

Good to hear you found the cause of the unresponsive behavior. Unfortunately this can occur when your strategy is heavy on CPU/memory load and using a large set of historical data.

I will forward your feedback regarding the Print statements to my superiors.