PH Optimizer unstable?

[molecool]

I am running NT7 7.1000.3 on Windows 7 on a 64-bit quad- core with 6GB of memory. Admittedly more RAM would aid my cause but it's a Mac G5 and I would have to replace the entire set to upgrade it to more. Yes, I'm running Windows natively and not in bootcamp.

Anyway, I am having very little luck with the PH Optimizer. Currently I'm in the process of optimizing a strategy with about 8 parameters, five of which have 4 sub settings. When running the optimizer on six months worth of data on a 1 min chart it immediately spikes toward 5GB of RAM in the performance tab of my task manager - CPU usage is rather mild at around 10% or less. After that the entire system just freezes up and I have kill the process. Have tried many settings and slimmed everything down (i.e. no logging and try to exit onBarUpdate early if no trade setup) but somehow I don't seem to be able to fix it.

I can't really share my strategy but I was wondering if such a scenario, which I incidentally find a rather limited, should really bring my system to my knees. I understand that it may take a while but given it's only a limited amount of params the memory should not blow up like that.

BTW, I'm having the same issue with the default optimizer as well. The only one that works is the NT7 Genetic optimizer but that one unfortunately is only using one core plus it does not seem to produce thorough mutations. For instance - there are a handful of settings I know test out very well (based on personal observation) and the GO never even comes close to those, despite an increase in children, mutation rate, etc.

Anyway, I'm kind of stuck here and would appreciate some input. A lot of work has gone into this strategy and I would like to make some progress over the long 4th of July weekend. So far it seems I'm banging my head against a wall, I'm sorry to say.

Thanks,

Molecool

[NinjaTrader_Bertrand]

Michael, we would unfortunately not be familiar with the custom optimizer you use - does it matter if you optimize for example on a shorter data span? There issues we resolved in later builds of NT7 with the GO optimizer, so I would suggest to upgrade to latest R6 and then rerun through your scenarios.

[molecool]

Okay, I'm downloading the latest build right now - will let you know how it flows. Question - I saw this fix in the comments for the .4 build:

Fixed
4375
Genetic optimizer
Genetic optimizer was erroneously only running on a single core even if strategy had no boolean or enum parameters

I actually noticed that when running the default genetic optimizer. And I have two versions of my strategy - one that uses enums and one that uses ints. Now, if I use the one with ints with the new .6 build will it use all four of my cores? And in general, why does it use only one core when encountering enums? Inquiring mind wants to know!

Thanks,

Molecool

[NinjaTrader_Bertrand]

Yes, with the fix it should use all your cores then if 'ints' are being employed instead of bools. I'll check with development on the reasons of this limitation.

[molecool]

Running build .6 now

I'm running .6 now and am trying the default GO. Seems like it's using all four cores but only at < 15% (see pic enclosed). What can I do to crank that up a little?

[koganam]

It is not so much it is unstable as it is that if a strategy does not generate many trades, then the optimization function quickly reaches saturation, and the PHGenetic does not seem to have a graceful recovery mechanism.

Most of the time, if that happens, I have to kill and restart NinjaTrader before I can do any more work.

[molecool]

That is not the case for me - I'm running it on a 1-min chart and there is at least one trade a week for each combination of params. Also, that memory limit is being hit within about 30 seconds on a run that should take seven to eight hours (when it works).

[koganam]

Easy to check Mole. Look in the output window and see if the "MinimumPerformance" figure has been hit. I have more than once tried to optimize a strategy where I had verified manually that trades MUST be taken, only to find that the PHGenetic hang for some particular optimization criterion: it might barf on "max profit factor", but not on "max avg profit" for example.

[molecool]

I don't see any of that in the output window during optimization. When would it appear - during optimization? Also, would I have to add special code in the OptimizationType in order to see that? BTW, I'm currently using the regular GO - the PHGO blows up on me within a minute. Perhaps it's because of that and I will take a look next time.

The other problem I have is with trying to optimize via SQN or ExpectancyScore. I am getting all those 777/888/999 results in my results window - which should not happen. Then I changed them to -777/-888/-999 and I'm still getting them in the results list for some reason. I would very much prefer to not even see them as I want to reserve it for high quality results - why list the bad stuff?

In general the quality of results I'm getting is also very bad - for instance: I have several high profit settings that I know via manual testing and none of them make it into my genetic optimizations. So, this means that the GO does not even come close to using values that produce good results, which it really should be doing after 7 hours of churning on 8 variables.

Any input would be appreciated, koganam - it seems you know your stuff and you can't beat hands on experience.

[koganam]

I don't see any of that in the output window during optimization. When would it appear - during optimization? Also, would I have to add special code in the OptimizationType in order to see that? BTW, I'm currently using the regular GO - the PHGO blows up on me within a minute. Perhaps it's because of that and I will take a look next time.

The other problem I have is with trying to optimize via SQN or ExpectancyScore. I am getting all those 777/888/999 results in my results window - which should not happen. Then I changed them to -777/-888/-999 and I'm still getting them in the results list for some reason. I would very much prefer to not even see them as I want to reserve it for high quality results - why list the bad stuff?

In general the quality of results I'm getting is also very bad - for instance: I have several high profit settings that I know via manual testing and none of them make it into my genetic optimizations. So, this means that the GO does not even come close to using values that produce good results, which it really should be doing after 7 hours of churning on 8 variables.

Any input would be appreciated, koganam - it seems you know your stuff and you can't beat hands on experience.

Best to show an example. I just run an PHGenetic optimization on my BreakoutTrader Strategy. Here is the end result that you get on a good run, (cut and pasted out of the output window):

ES 09-11: gen 33/200, 134.2 ips, max PERF: 1.47, stability PERF: 1.34 SwingStrength=3, LookBackPeriod=50, DoubleTouchToleranceTicks=0, StopLossTicks=11, TargetTicks=9, TrailStopTicks=18, EntryOffsetTicks=0, NumTargets=1, ReverseOnOppositeSignal=False, CheckMarketPositionBeforeEntry=True, UseTrailingStop=False, TurningMode=Strict, EntryMode=Passive, TrendStrength=Medium
2.06219609550766, 1000000000
ES 09-11: PH Optimizer End @ 6/30/2011 7:30:10 PM, duration: 00:01:13.7187500 (00:00:02.2339015 per iteration)
SwingStrength= 1
LookBackPeriod= 50
DoubleTouchToleranceTicks= 0
StopLossTicks= 30
TargetTicks= 8
TrailStopTicks= 18
EntryOffsetTicks= 0
NumTargets= 1
ReverseOnOppositeSignal= False
CheckMarketPositionBeforeEntry= True
UseTrailingStop= False
TurningMode= Strict
EntryMode= Passive
TrendStrength= High

(All emphasis mine). The green text are parameters. You see there are quite a few of all varieties. The red text is the MinimumPerformance value of which I spoke. I have actually upped it by a factor of 100 from the default. See the blue text? That tells me that the optimization run terminated after 33 iterations of the 200 that I specified. Why? Because it had hit the MinimumPerformance figure. When I used the default value of MinimumPerformance, it terminated after 17 runs.

When I have had the hang, that MinimumPerformance figure has usually been considerably exceeded (actually created a Processor overflow, with a number too large, which is probably what caused the hang). As many performance criteria have a divisor that includes the number of trades, if that denominator is really small, it is easy to get a divide overflow error. You will see an exception from PHGenetic in the NT log.

The problem with all Genetic Optimizers is that they can get stuck on a local optimum point. There is little to do but sometimes change the parameters in such a fashion as to either avoid the capture point, or make it suboptimal.

[molecool]

Now wait a minute - the minimum performance number is by default set to 0 - which means it's being ignored. Frankly, I haven't messed with that as I could not find any documentation on that value and have no idea what to set it to in the first place.

If it's set to 0 is it even part of the equation? And if so, what do you suggest I should set it to?

[koganam]

Now wait a minute - the minimum performance number is by default set to 0 - which means it's being ignored. Frankly, I haven't messed with that as I could not find any documentation on that value and have no idea what to set it to in the first place.

If it's set to 0 is it even part of the equation? And if so, what do you suggest I should set it to?

Maybe in the NT GO optimizer, not in PHGenetic. They may have been sourced off the same code by Pete S., but PHGenetic is a bit more advanced and uses a slightly different paradigm. In fact, if you look at the options, they are not even the same.

I thought that this thread was talking about PHGenetic. My comments are about PHGenetic. We have to stick to common ground, or there is no basis for a discussion.

[NinjaTrader_Bertrand]

Molecool, the picture you shared looks accurate, using the GA it would not 'peg' CPU's to a constant max as the default optimizer as it goes through the iterations needed. The one core limitation for bools / enums is tied to the NT core per development and it's on our list to look into for the future consideration.

[molecool]

Koganam: I was talking about either and got the gist of what you were saying. Thanks for taking the time to elaborate, very helpful.

[molecool]

Molecool, the picture you shared looks accurate, using the GA it would not 'peg' CPU's to a constant max as the default optimizer as it goes through the iterations needed. The one core limitation for bools / enums is tied to the NT core per development and it's on our list to look into for the future consideration.

I did a run last night with the .6 build and it was finally hitting some of the combinations I expected to see. Thanks for the clarification on the CPU load - looks like at least I have one GO working properly now.

Hopefully thanks to koganam's input I can shift to the PH OG now.