Default optimizer: variable results

[ch9090]

Dear Support,

I've been trying to optimize (max. profit factor) a strategy few parameters on a portfolio. First I've used the genetic optimizer to narrow the parameters ranges and in a second step I've run the default optimizer to come up with precise figures.

It my understanding (based on the manual) that the default test every combination possible to find the best result possible. Thus my surprise, all things being equals (portfolio, time range, parameters), that the optimization returns slightly different results every time I run it.

It is not logical captain?
Many thanks for the help,
ch90

[NinjaTrader_Bertrand]

ch90, correct the default would be exhaustive searching through the complete parameter space - were you running in a connected or disconnected state here and observing those changed results?

[ch9090]

Dear Bertrand, good point, I am disconnected to save speed. Data series is day, same time frame, same portfolio, etc. Maybe I am missing something?

[NinjaTrader_Bertrand]

Ok on the process you take - do you rerun both the genetic and default sequentially to test this? Or really only the default on the exact same parameter space?

[ch9090]

I fixed the parameters space and run the default a couple of times because it was quite quick and I thought it might "take corners". The results provided are close but not the same. For example profit factor could be 2.43 or 2.37 with a MACD(10,21,11) or MACD (11,21,10) or MACD (10,22,10). Close but not perfectly equal.

[NinjaTrader_Bertrand]

Can you reproduce this as well with our default SampleMACrossOver strategy? If not then are you using any custom resources in the strategy that could change from run to run?

[ch9090]

SampleMACrossOver strategy returns the same results. What exactly do you mean by a "custom resource"?

Right now I am using basic indicators (RSI,MACD,price and volume). Price and volume are fixed. I am trying to optimize RSI and MACD. No rocket science here!

[NinjaTrader_Bertrand]

For example reading / write to textfile or a database or online resource for signals / inputs. Since it works as expected for you with the sample strategy, please simplify your setup to understand where this could stem from here.

[ch9090]

I'll go through it thoroughly.

Another strange thing I was not expecting. When I backtest the optimized set of parameters given by the optimization it also generates sligthly different results (PF, total net profit, # of trades, etc.).

[ch9090]

Dear Bertrand,

Please find below another few facts and observations that I hope will help.

Still optimizing my strategy on 1800 stocks, 5-year time frame (daily data) and a couple of good old indicators (no custom resources) such as RSI, MACD, price and volume.

I've used the default optimizer and it optmized 340 000 combinations in less than 10'. It is extremely quick as usually it takes 30' (disconnected state) to run 1 backtest combination. In theory it should have run during 120 days before producing a result. Thus my question, does it eliminate close combinations or can it narrow it down somehow?

I've also tested the same strategy with the genetic optimizer which provided some quick results and a PF of 12.3 with different settings than the default optimizer which generated a PF of 5.8. Clearly the GO produced better results than the default optimizer! It should be the opposite no?

Kind regards,
ch90

[NinjaTrader_Bertrand]

ch90, it's unfortunately next to impossible to deduct from this very large setup and a custom strategy involved. How did you arrive at the 340000 combinations needed? You're right for this large # the time was likely to quick - the default optimizer would not eliminate any iterations though like the GA, are you sure all stocks are being optimized here and that it would not skip certain instruments due to a data issue for example?

[ch9090]

Hi Bertrand, the # of combinations reflect the optimization ranges of my 7 parameters (10*10*6*12...).

I also filter stocks by price and volume so I guess out of the 1800 stocks the strategy will only retain 200 or 300, not more.

But still it is quite fast! I'll now try to optimize only 1 parameter and see how fast it is and then I'll add a second one, etc. until I reach 7. I'll keep you posted hoping this is of interest to the support, programming teams and other users.

cheers, ch90

[ch9090]

I've re-created my strategy with the wizard and tested it without manually editing the code.

I am optimizing the max profit factor (aggregated results true) and the GO gives me better results (PF 10) than the default optimizer (PF 5)! I've no idea why.

[NinjaTrader_Bertrand]

ch90, so you're sure you're running on the same session template, data, parameter ranges etc? So the optimization criterion is the profit factor here for both compared runs? Can you perhaps share the strategy and setup needed so I could test on our end to check into? Have you tried reducing the # of instruments you run this on, so for example only a DOW30 list? Would the same differences occur?

[ch9090]

Dear Bertrand,

Indeed I've fixed all parameters ranges, session template, time, list of instruments and compared both PF factors between the genetic optimizer and the default one.

I am a little bit desperate now especially since now all strategies (including sampleMAcrossover returning very strange results in the sense it only finds 1 to 3 trades while I used to have a few hundreds. Again same parameters, instruments, etc.

Could there be a dll or a corrupted file somewhere? Should I re-install NT? I'll send you my custom strategy by email. It is not very big and maybe you'll see something that I've missed (I just hope it is not too obvious or I'll have to commit a virtual ritual suicide to avoid the shame).

Many thanks for your perseverance on this one,

chris90