Getting started w/optimization

[NinjaTrader_Brett]

Hello,

Can you post a quick screenshot of an example of the optimization settings you used and then also a screenshot of the parameters used for the WFO run.\

Does this occur with the SameMACrossOver on your system or just your strategy?


I look forward to assisting you further.

[bluelou]

Hello,

Can you post a quick screenshot of an example of the optimization settings you used and then also a screenshot of the parameters used for the WFO run.\

Does this occur with the SameMACrossOver on your system or just your strategy?


I look forward to assisting you further.

I'm pretty sure it's happening w/just a simple MA XO, too. Here's NT's MA XO strategy where I've added a single indicator.

I've attached 4 pix:
1) pic # NT4 shows the WFO inputs. The date range is 010110-113010. The strategy is the NT7 canned MA XO strategy + a single volatility indicator added.

2) pic # NT1 shows the P&L over the test period is -30.14% and includes 11 trades.

3) pic # NT2 shows the parameter values used to generate the -30.14% return. I don't know where these parameter values came from. I'm guessing they're part of the WFO optimization but I'm not sure. That's part of what I'm trying to figure out.

4) pic # NT3 shows the individual WFO tests. You can see that these tests don't add up to the -30.14% return in pic # NT1.

I've attached the indicator and will attach the strategy in the next post since only 5 attachments are allowed. Again. this is the canned NT MA XO strategy w/a single indicator added. I'm getting the same type of output for all of my strategies using WFO.

[bluelou]

Strategy file attached.

[NinjaTrader_Brett]

Hello,

Thanks for posting this. I did not have a change to look into today. I will respond tomarrow after I've looked into this.

Thanks for your patience.

[bluelou]

Brett/Josh,
Re: post #13 in this thread - Problems with Genetic Optimizer

In addition to the problems with the WFO I'm also having problems with the Genetic Optimizer (GO). Per Josh's suggestion regarding unusual GO results in a custom strategy I added log statements and confirmed that the GO was not running any permutations of parameter values. The optimizer would just stay w/the initial parameter values throughout the test as evidenced by a constant value in the log file.

So, I tried going back to basics by running a GO of the simple MA XO strategy attached in posts #17 and #18. I've attached the GO settings as a .jpg file. The genetic optimization of this simple 3 parameter strategy never completed and I had to kill the process after 2 1/2 hours. Also, NT7 was using 4.8 GB RAM when I killed it and I've had to reboot the PC 2x so far after running the GO.

Please try to run the attached strategy on your own using the GO settings that I've used and let me know what the problem is or what I should be doing differently. Again, this is just NT's canned MA XO strategy plus the simplest custom indicator that I could think of.

[bluelou]

Is anyone from NT going to try to help me here? It's been over 24 hours since I posted the 1st of 2 problems (and nearly that long for the 2nd problem) that I'm having with optimization.

GO and WFO aren't working for the simplest of strategies that I've posted here for anyone from NT to test.

[NinjaTrader_Brett]

Hello,

Yes we will look into I posted that I would respond today on this. Most likely I will respond after market close when I have the ability to test this. As of currently with NinjaTrader 7 production release this requires my full attention at this time, however after the market close today I should be able to test this and get back to you.

Thanks for your patience.

[bluelou]

Okay, thanks for the update. I'll look to hear back from you after the close.

[NinjaTrader_Brett]

Hello,

Can you please try the optimization run again on SampleMAXO, the optimize run with all your exact settings took me 30 seconds. With no crash.

Also, on your original question:

The GO will not get the same results with each run this is the nature of a GO as GO does not guarentee optimize absolutes.

Also take a look at my screenshot I do not see anything off here, All parameters tested are in the optimize range and are as I expect.

What do you see wrong here again? We need to narrow this down to exact issue.

I look forward to assisting you further.

[bluelou]

Brett,
Did you try the GO settings (not WFO, but GO) I used in post #20, too? BTW, I never said that I expected to get the same results in different optimization trials. I'm not sure why you mentioned that.

I'll try the MAXO test again but it didn't work for me earlier today. I'll get back to you in a few minutes.

[NinjaTrader_Brett]

Hello,

Just ran the GO, no issues.

[bluelou]

Did you try it on AUDUSD data? It's still not working for me. I killed the process after 10 minutes. It was using >50% of my CPU and > 4 GB memory. I'l try it on another data set - but I'm not sure why that would matter.

What else would account for the fact WFO and GO are working for you with the same strategy but aren't working for me at all?

[NinjaTrader_JoshP]

Bluelou,

First things first. I suggest you decrease your GO settings. You are telling GO to run more iterations than it would have to through a brute force test with your parameter settings.

5*5*10 = 250 combinations possible.
Your current go settings is asking for testing of 25*20 = 500 iterations.

With those settings, you might as well run the default exhaustive optimizer.

[bluelou]

Bluelou,

First things first. I suggest you decrease your GO settings. You are telling GO to run more iterations than it would have to through a brute force test with your parameter settings.

5*5*10 = 250 combinations possible.
Your current go settings is asking for testing of 25*20 = 500 iterations.

With those settings, you might as well run the default exhaustive optimizer.

Josh,
I've tried your suggestion and went w/5 generations rather than 20. The GO still isn't working properly. I've just done 2 test runs of the GO, haven't tried WFO again yet.

On the 1st run, generations = 5, the process took about 5 minutes and it didn't cycle through all of the parameter values as evidenced by a log file. For one of the three parameters the GO only tested a single value. CPU usage hit 95% and memory hit > 4 GB. Memory stayed bloated > 1.5 GB for several minutes after the test so I restarted NT.

I tried the exact same GO test a 2nd time to see if it would cycle through more of the parameter space but I killed the process after 20 minutes of waiting. Again, CPU usage was in the 70-95% range and memory usage was in the 3-4 GB range.

After the failed optimizations and when re-opening NT7 the memory remains bloated - currently at 900 MB - even though NT isn't even being used for testing/trading - it's just open. I'm using v23. I'm going to try the latest release to see if that makes a difference.

Any other suggestions? We're both running the exact same code but with totally different outcomes.

[NinjaTrader_Brett]

Hello,

Your outcome is expected however.

The definition of a genetick optimization is that it wont test all parameter values. It does random sampling as its start bases. The goal of GO is to allow the backtest to not take as long.

Heres the test for you on your system.

The GO that you ran that took 5 minutes. If you run this test over again please insure it takes about 5 minutes and the results are consistant.

Then please run a normal optimize instead of the GO. This test should take longer. Does it, or does it not?