Assessing Completeness of the Genetic Optimizer result

[tony_28217]

Issue: We only know THAT the GO achieved an objective, but we don't know HOW GOOD that objective was relative to what could have happened.

Idea 1:
I realize we could just increase the generation size and # of generations amongst other variable but... I had a thought.

What I am thinking : I'd like to say "for a date range, time frame, strategy, and instrument, we ran [x,y,z,a,b,c,d,e,f] GO criteria with parameters [par a, par b, par c, etc]" and while we were able to achieve the appropriate result, is it the best?

Question: Is there a logical way to record the permutations of parameters as well as the above criteria (date, time, instrument, etc) that were inputted into one GO (in a list or array, perhaps?) then ingest this into a new GO? Basiclaly tell it, this is what has been run... where do you want to go from here? It's not easy but... if we ingest the history, and it's all "complete" we'll run one generation and be done. If not, we'll know more generations are needed.

We could change certain flexible GO paramaters on the second round (mutation rate, etc) but not the core dates/strategy between rounds. This isn't necessary but could be used for robustness...

I'm looking for conceptual help here..

Idea 2:
Alternatively: Given the genetic optimizers opaqueness in what is happening IN process natively, there would appear to need some sort "calculator" that predicts how many tests might need to be run to sufficiently cover [xxxxx] many permutations?

Asked another way: Is there a formula that allows for the derivation of settings of the Genetic Optimizer to determine it's likelihood of success, with the intent to drive fewer runs for time/resource efficiency?

Please don't say just run more from the beginning because we can't really know it's "done" unless it stops early...

Thanks in Advance

[NinjaTrader_Emily]

Hello tony_28217,

Thank you for your post.

There are two settings that could result in the optimizer terminating early if they are met. These could be used as a tool to decide what parameters to select before starting another optimization. One is Convergence Threshold and the other is Minimum Performance. Here are the descriptions of each from the help guide:

Convergence Threshold:

Setting this will terminate the Genetic Optimization if there is more than a certain number of duplicate children in a single generation, defined by the Convergence Threshold value. This allows the optimization to terminate if no new work is getting done because it has already converged in on the most optimal solution. Example: In the screenshot above Generation size is set to 25, therefore each generation will contain 25 children, if 20 of these children are duplicates that have already been tested then the optimization will be terminated.

Minimum Performance:

If this performance value is reached before all generations are evaluated the optimizer will end and present results immediately, where the type of this value is directly tied to your used optimization fitness metric (i.e. Profit Factor). A Value of 0 means no minimum performance is in use.

The two ideas you have suggested are not currently features of the Genetic Optimizer, though I will gladly submit those as feature requests to the Development Team. I will follow up with internal tracking numbers for your reference as soon as the requests are created.

Thanks in advance for your patience.

[tony_28217]

Thanks Emily.

I do understand both Convergence threshold and Minimum Performance. They both feature prominently in my setting changes for GO but they are good to call out nonetheless.

I'll wait for some indication of progress on this one. I realize it's lower priority but could persuade more people to use the GO and get more done if they know just HOW MUCH faster they can get through optimization than just by using the standard optimization.

There is one other question that comes to mind:

I have a 24 core processor and NT8 can typically run 24 iterations of an optimization at once. I wasn't thinking and set the Generation Size to 36 (or 1.5 * [my core count]). I then realized this may be problematic as the mutation would not occur until after the 36 are run (e.g. 24 then 12, then 24, then 12, etc). Basically is this like I described it (and for efficiency) we should have Generation size = integer * core count in the docs? Efficiency wise.

Thanks!
Tony

[NinjaTrader_Emily]

Hello Tony,

Thank you for your reply.

Optimizations are designed to take advantage of all available resources. It would require some testing to see what happens with the cores when you set the generation size to a number larger than the number of cores you have. You could probably see how this works in the Task Manager:

• Right-click the Windows Start menu and select Task Manager
• If it is not already selected, click "More details" at the bottom of the window
• Select the "Performance" tab
• Click CPU
• Right-click the graph and check that Change graph to has "Logical processors" selected

This will give you a graph for each core of your PC. You may then view these graphs while running a GO to see how the cores are utilized. Performance can also vary if you are using a virtual cloud/server. For more info on optimization performance in general, please see the "Understanding factors that affect optimization performance" section on the following page:

https://ninjatrader.com/support/helpGuides/nt8/optimize_a_strategy.htm

I will be following up once I have the IDs for the feature requests. In the meantime, please don't hesitate to reach out with any additional questions or concerns.

[QuantKey_Bruce]

tony_28217 Even if you did all of these things, you can't be certain a GO has achieved the global maximum fitness. That's inherent in the fact that there could be various local optima that the GO hasn't even seen nor gotten near in a large optimization space, so regardless of what heuristic you add, you can't know what's out there if you haven't gotten a taste of it nor near it. I think it would be important to accept that the GO is doing a good job of sampling the space, given the resources it is given, but if you truly want to sample the space evenly and know that you've sampled in each "area" of a continuous optimization space, you're going to need to do an exhaustive search using some reasonable interval in order to know that. This is even more true if the space has irregularities or is not linear - in that case, you may well need to sample nearly every value to have high confidence you haven't missed the global optimum. But, all of this is largely moot - in trading the "best" parameters are usually shifting, so reaching the absolute global optimum is not necessary. What is necessary is to have a good business process around the optimization and to get a "good" result even if you can't be confident it is the "absolute best", and to rinse and repeat when appropriate.

[tony_28217]

Fair point QuantKey_Bruce . I guess that's why it might be useful if there was some sort of indicator or 'calculator' that conveys "this is an approximately sufficient set of GO criteria for such a space?" Or "this set of GO criteria would likely be 'good' to analyze of space of sizes [x] to [y]"

Just thinking a feedback loop within the program even with a binary check on "does this seem sufficient given the inputs" could be helpful.

I realize the inherent flaw of restarting an experiment midstream (which must rely on previous conjecture) and do think that part was a stretch for usability.

I just am striving for some context of what 'good' could be considered. even if it's a range of 'goodness'

[NinjaTrader_Emily]

Hello Tony,

Thanks for your patience.

The internal tracking numbers for your feature requests are SFT-6104 for idea Home and SFT-6105 for idea Forum of your initial post in this thread. Please reference these internal tracking numbers when contacting Platform Support if you ever have questions regarding these feature requests.

When a feature request is implemented, you'll find a description of the new feature in the release notes:

• https://ninjatrader.com/support/help...ease_notes.htm

Thank you for using NinjaTrader.