Question regarding the manner in which the Strategy Analyzer utilizes system resources; specifically RAM.
For analytical reasons I am retaining ALL iterations when running an exhaustive optimization. My system has 64GB DDR4 RAM (4 x 16GB, dual-channel @ 1600mHz).
My question:
How does the Strategy Analyzer "sort-mechanism" (to the extent there is one) function? I'll give an example: say you have an optimization with 10,000 iterations -- when setting the "Keep Best # Results", how does the Strategy Analyzer handle the retention logic and is there any check on whether ALL iterations are requested thereby deeming any inter-processing sorting irrelevant? If "Keep Best # Results" were 10, for example, then how I imagine the Strategy Analyzer functions is that it runs the first 10 iterations, stores those in memory, and then on the 11th iteration it would referentially compare to the 10th-worst iteration and if Iteration11 > Iteration10 then it retains Iteration11 and removes Iteration10. This then repeats for the remaining n iterations until all iterations have completed. Is this a fair characterization?
What I notice when running these iteration-dense optimizations is that my RAM space is fully consumed and after that threshold is hit a pagefiling mechanism takes over and RAM is incrementally "freed", albeit only a couple of gigs at a time, and there is activity on the system disk (SSD). This makes the remainder of the optimization extremely slow.
In the unique instance in which ALL optimization iterations are being requested via the "Keep Best # Results" parameter, wouldn't it make more sense for the Strategy Analyzer to dump the entirety of the 64GB to disk and start fresh? I'm sure there are probably numerous architectural reasons why this may not be possible but I'm simply curious to understand the mechanics better.
Thanks so much for the info.
For analytical reasons I am retaining ALL iterations when running an exhaustive optimization. My system has 64GB DDR4 RAM (4 x 16GB, dual-channel @ 1600mHz).
My question:
How does the Strategy Analyzer "sort-mechanism" (to the extent there is one) function? I'll give an example: say you have an optimization with 10,000 iterations -- when setting the "Keep Best # Results", how does the Strategy Analyzer handle the retention logic and is there any check on whether ALL iterations are requested thereby deeming any inter-processing sorting irrelevant? If "Keep Best # Results" were 10, for example, then how I imagine the Strategy Analyzer functions is that it runs the first 10 iterations, stores those in memory, and then on the 11th iteration it would referentially compare to the 10th-worst iteration and if Iteration11 > Iteration10 then it retains Iteration11 and removes Iteration10. This then repeats for the remaining n iterations until all iterations have completed. Is this a fair characterization?
What I notice when running these iteration-dense optimizations is that my RAM space is fully consumed and after that threshold is hit a pagefiling mechanism takes over and RAM is incrementally "freed", albeit only a couple of gigs at a time, and there is activity on the system disk (SSD). This makes the remainder of the optimization extremely slow.
In the unique instance in which ALL optimization iterations are being requested via the "Keep Best # Results" parameter, wouldn't it make more sense for the Strategy Analyzer to dump the entirety of the 64GB to disk and start fresh? I'm sure there are probably numerous architectural reasons why this may not be possible but I'm simply curious to understand the mechanics better.
Thanks so much for the info.
Comment