Different Results From Walk-Forward Optimization With Same Parameters

[JakeOfSpades]

Hi there, I am running a Walk-Forward Optimization with the Sample MA Crossover optimizing the fast param 10-15, slow param 25-30 with a optimization period of 7 days and test period of 7 days. I am getting different results even with the same parameters, I hit run wait for results then hit run again and will get vastly different results. Why could this be? See below for more details.



Here is the first result





Then I ran it a second time with the exact same parameters, not changing anything. Those are vastly different results even though I changed nothing, why could this be?

[tgn55]

In my experience, Strategy Analyzer reporting is rather buggy. That said, a closer look at your results clearly indicates a big difference in Short trades ($21K vs $9K) which mostly accounts for the overall difference. You could look into that - but why are you getting that? Sorry... I have no clear answers given what you showed.

If you open each of the two optimization runs in the Log tab, I advise looking at the Trades view... that should show you exactly what changed. But why? If you really changed nothing in ANY setting, it MAY be something like a minuscule difference in floating point values ultimately triggering different trades, but that still begs the question... why would it be different.

Finally... a word of caution. Beware putting too much confidence in the optimization process... there be monsters there! If you want to understand why I say that... look closely at one of my previous posts on the subject here. Warning: some of the comments kinda miss the point I was trying to make, and go down rabbit holes. Bottom line is... the walk-forward process itself induces significant variation in results ... even when EVERYTHING else remains the same. Caveat Emptor...

Good luck...
T.

[NinjaTrader_Kate]

Hello JakeOfSpades,

Thank you for your post.

We've been able to reproduce this behavior on our end with the latest version of the platform. I'm reaching out to our development team to see if this is expected or not. I will let you know as soon as I hear back from them. We're tracking this issue under the number TS-2571.

Thanks in advance; I look forward to assisting you further.

[JakeOfSpades]

Hello JakeOfSpades,

Thank you for your post.

We've been able to reproduce this behavior on our end with the latest version of the platform. I'm reaching out to our development team to see if this is expected or not. I will let you know as soon as I hear back from them. We're tracking this issue under the number TS-2571.

Thanks in advance; I look forward to assisting you further.

Sounds good keep me posted.

[JakeOfSpades]

In my experience, Strategy Analyzer reporting is rather buggy. That said, a closer look at your results clearly indicates a big difference in Short trades ($21K vs $9K) which mostly accounts for the overall difference. You could look into that - but why are you getting that? Sorry... I have no clear answers given what you showed.

If you open each of the two optimization runs in the Log tab, I advise looking at the Trades view... that should show you exactly what changed. But why? If you really changed nothing in ANY setting, it MAY be something like a minuscule difference in floating point values ultimately triggering different trades, but that still begs the question... why would it be different.

Finally... a word of caution. Beware putting too much confidence in the optimization process... there be monsters there! If you want to understand why I say that... look closely at one of my previous posts on the subject here. Warning: some of the comments kinda miss the point I was trying to make, and go down rabbit holes. Bottom line is... the walk-forward process itself induces significant variation in results ... even when EVERYTHING else remains the same. Caveat Emptor...

Good luck...
T.

Could it be because my optimization period and test period are the same value (both 7 days). Anyways I'm pessimistic so I take it with a grain of salt.

[tgn55]

Not quite sure what you are asking... the results should NOT be different, good that Kate has raised this as an issue with the developers.

I have done a LOT of backtesting and digging into the SA reports (and posted many articles about it) That said, I don't consider myself an expert on walk-forward testing by any means... but, I did demonstrate some fairly significant "odd" results in my other post. Not sure how much of that you digested, so I'll summarise my findings (briefly) here... The simple fact of splitting up the entire test period into multiple, discrete chunks... without changing ANY parameters, can radically alter the P/L result. Period. Of course, in real life, the whole point of walk-forward stuff is to see if a strategy really performs reliably, with less reliance on hindsight... including tweaking parameters as the price action changes from period to period. Fine... Just be aware that there are things afoot that are somewhat arbitrary that can have a major impact on the final reported performance metric. Dang, this stuff can be hard! And the naive can easily be suckered into thinking things are much better than they really are.

I'm not saying you should not subject a strategy to walk-forward testing... just don't be over confident that if it shines... you have found the holy grail. A big chunk of the reported goodness MAY be just noise induced by the "let's carve the year up into monthly or weekly blocks" kind of process...

Look for consistency in trade results... and allow a good margin of error. If/when you find a strategy that has a smooth P/L curve, and a fairly decent Profit Factor (by which I mean one that STILL looks nice even after you hack away some percentage due to the aforementioned anomalies)... then, you will have something worth pursuing.

FWIW.. I find the Optimisation Graph in Ninja that displays the spread of ALL parameter combinations very illuminating... I recommend you explore it. You get to see how often things totally suck... as well as how often you get winning combos. Don't ignore this...

Hope that helps,
T.

[JakeOfSpades]

So far from my findings, if the optimization period and test period are the same value (for example 7 days optimization period, 7 days test period) I get the random result issue. Some strategies, the pnl deviates 10-20% and some up to 50%.

Now, if I do a super long optimization period say 365 days, and a test period of 180 days the results are always the same which is exactly what I want.

Thus, the bug happens when the optimization period and test period are the same length.

[NinjaTrader_Kate]

Hello JakeOfSpades,

Thank you for your reply.

I've heard back from our development team and this is expected.

Here's the rundown on what's happening:

This is not a "bug," per se, but rather a result of the way that Walk-Forward Optimizations use different threads to run optimization iterations. When multiple optimizations run concurrently on different threads, each one can finish at a different time, and will then add its performance results to the list, which is then ordered properly. This should only occur when different optimizations in the test (different parameter combinations) yield identical performance results. When iteration A with Performance X finishes after iteration B, which also has Performance X, then users can see the related lines in the list of iterations show up in an unexpected order, and the order could even change each time the Walk-Forward Optimization is run. It should also be a fairly rare case that two or more optimization iterations in a Walk-Forward test would yield identical performance results, but when you are iterating over smaller time periods this becomes more likely.

Please let us know if we may be of further assistance to you.