Completely different results running the same walk-forward optimization

[haakonflaar]

I get (sometimes extremely) different results running the same walk-forward optimization on the same strategy with the same time period, parameters, optimization objective etc etc - everything is the same. Every time I open up a new strategy analyzer and punch in the exact identical values I get a different result. Why is that? How can it be fixed?

I have seen that this issue has been addressed in previous posts, but I have not seen any clearcut answer as to why it is happening, or a fix for it.

[NinjaTrader_Jesse]

Hello haakonflaar,

If you located another post where this has been addressed please add a link to that so we can see if that post is relevant to your question.

In regard to why your tests were different that is not something I would be able to answer without knowing that the tests had performed identically. If you are getting very different results between tests that could be related to the logic your strategy is using. In most situations when you see something happen differently and produce different results you would need to add prints into the strategy so you can visibly see how your conditions evaluated. That lets you gather information for each test which you can use to then compare those results to know what specifically happened differently.

You can find information in the following post that goes over how to debug the results between two tests, that can be from the same tool as well. The steps in that post go over how to add debugging and compare the tests. https://ninjatrader.com/support/foru...ay-performance

[QuantKey_Bruce]

Genetic or brute force? If you are using genetic, then definitely the results will be different every time because that is a stochastic method.

[haakonflaar]

NinjaTrader_Jesse Here are some posts with the same issue:

Different Results From Walk-Forward Optimization With Same Parameters - NinjaTrader Support Forum
Different results for walkforward optimization for each run - NinjaTrader Support Forum
Walk Forward - same parameters, different results? - NinjaTrader Support Forum
Walk forward gives different results everytime I run it - NinjaTrader Support Forum
Same Walk Forward Optimization, Different Results - NinjaTrader Support Forum

There has been some answer saying it's due to threading when running optimizations concurrently. I don't quite get this and definitely not why it would cause different results on identical runs?

[NinjaTrader_Jesse]

Hello haakonflaar,

In the first post you linked that would likely be the best description of reasons for expected differences. That information was provided by the development team and in that scenario that would be expected to get different results based on how threading works. It was also mentioned in developments quote that It should also be a fairly rare case that two or more optimization iterations in a Walk-Forward test would yield identical performance results which is good to keep in mind. When using walk forward testing there is a general expectation of differences between tests due to how tha type of optimization works.

If you are doing a backtest instead of an optimization/walk forward that would be where the general expectation would be that the test should be identical assuming that all parameters were the same, the strategies logic executed identically and identical data was used.

[haakonflaar]

Hello NinjaTrader_Jesse,

I don't understand why you are expected to get different results running identical walk-forward optimization runs. The data being fed to the strategy should be the same, the way the strategy behave should be the same, so how come results are different? At least for the total net profit. I can understand that the max drawdown will differ in two trades if, due to threading, they are captured in different orders, but that should not have an effect on the final total net profit. In either case, the order of trades should be ordered correctly before giving the final result. I am confused here...

[NinjaTrader_Jesse]

Hello haakonflaar,

The description in post # 8 of the forum thread you linked explains why that difference would be expected based on developments assessment in that use case. If you are doing a similar test that would be a way to explain differences in that situation.

To better understand why the results are different in your specific tests you can dig into the executions that were produced to see how each test had performed and where they deviate.You can also use prints in your strategy to compare how the logic was run in each of the optimizations to ensure the tests were in fact identical. Please see the link in post # 2 for an example of how you can debug your strategy.