Walk Forward - same parameters, different results?

[puravida]

Hello Bertrand,

thanks for your response.

I already tried doing the WF in disconnected state, but the result is still the same. I uploaded two sample screenshots where you can see the control center is disconnected. The results vary again which in my opinion should not be the case.

@cachevery
Could you please stop hijacking my thread?

[NinjaTrader_Brandon]

Hello cachevery,

When you do a test in Market Replay to get the same matching results each time you will want to apply your strategy and run it with the replay controller set to the exact same time. I would recommend playing it at slower speeds. Ensure also that your chart has the same settings for each test.
Please note that when using Market Replay all historical data from the chart is backfilled from your Historical Data Manager, not past replay data. If your historical data was to be changed between your tests this can modify your results.
When backtesting in the Strategy Analyzer, this is where you would want to ensure not to be connected so that new data is not reloaded when backtesting.

[NinjaTrader_Brandon]

Hello puravida,

Please send an email to support[at]ninjatrader[dot]com with a link to this forum topic. I would like to set up a remote call to look into this further on your end.

[cachevery]

Hello cachevery,

When you do a test in Market Replay to get the same matching results each time you will want to run apply your strategy and run it with the replay controller set to the exact same time. I would recommend playing it at slower speeds. Ensure also that your chart has the same settings for each test.
Please note that when using Market Replay all historical data from the chart is backfilled from your Historical Data Manager, not past replay data. If your historical data was to be changed between your tests this can modify your results.
When backtesting in the Strategy Analyzer, this is where you would want to ensure not to be connected so that new data is not reloaded when backtesting.

Hi Brandon,what do you mean by i`d want to run apply my strategy,i don`t get it.

I always start the replay from the same place,i also use the same charts,same parameters,same replay data,i never update anything.I did about 20 iterations with al the same-same-same-same-same,and got 20 different redsults.I`m not interested in Strategy Analyzer,i only interested in the Market Replay.So how do i get it consistently?

Thank you

[NinjaTrader_Brandon]

Hello cachevery,

Thanks for your reply.

I have edited my reply to just say apply.

This original post
What speed do you run the replay during these tests?
What time From and To date?
Do you see the same if you use use the SampleMACrossover?

This original post from puravida is about the Strategy Wizard, that is why information about the Strategy Wizard is included.

[NinjaTrader_Brandon]

Hello puravida,

Thanks for your time on the phone today to gather additional information to look into this further.

I have been able to reproduce this on my end.
What is happening is we are optimizing on Max. Profit Factor.
If you review the Optimizer tab we see that there are multiple results with matching values under Profit Factor.
These different Profit Factors however have different Total Net Profits/Gross Profit results.
This then leaves the Walk Forward Optimization to base it's results depending just on the order of the results from the optimization and this order can sometimes vary.

[puravida]

Hi Brandon,

thanks for your support so far.

Neverless i see the same behaviour when i change from "max. profit factor" to "%-profitable" or even "max. net profit".

I have attached two sample screenshots with different results optimizing on max. net profit. I raised the "min. bars required" and used a value for the walk that can be divided by 5... why is it happening here too?

[NinjaTrader_PatrickH]

Hello puravida,

Thank you for your response.

Under the Optimizer tab are you seeing multiple results for the optimize on setting?

[cachevery]

Hello cachevery,

This original post

The responses in line

What speed do you run the replay during these tests?

I run it at the max speed - x500

What time From and To date?

Always the same

Do you see the same if you use use the SampleMACrossover?

Yes,with the SampleMACrossover the results are the same.

[puravida]

Hi PatrickH,

i have attached a screenshot of the optimizer tab, all results are "0" on performance... not sure what this means.

[NinjaTrader_Brandon]

Hello puravida,

Thanks for your reply.

By clicking on the column name in which you are optimizing on you can sort the value from greatest to least. Having multiple matches for the best value will confirm our findings.

cachevery, for replay I would recommend playing it at a slower speed. It is possible for more complicated strategies to get different results when playing at the fastest speed as the PC may process the orders at slightly different times since it would be taking lots of PC resources to play that fast.

[puravida]

Hi Brandon,

not sure i am clicking correctly, so here are my steps:
1) click "Run Walk Forward Optimization"
2) click Optimizer Tab
3) sort / scroll through available listing, clicking on some items

Every value is "0" in the tab as i posted the screenshot in the last posting. If i sort it, there is still nothing... so yes, all values are the same which is Performance of 0, but since the wf is giving results in "Instruments" and "Walk forward" tabs i do not understand what is actually displayed here or what i could read out of it. It's listing over 50 trades in the "Walk forward" tab, so actually there should be something other than 0 in the "Optimizer" tab, right? Or am i misunderstanding the concept?

[NinjaTrader_Brandon]

Hello puravida,

Thanks for your reply.

From the screen shot you provided, what 'Optimize On..' setting did you use?

[puravida]

Hello Brandon,

Thanks for your support. I used "max. net profit" in that case, but it seems to be the same whatever i choose to optimize on (%profitable, max. avg profit,...) always no visible result in "Optimizer" while there are trades in "Instruments" and "Walk forward" tab and still different results without changing parameters.

[NinjaTrader_Brandon]

Hello puravida,

Thanks for your reply.

From the screen shot you provided it looks like it was being sorted by Profit Factor.
Either way however if you were to sort by the 'Optimize On..' setting I'm sure you will see multiple results with the same value which will verify why the results change at times when running the walk forward multiple times.