Backtest vs Walk Forward results

[lookOutBelow]

I'm confused why I'm getting different results when I back test over a given set of dates and then do a walk forward over those same dates (using the same static values and no range to optimize through).

Shouldn't I get identical results? Since there are no ranges of values to optimize through?



p.s. when I do the walk forward, I'm changing the start/stop dates so that the actual walk forward result dates line up with the backtest dates.

[NinjaTrader_JoshP]

No, these will not be the same. A backtest starts when the "Min. bars required" parameter is met. This means the first 20 or so bars are ignored in the date range you select. A walk-forward on the other hand has already satisfied the "Min. bars required" parameter during the test period and when it gets to the optimization period it does not need to ignore those first 20 bars. This can result in different test results if your strategy trades during those times.

[lookOutBelow]

Thanks for the response. But should they vary by a large factor? I've uploaded some pictures to illustrate the difference.

Please note that the Date Ranges are different on the Walk Forward and Backtest so that the actual result (date) ranges are the same...

Notice the $10,000+ difference with the Sample MA crossover.

Thanks

[NinjaTrader_JoshP]

lookOutBelow,

Your tests are not comparable. Your backtest is of 3 months while your walk forwards are of 1 week periods. When you have a backtest of 3 months that means throughout the whole test positions carry over between each day and such for the whole of 3 months. When you have a walk-forward of 1 week periods it means you end each test at the end of that 1 week. All positions close out at the end of the week. These are not comparable tests. You would need to compare equal length tests against each other if you wanted to compare.

[lookOutBelow]

I must be dense, because I'm not following you. Yes, the Walk Forwards are 1-week periods, but when you add up all of those 1-week periods, you get 13 weeks and $13480.

As opposed to the backtest that is going over 13 weeks and $3680.

Can you clarify further? Because I'm still not understanding your explanation.


thanks

[NinjaTrader_JoshP]

lookOutBelow,

When you use a walk-forward period of 1 week, those 1 week periods are all self enclosed periods. Meaning all performance is chopped off and ended at the end of every single week. When you run a 3 month backtest, the results are not chopped off and ended at each week. If the position goes over the week, it goes over. That is the difference. Hopefully that makes sense.

[lookOutBelow]

lookOutBelow,

When you use a walk-forward period of 1 week, those 1 week periods are all self enclosed periods. Meaning all performance is chopped off and ended at the end of every single week. When you run a 3 month backtest, the results are not chopped off and ended at each week. If the position goes over the week, it goes over. That is the difference. Hopefully that makes sense.

So, what does that mean if I have a strategy that closes every position every day at 3pm? There are not any positions that would carry over from one week to the next..

[NinjaTrader_JoshP]

lookOutBelow,

Basically the comparison you are running is not an apples to apples comparison. To find the differences I suggest you just open up the test and look into the Trades tab and see what exactly is different. There can be any number of reasons and you will need to evaluate it on a case by case basis.

[lookOutBelow]

lookOutBelow,

Basically the comparison you are running is not an apples to apples comparison. To find the differences I suggest you just open up the test and look into the Trades tab and see what exactly is different. There can be any number of reasons and you will need to evaluate it on a case by case basis.

The biggest glaring difference I see right off the bat is that during the backtest, it doesn't start trading on the day I tell it to. I set it for 6/29 and it started on 6/25.

Why?

[NinjaTrader_JoshP]

Depends on your session template, PC time zone, instrument tested on. Start/end dates can "bleed" to include prior session under certain relationships.

[bertochi]

lookOutBelow,

Your tests are not comparable. Your backtest is of 3 months while your walk forwards are of 1 week periods. When you have a backtest of 3 months that means throughout the whole test positions carry over between each day and such for the whole of 3 months. When you have a walk-forward of 1 week periods it means you end each test at the end of that 1 week. All positions close out at the end of the week. These are not comparable tests. You would need to compare equal length tests against each other if you wanted to compare.

--------------------------
This is an old post but I would like to submit a request to modify the way the "walk forward optimization" is being performed at the moment. Josh said quite rightly that "When you have a walk-forward of 1 week periods it means you end each test at the end of that 1 week. All positions close out at the end of the week." and there is actually no need for that, no need to close the position "at the end of each test period". Say for example, you are running a 1 year walk forward optimization with parameters (optimize 30 days, test 10 days), we (I mean, "I") would like to see the walk forward for the whole year, but if the positions are being closed at the end of each test period, 10 days in this case, and then waiting for a trigger to open a position on the next test period, it defeats the whole idea of the walk forward optimization, at least in my opinion. Anybody has any thoughts on this? Thanks