Walk Forward Optmization and weekends

[Strategery]

Hi, I am new to WF optimization.
Since there is no way to batch optimizations in NT I am trying to use WF to get daily optimization levels over a time period instead of running them one at a time with the standard optimizer.
What I am looking to do in the WF is pretty simple: run an optimization for an entire trading day and then test the optimal parameters on the following trading day.
I am running it on MNQ and using the default instrument setting trading hours with an optimization period of one and a test period of one.
The WF optimization numbers look ok until it hits a weekend. I would expect the optimization to run on Friday's session data (Thurs open at 5:00 PM till Friday close at 4:00 PM) and apply those levels to the following trading day (Sunday open at 5:00 PM till Monday close at 4:00 PM).
Instead, it applies the Friday levels to Saturday, which generates no trades obviously. Then it creates seemingly random Saturday levels that are applied to Sunday creating no trades. Then it creates optimized levels for Sunday and applies them to Monday which gives me bad results. It gets back on track for Tuesday's trading since Monday was an actual trading day.
I thought about using RTH or ETH hours but I want both in my optimization. It is my understanding that the default instrument session settings for a CME index product are both of the sessions combined.
I must be doing something wrong. This can't be normal behavior.
The testing/result data is really not that important to me as I only need the optimization numbers for each day (though I may have some use for it later on)
All I really need is the ability to run an optimization and log the daily results for a date range. I do not need the optimal results for the entire range, I need to reset and log the results at the end of each trading day and then start a fresh run for the following day.

Any help with the WF weekend issue I am having or if there is a way to batch daily optimization runs over a series of days would be greatly appreciated.

Thanks,
Mike

[NinjaTrader_BrandonH]

Hello Mike,

Thanks for your post.

Please try reproducing the behavior using the SampleMACrossOver strategy.

Then send me a screenshot of the exact settings used to reproduce the behavior with the SampleMACrossOver so I may accurately assist you.

• To send a screenshot with Windows 10 or newer I would recommend using the Windows Snipping Tool.
  • Click here for instructions
• Alternatively to send a screenshot press Alt + PRINT SCREEN to take a screenshot of the selected window. Then go to Start--> Accessories--> Paint, and press CTRL + V to paste the image. Lastly, save it as a jpeg file and send the file as an attachment.
  • Click here for detailed instruction

[Strategery]

Here is a screenshot of the simple ma strategy. On the 10th which was a Friday the optimized parameters were 20/25.. Then the optimization runs on Saturday and Sunday generating 2/25 and 12/25.

[NinjaTrader_BrandonH]

Hello Strategery,

Thanks for that information.

Yes, the Walk Forward optimization will process dates on Saturday and Sunday when the market is closed. There are no options currently available in the Strategy Analyzer for skipping weekend dates when the market is closed. We are tracking interest in a feature request for this and I have added your vote. This request is being tracked under the number SFT-5949.

As with all feature requests, interest is tracked before implementation is considered, so we cannot offer an ETA or promise of fulfillment. If implemented, it will be noted on the Release Notes page of the Help Guide.

Release Notes — https://ninjatrader.com/support/help...ease_notes.htm

[Strategery]

I must have explained myself wrong, let me try again.

I am currently using a one-day optimization period and a one-day test period.
With those settings, it seems that the Monday test will always be invalid since the test is using levels from Sunday that seem to be generated randomly since it is not a valid trading day.
Using that logic If I change the optimization period to 3 doesn't that mean that the 3-day optimization will include the bad parameters that were generated over the weekend instead of the last 3 trading days? So a Monday test will be executed based on an optimization run using data from Friday, Saturday, and Sunday. And a Tuesday test will use levels from Saturday, Sunday, and Monday?
Hopefully, that makes more sense.

Thanks,
Mike

[NinjaTrader_BrandonH]

Hello Strategery,

Thanks for your notes.

Since the trading hours for Monday start on Sunday evening, you may see trades on Sunday and Monday when reviewing optimization results for a date that falls on a Monday.

When running a walk-forward optimization in the Strategy Analyzer, it is optimizing the strategy based on the optimization fitness.

The 'Optimize on' property sets the optimization fitness to base the optimization results on. The 'Optimizer' property sets the optimization algorithm that is used. NinjaTrader comes with "Default" and "Genetic" optimizer algorithms. The 'Optimization period' property sets the number of days used for the "in sample" optimization data set. The 'Test period' property sets the number of days used for the "out of sample" real backtest using the optimized input values generated from the "in sample" period.

From the Start date to the End date the walk forward optimization will do a standard optimization on the number of days set for parameter "Optimization period (days)". This is known as the "In Sample" test period. After the optimization period NinjaTrader will use the best parameter combination found and test that forward on non-optimized data that has not been seen yet for the number of days set for parameter "Test period (days)". This is known as the "Out of sample" test period. Please see the graph below for a better understanding of how the walk forward results are found.

Understanding Walk Forward Test Results

Walk Forward Optimization:

https://ninjatrader.com/support/helpGuides/nt8/walk_forward_optimize_a_strate.htm

[Strategery]

Thanks, Brandon

I do expect potential trades from Sunday night on a Monday test since the session starts Sunday at 1700 and ends Monday at 1400. I believe I have a pretty good grasp on the concept of how walk-forward optimizations work now but am still confused about the results I am seeing.
As mentioned, I am using the standard session defaults for MNQ which include RTH and ETH. I also created a custom session that included both sessions (5:00PM to 4:00PM Sunday through Friday) and got the same results.

I don't seem to be explaining myself well in the previous messages so I am attaching another screenshot.
In it, you will see a walk-forward optimization using the Simple MA with a one-period optimization and a one-period test.
I added some notes in the log to highlight my confusion.
WF Test A on Friday 6/10/22 looks good. I confirmed below in the log by doing a standard optimization A1 and backtest A2. Levels and P&L match up.
There are then two entries in the WF test for trading sessions that do not exist B1 and B2.
The optimizer used the 29/50 levels from Thur/Fri and applied them to Saturday B1 which obviously generated no trades, It then generated levels of 5/50 with no data from Fri/Sat and tested them on Sunday B2 (no trades). New levels 7/50 were created from no data on Sat/Sun and tested on Sun/Mon which resulted in a net profit of -638.50 in the results.
The levels that should have been tested on the Sun/Mon session are those from the previous session (29/50 from Thur/Fri) which result in a net profit of +93.50 B2
Once we get to Tuesday the test is correct since it is using levels from the correct previous session.

I tried using > one-day testing period and the levels generated do not seem accurate if the lookback touches a weekend. On a Monday with a 3-day look back, the optimizer generates levels from only one trading session (Friday) although one is expecting optimization over 3 sessions. It seems to break down like this:
1 day optimization days with correct levels to test= TUE-FRI
2 day optimization days with correct levels to test= WED-FRI
3 day optimization days with correct levels to test= THUR-FRI
4 day optimization days with correct levels to test= FRI
> 5 days will be bouncing back and forth between generating levels using 3 or 4 trading sessions.

Unless I am missing something, the wf optimizer seems to be ignoring the days of the week defined in the trading session, which produces very sporadic/inaccurate results. If a user is testing a strategy based on parameters optimized from the previous 3 trading sessions, they do not expect Monday to use data from only Friday's session, Tuesday to only use data from Monday, Wednesday to use data from Monday and Tuesday, and the correct amount of sample size on Thursday and Friday.
Hopefully, this makes a bit more sense.
For all I know, I may not understand what the tool is supposed to be used for but in my case, having my optimization sample size be 33%, 66%, or 100% of what I expect it to be depending on the day of the week and testing on days with levels that were generated with non-existant sessions does not compute.

By the way, both of your links led to 404 errors. The pages must have ben moved or something.

Thanks,
Mike

[NinjaTrader_BrandonH]

Hello Strategery,

Thanks for your notes.

When running a Walk Forward optimization, Saturday and Sunday will be optimized if those days are included in the optimization start date and end date tiemframe. Saturday will be optimized from Friday evening to Saturday. Sunday will be optimized from Saturday evening to Sunday evening. and Monday will be optimized from Sunday evening to Monday evening.

Since there is no market data from Friday 5:00PM to Saturday 4:00PM, the results will be 0 trades calculated. Since there is no market data from Saturday 5:00PM to Sunday 4:00PM, the results will be 0 trades calculated.

Since there is market data from Sunday 5:00PM to Monday 4:00PM, you would look at the Monday results which will show the trades calculated from Sunday at 5:00PM to Monday at 4:00PM.

There are currently no options available to skip the Friday 5:00PM to Sunday 4:00PM dates when the market is closed when running an optimization. Interest in this is being tracked under the feature request mentioned in post # 4.

In regard to the help guide page links leading to a 404 Error page, this has been reported to the NinjaTrader IT team and they are looking into this matter further to provide a resolution.

[Strategery]

Here is how I understand things, correct me if I am wrong:

Optimized parameters used in the test period are not optimized using a fixed number of trading sessions.
Optimization is done on a variable amount of trading sessions.
The number of trading sessions used in the optimization = optimization period set by the user - the number of weekend/holiday days that occur in that period.

Fundamental problems this causes:
Depending on the optimization settings and the number of days without trading sessions the tests are performed using parameters that can vary from 100% of the optimization period to less than half of that amount (a 9-day optimization period test on Monday with a holiday on the previous Wednesday is only optimized on 4 trading sessions instead of 9).
This can cause a huge amount of variance in our results, especially if the optimizations require a somewhat equal amount of data to be analyzed for each optimization.

When using optimization periods that are equal to or less than the number of days without trading sessions, the parameters generated to test the next trading session are completely wrong as they were created using no market data. A minimum of 20% of the results in the test are garbage if we are only talking about Mondays being tested on data from Sunday or Saturday/Sunday.

The whole idea of walk-forward optimization is to generate parameters to optimize a strategy on a defined number of market data sets (days in the case of NT) and test those parameters on a defined number of days of future market data while rolling through a fixed date range.

The optimizations are not performed on the same amount of market data sets. The amount of data sets analyzed varies as the optimization rolls through the start/end date depending on how many days in the optimization period are actual trading days. In my examples above, the optimizer can reduce the number of market data sets to more than 50%.
The same goes for testing. With every run, the tester applies parameters to trading on Saturday and Sunday generating 0 P&L rows in the log, throwing off strategy analysis metrics, and dropping the good parameters that should have been kept for testing in the next trading session.

You said this is being tracked as a possible feature request.
Unless I am wrong in my rambling above, I can't see how this isn't considered a fundamental issue with the module.

Grab the session details and holiday list from the instrument's trading hours template.
Match dates from the user's start/end range with dates that have valid trading sessions.
Remove holidays.
Use the EOD date as the test date.
Run the walk forward on the date range with valid trading sessions.


Thanks,
Mike

[Strategery]

I just reviewed the Ninjatrader online manual and the first sentence describing how walk-forward optimization is done reads:
"Walk Forward optimization is the process by which you optimize strategy input parameters on a historical segment of market data"
The application requires an integer to define how many segments of market data we want to use for the test.
When the WF is run it uses segments of market data if they are available, makes up seemingly random parameters, and uses those for days where there is no segment of market data. It then tests on days with no segment of market data, and/or applies parameters generated with an incorrect proper amount of historical data segments input into the optimization period.
Again, this seems more like a bug than a feature request, especially since the manual refers to "segments of market data".
I have been manually doing "walk forward" testing for about a year. Most of what I can do can't be replicated in the application due to how I generate and test my parameters.
One simple test I do can be replicated in the version that is included with NT. The results were so far off I dug a little deeper and that's why I started this thread.
In my eyes, any test run with the tool is not accurate to real life unless a trader is throwing away optimized data and generating $0 P/L days on weekends and making up parameters to test with after a non-trading session day or days.

[NinjaTrader_BrandonH]

Hello Strategery,

Thanks for your notes.

To make sure we are on the same page, please clearly demonstrate what exactly it is you are talking about by using the SampleMACrossover strategy in a very simple test.

Also, provide the exact steps and settings you use for your simple test so that I may try reproducing in on my end and investigating further.

[Strategery]

It's all above. Two screenshots and absurdly verbose detail.

I'll type out a scenario again. You can use any strategy you want. I trade CME Equity Index futures with default instrument trading hours, so it should produce the same results on any of those products.
Create a walk-forward optimization/test that optimizes parameters over two trading sessions and tests those optimized parameters on the following trading session (2,1).
Use last Wednesday and the end date and set the start date to the Wednesday before that. 5/24-5/31

Here is what you will get:

WED 5/24
No trades and no entries in the log since two trading sessions are needed for optimization. We currently have zero sessions.

THUR 5/25
No trades and no entries in the log since two trading sessions are needed for optimization. We currently only have data from one session: 2/24

FRI 5/26
A test is performed since we now have two sessions of data to optimize with. Optimizations from 5/24 and 5/25 can be replicated with a standard optimization and P&L can be verified using a standard backtest for 5/26 with those parameters.

SAT 5/27
There is no trading session. A test is done anyway using optimizations from 5/25 (valid session of 23 hours of data) and 5/26 (valid session of 23 hours of data) and logged as a 0 P/L day.

SUN 5/28
There is no trading session. A test is done anyway using optimizations from 5/26 (valid session of 23 hours of data) and 5/27 (no session 0 data) and logged as a 0 P/L day.

MON 5/29
The previous two trading sessions were 5/25 and 5/26. Optimizations from these two sessions are what we want to use today.
Instead, the test is performed with parameters that were generated with data from 5/27 and 5/28. I do not know how any parameters are being created at all since there are no sessions/no data for those days. The optimized parameters do not match what we expected from 5/25 and 5/26 so those days are getting dropped since it is incorrectly assuming trading sessions on 5/27 and 5/28.

My only hunch on how the optimizer is generating parameters with no data is that since the Monday session starts on Sunday evening, the optimizer is using data from Sunday 5:00PM-midnight to generate the parameters. That is the only data available on the weekend. So instead of optimizing on the two previous sessions' 46 hours of data, we are using 7 hours of data.
Now that I think about it as I type this..
If the test on Monday is using optimized parameters from Sunday evening, doesn't that means for the first five hours of the Monday session (which starts Sunday at 5:00 PM), we are testing with parameters that were optimized using data from the same session we are testing on?
I do not know if this is the case, but either way, we wanted two trading sessions (46 hours worth of data) and we are either getting random parameters created from no trading sessions, or we are testing using parameters that were optimized on the first five hours in the future. A crystal ball!

TUE 5/30
Paramaters are the result of one trading session (Monday 5/29) since there was no trading session on Sunday
We wanted top optimize on Friday and Monday. Our test is performed using parameters that were optimized on 50% of the market data we expected.


WED 5/31
Since we have two valid trading sessions directly behind us and our test period is two, we have valid optimized parameters.
The test is valid.

This is pretty much the same thing I explained and provided screenshots of above, besides the new hypothesis on how optimized parameters are being created with no data to analyze.
Let me know if you need any clarification on anything,

Thanks,
Mike

[NinjaTrader_BrandonH]

Hello Mike,

Thanks for your notes.

The parameters are found based on the optimization fitness that you choose.

If there were no trades for one of the weekend days, that wouldn't affect the results since that is not going to be ranked like a normal result.

It will include days that it processed because that is how the Walk Forward optimization works. When there are days with no trades the optimization has no max profit factor to rank those days so it won't affect the next set of parameters it calculates.

You could view the video in the help guide which goes over how the Walk Forward Optimization works. The test will go through all the days you select, the trades are only able to be placed when there is a trading session and market data

Walk Forward Optimization: https://ninjatrader.com/support/help...d+optimization

[Strategery]

I wish someone there would look at the examples I have been providing and address my comments/questions.
Asking me for examples and then copying and pasting what is in the manual hurts my soul.

Let's zoom in on one issue:
Attached is a screenshot of a walk-forward optimization.
Optimize on two days, test on one
Start Date= Saturday 5/21/23 End Date= Wednesday 5/25/23
The Monday 5/23 test is done using parameters 9/50.
Which "historical segments of market data" were used to generate those parameters?

[NinjaTrader_BrandonH]

Hello Strategery,

Thanks for your notes.

I have reached out to the Development team to request information about your inquiry.

This forum thread will be updated with more information as soon as I hear back from the Development team.

Thank you for your patience.