IS+OoS + IS+OoS + IS+OoS test also ranking the OoS period. Not a Walk Forward

[futurenow]

Hello

I want to perform a kind of test in a strategy that could be seen that is “similar” to a Walk Forward Optimization but in a way the Optimization periods run in different and separate time dates than the Test periods, because in the Walk Forward test an Optimization period (except the first one) runs in range of dates that a previous Test period also run, and one of the main purposes here is that the periods be completely separated from each other.

The idea would be for example to take a year and specify a whole test in the next way:

Range of dates	Kind of test
Jan-Feb-Mar	Optimization period
Apr	Backtest period
May-Jun-Jul	Optimization period
Aug	Backtest period
Sep-Oct-Nov	Optimization period
Dec	Backtest period

Ok, into a strategy code I have the idea about to handle that an Optimization or Backtest be performed only for specific months of the year, but what I don’t know how to handle is to specify that the Optimizer (or in general the Strategy Analyzer) performs an Optimization for some specific months AND at the same time it also performs the Backtest with the optimization settings for a specific unseen range of dates, doing this iteratively as shown at the bottom part of the attached picture, and doing this ranking the results ordered from the best to the worst result with the important part of taking into consideration both, the Optimization periods sum result + the Test/Backtest periods sum result, in the way that when the Strategy Analyzer have 2 whole test results like for example:

- First whole test result:
Optimization periods sum result with a Profit Factor = 1.55, and Backest periods sum result with a Profit Factor = 0.80

- Second whole test result:
Optimization periods sum result with a Profit Factor = 1.30, and Backest periods sum result with a Profit Factor = 1.20

Then, in this example the second whole test would be raked as better result than the first whole test, because in the first whole test the Backest part had a poor performance comparing it with the Optimization part, and in the second whole test although the Optimization part had a Profit Factor lower than 1.55, it had a Backtest part with a similar and acceptable Profit Factor, considering that the Profit Factor in the Backtest part could be slightly worst or greater than in the Optimization part, but the key point is that both part of the whole test must have a good or acceptable result based on the Optimization Fitness with which we are working, in this case the Profit Factor.


I attached a picture to have a visual idea about what I called “Continuously segmented Optimization period + Test period”, but in general terms I think the process would be about how to handle the next steps in the strategy code and/or the Strategy Analyzer:

- Step 1:
First, perform an Optimization only for the given periods, from X date to Y date, also from Z date to W date,...

- Step 2:
Then, perform a Backtest with the parameters of the optimization of the step 1 for the unseen given periods, from A date to B date, also from C date to D date,...

- Step 3:
Repeat the step 1 and 2 for all the parameters combination just like any other standard Optimization.

- Step 4:
Finally, rank/order the results like any other standard Optimization keeping for example the 10 whole test best results, not only considering the Optimization results but considering at the same time both, the Optimization part results + the Backtest part results.


Thank you in advance for the help

[NinjaTrader_BrandonH]

Hello futurenow,

Thanks for your post.

Changing how NinjaTrader is coded internally is not supported so we cannot provide further guidance on this topic.

That said, we are tracking this in an internal feature request ticket and I have added your vote. This request is being tracked under the number SFT-5652.

As with all feature requests, interest is tracked before implementation is considered, so we cannot offer an ETA or promise of fulfillment. If implemented, it will be noted in the Release Notes page of the Help Guide.

Release Notes — https://ninjatrader.com/support/help...ease_notes.htm

Please let us know if we may be of further assistance to you.

[futurenow]

Changing how NinjaTrader is coded internally is not supported so we cannot provide further guidance on this topic.

I see your answer is more focused about what the Strategy Analyzer actually can do, but I think it could be alternatives, and maybe this could be worked with a custom Optimizer staring from something like in the next other post:

https://ninjatrader.com/support/forum/forum/ninjatrader-8/strategy-development/1204766-optimizer-custom?p=1205019#post1205019

Maybe this could be achieved by working with the NinjaScript Output window, showing the desired information and results in that window, that is a way that doesn’t need any change of NinjaTrader.

What about to take this way as alternative?


Other way I was thinking about to handle this could be inserting in a strategy code the way to calculate the Profit Factor for the desired Optimization periods, and separately also calculate the Profit Factor for the Backtest periods, to not to mix both set of periods, and if the Strategy Analyzer can’t show that separated information about the Profit Factor for both set of periods, then it could be shown in the NinjaScript Output window, showing the ranked results in that Window.

Could you please provide a starting point, “frame”, or Sample about how to work the idea I just described or an improved way you or your development colleagues could provide.

Thank you

[NinjaTrader_BrandonH]

Hello futurenow,

Thanks for your note.

It would not be possible to create your own custom Optimizer that works like the Walk-Forward optimization that uses a sample out for the next tests and so on.

A custom Optimization script would change how a single optimization works and ranks the results.

We do not have any supported workarounds you could use to accomplish this.

Please let me know if I may assist further.