Capture Walk Forward Information

[Steve L]

I want to programmatically capture the results of individual walk forward runs so I can calculate some additional analytics.

What's the best way to hook into this?

Are there properties that hold related information like the run number or date ranges currently being processed?

Thanks.

[NinjaTrader_ChelseaB]

Hello Steve L,

While the walk forward results are saved in the Strategy Analyzer Logs, unfortunately there isn't a supported way to access the results programmatically.

I am happy to submit a feature request for the development team to consider this if you would like.

[Steve L]

I'm wondering if a custom optimizer might get me there.

Does an optimizer stay alive for the life of the walk forward test or is it only for each step in the walk forward (each optimization).

Is there a way to know if I'm on the last optimization of the walkforward so that I could save some results by writing to a file?

Can you help me understand the following terms from this page of the help guide?

https://ninjatrader.com/support/helpGuides/nt8/en-us/onoptimize.htm?zoom_highlightsub=onoptimize

* optimization run
* iteration

I'm thinking that one optimization run is the set of all backtests for every combination of parameters (assuming just a standard optimization).

And that an iteration is exactly one backtest with one set of parameter values.

Thanks.

[NinjaTrader_ChelseaB]

Hello Steve L,

An optimization is a series of backtests. The collection holding the iterations of backtests is not exposed in NinjaScript.

An iteration is a specific backtest.

An optimization run is the start of a group of backtests. Backtest iterations are not run synchronously, they are run asynchronously, in groups, to utilize as much of the CPU as possible and perform the test as quick as possible. (After a run, that instance may or may not be re-used depending on the IsInstantiatedOnEachOptimizationIteration setting)


It might be possible to use NumberOfIterations and count the iterations..

Below is a link to a forum thread where something similar was discussed you may find helpful.
https://ninjatrader.com/support/foru...61#post1038261

[Steve L]

I was able to get most of what I needed using a custom optimizer.

See the end for my feature request...

For others who land on this page, I copied the default optimizer and modified the OnOptimize method as follows:

        protected override void OnOptimize()
        {
            Iterate(0);

            WaitForIterationsCompleted();
            if (!IsAborted)
            {
                Print("--- WalkForward Run ---");
                int rows = Math.Min((int)NumberOfIterations, Results.Length);
                for (int i = 0; i<rows; i++)
                {
                    SystemPerformance sp = Results[i];

                    DateTime dtFirstEntry = sp.AllTrades.Count > 0
                        ? sp.AllTrades.Min(t => t.Entry.Time)
                        : DateTime.MinValue;
                    DateTime dtLastExit = sp.AllTrades.Count > 0
                        ? sp.AllTrades.Max(t => t.Exit.Time)
                        : DateTime.MinValue;

                    string parmsText = "";
                    sp.ParameterValues.ToList().ForEach(o => parmsText += o.ToString() + ",");
                    parmsText = parmsText.TrimEnd(",");

                    string text = String.Format("{0} to {1}, NetProfit={2}, Parameters={3}",
                        dtFirstEntry.ToString("MM/dd/yyyy"),
                        dtLastExit.ToString("MM/dd/yyyy"),
                        sp.AllTrades.TradesPerformance.NetProfit.ToString("F2"),
                        parmsText);
                    Print(text);
                }
            }

        }

This produces this output when running a walk forward test using the built in MA Crossover strategy.
I used 2 year optimization period and 1 year test period on daily bars.

--- WalkForward Run ---
02/03/2014 to 12/31/2015, NetProfit=12925.00, Parameters=20,25
02/25/2014 to 12/31/2015, NetProfit=912.50, Parameters=15,25
02/20/2014 to 12/31/2015, NetProfit=-25900.00, Parameters=10,25
--- WalkForward Run ---
02/03/2015 to 12/30/2016, NetProfit=10325.00, Parameters=20,25
02/06/2015 to 12/30/2016, NetProfit=9862.50, Parameters=15,25
03/13/2015 to 12/30/2016, NetProfit=-4025.00, Parameters=10,25
--- WalkForward Run ---
02/25/2016 to 12/29/2017, NetProfit=21637.50, Parameters=10,25
02/03/2016 to 12/29/2017, NetProfit=4187.50, Parameters=20,25
02/19/2016 to 12/29/2017, NetProfit=-3637.50, Parameters=15,25
--- WalkForward Run ---
02/02/2017 to 12/31/2018, NetProfit=14112.50, Parameters=20,25
03/27/2017 to 12/31/2018, NetProfit=1887.50, Parameters=15,25
03/24/2017 to 12/31/2018, NetProfit=-9587.50, Parameters=10,25

As you can see this gives us access to the individual optimizations, but no way to gather them together
as a set and derive analytics on the whole set of optimizations in the walk-forward analysis.

In my case, I can probably write to a file and aggregate the data from there.

So that would be my feature request: that the optimizer type (or something similar) provide events before
and after each walk forward test.

In keeping with the NinjaTrader design patterns, these could be like OnBarUpdate in a strategy where we
just inherit the Optimizer class and override the methods as needed:

OnBeforeWalkForward
OnAfterWalkForward

[tgn55]

Steve all the above (and more) is provided in Ninja if you open the Log tab of Strategy Analyzer... and you can export your entire history of optimizations to Excel to analyse further if you wish.

If you really need to get it programmatically you could also read the XML files (where all this is stored) under the strategyanalyzerlogs folder . What is NOT stored are the individual trades... just the summary data for each backtest.

Hope it helps,
T.

[Steve L]

Thanks T. That's very helpful.

[NinjaTrader_ChelseaB]

Hello Steve L,

Thank you for your notes.

I'll submit your feature request for an event triggered as an optimization begins and an event that triggers when the optimization ends.

Once I have a tracking ID for this request I will post in this thread for future reference.

[NinjaTrader_ChelseaB]

Hello Steve L,

I've received tracking ID# SFT-4268 for this request for events that trigger before and after an optimization.

Please note it is up to the NinjaTrader Development team to decide if or when any request will be implemented.

Thank you for your request on this.

[fx.practic]

Please, take my vote, also.

[NinjaTrader_ChelseaB]

Hello fx.practic,

I've added your vote to SFT-4268.

Thanks for your voice.