Announcement

Collapse
No announcement yet.

Partner 728x90

Collapse

Questions regarding WFO and overfitting to all Senior traders in this forum

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    Questions regarding WFO and overfitting to all Senior traders in this forum

    Hello,

    I have two questions concerning Walkforward Optimization, and I'm specifically seeking insights from seasoned backtesters. While opinions are valuable, I'm particularly interested in feedback from those with years of hands-on experience who can offer well-informed perspectives.

    I'd love to hear your expert opinions on the best practices for setting the intervals and steps for each parameter when running Walkforward Optimization, especially in the context of intraday strategies.

    Thank you for your time, and I eagerly await your insights.​

    What are the Best Practices for Interval and Step Settings in Walkforward Optimization (for intraday strategies)?

    In Walkforward Optimization, parameters are first optimized using in-sample data during the "Optimization Period (days)." These optimized parameters are then tested on out-of-sample data in the "Test Period (days)." My initial hypothesis is that using smaller steps for the optimization could risk overfitting. However, since the highly optimized parameters are ultimately validated on out-of-sample data, it may not be a significant concern.

    Small steps:
    Click image for larger version  Name:	small steps.png Views:	0 Size:	9.7 KB ID:	1272416

    Large steps:
    Click image for larger version  Name:	large steps.png Views:	0 Size:	10.3 KB ID:	1272417

    What is best practice on "Optimization Period (days)" and "Test period (days)" when running WFO if you build intraday strategies?​​

    Turning to the specifics of the "Optimization Period (days)" and "Test Period (days)" in Walkforward Optimization, I primarily focus on intraday trading strategies. My hypothesis is to keep these periods relatively short, given that market conditions can change rapidly. I've experimented with both 90-day and 365-day durations for the "Optimization Period (days)" and "Test Period (days)." Interestingly, my strategies have yielded better results in the WFO tests when optimized over 90 days as opposed to 365 days. Additionally, aligning the 90-day cycle with contract rollovers offers a practical advantage.

    Click image for larger version  Name:	Optimization Period & Test Period.png Views:	0 Size:	7.8 KB ID:	1272418
    Last edited by prisonbreaker82; 10-11-2023, 02:18 AM. Reason: improved text

    #2
    Adding the results. With 90 days. It took me 40min to run the small steps vs 1min to run the large steps...

    using small steps WFO
    Click image for larger version

Name:	summary-small steps.png
Views:	196
Size:	183.1 KB
ID:	1272420

    using large steps WFO
    Click image for larger version

Name:	summary-large steps.png
Views:	203
Size:	180.8 KB
ID:	1272421

    Comment


      #3
      Different Optimization Period (days) and Test Period (days)

      30-30
      Click image for larger version

Name:	30-30.png
Views:	202
Size:	44.1 KB
ID:	1272435

      90-30
      Click image for larger version

Name:	90-30.png
Views:	195
Size:	44.6 KB
ID:	1272436

      120-30
      Click image for larger version

Name:	120-30.png
Views:	197
Size:	44.2 KB
ID:	1272437

      90-90
      Click image for larger version

Name:	image.png
Views:	194
Size:	45.7 KB
ID:	1272438
      180-90
      Click image for larger version

Name:	image.png
Views:	197
Size:	44.4 KB
ID:	1272439

      Comment


        #4

        360-90

        180-180

        360-180

        720-180

        360-360

        720-360
        ​​

        Comment


          #5
          Hello prisonbreaker82,

          This thread will remain open for community members to provide insight.

          I wanted to provide a video on how the Walk Forward Optimization works.

          Comment


            #6
            NinjaTrader_ChelseaB

            I have two computers with the same setup, but getting different backtest results...

            I have deleted all minute data on both computers and downloaded it again (I thought, this must be a data issue). I run back test and it shows different results still (not on all instruments, but more than half)! I copy the whole minute folder to the other PC, I still get different results! The YELLOW MARKED items are the same on the two computers, but as you can see multiple instrument do not have the same values! How can this be possible?

            How should I be able to trust the backtest result if you platform is working like this?
            Attached Files
            Last edited by prisonbreaker82; 10-26-2023, 01:42 AM.

            Comment


              #7
              The WFO is still not the same. How can a user take any decison on a WFO if the results are this different for the same strategy, same data(!).

              Can also add this comment from Fat Tails (I emailed him asked about this)... I bet you know about those bugs, the questions is why are you not taking any actions?

              "The NinjaTrader in-built walk-forward analysis has a few bugs relating to the indicator training period (first period when an indicator does not return results, because it is not trained).
              Therefore, I would rather do the walk forward optimization manually."
              Attached Files
              Last edited by prisonbreaker82; 10-26-2023, 02:36 AM.

              Comment


                #8
                Originally posted by prisonbreaker82 View Post
                NinjaTrader_ChelseaB

                I have two computers with the same setup, but getting different backtest results...

                I have deleted all minute data on both computers and downloaded it again (I thought, this must be a data issue). I run back test and it shows different results still (not on all instruments, but more than half)! I copy the whole minute folder to the other PC, I still get different results! The YELLOW MARKED items are the same on the two computers, but as you can see multiple instrument do not have the same values! How can this be possible?

                How should I be able to trust the backtest result if you platform is working like this?
                I removed all files inside the cache folder and day data on both computers, and now the backtest is the same.

                Click image for larger version

Name:	image.png
Views:	158
Size:	10.4 KB
ID:	1274787

                Comment


                  #9
                  Originally posted by prisonbreaker82 View Post
                  The WFO is still not the same. How can a user take any decison on a WFO if the results are this different for the same strategy, same data(!).

                  Can also add this comment from Fat Tails (I emailed him asked about this)... I bet you know about those bugs, the questions is why are you not taking any actions?

                  "The NinjaTrader in-built walk-forward analysis has a few bugs relating to the indicator training period (first period when an indicator does not return results, because it is not trained).
                  Therefore, I would rather do the walk forward optimization manually."
                  And here you can see evidence off this bug...
                  Attached Files

                  Comment


                    #10
                    The backtest results are identical on both my PC and my laptop, see attached. But the WFO is completly different. Different profit factors for most of the instruments. How can a trader take a decision on if the strategy is profitable in a WFO exercise if cannot trust the result that Ninjatrader comes up with?
                    Attached Files
                    Last edited by prisonbreaker82; 10-26-2023, 03:09 AM.

                    Comment


                      #11
                      Hello prisonbreaker82,

                      Are both computers connecting to the same data feed? (different connection types can provide different data)

                      If the data is the same, the code is the same, and the settings are the same, I would expect the results to be the same.

                      I will need to you test a specific instrument that is returning different results and write the information the to a text file so that we can pinpoint exactly what is causing differences.

                      Below is a link to a forum post with sample code and a demonstration video.
                      Citizens of the NinjaTrader Community, A common question we hear from clients is 'why are results from backtest different from real-time or from market replay?'. Live orders are filled on an exchange with a trading partner on an agreed upon price based on market dynamics. Backtest orders are not using these market dynamics.

                      Comment

                      Latest Posts

                      Collapse

                      Topics Statistics Last Post
                      Started by futurenow, 12-06-2021, 05:49 PM
                      18 responses
                      879 views
                      0 likes
                      Last Post dj0ntz
                      by dj0ntz
                       
                      Started by nailz420, 05-14-2025, 09:14 AM
                      1 response
                      119 views
                      0 likes
                      Last Post NinjaTrader_ChristopherJ  
                      Started by NinjaTrader_Brett, 05-12-2025, 03:19 PM
                      0 responses
                      526 views
                      1 like
                      Last Post NinjaTrader_Brett  
                      Started by domjabs, 05-12-2025, 01:55 PM
                      2 responses
                      85 views
                      0 likes
                      Last Post domjabs
                      by domjabs
                       
                      Started by Morning Cup Of Trades, 05-12-2025, 11:50 AM
                      1 response
                      124 views
                      0 likes
                      Last Post NinjaTrader_ChristopherJ  
                      Working...
                      X