Strategy Optimization in NJ8 without Overfitting/Underfitting

[rezamerik]

What's the most effective way to optimize a simple strategy in NJ8 like MA crossover (or any strategy for that matter) while avoiding overfitting/underfitting?

Let's say I want my strategy to perform best in Dec 2023 - would it be best to optimize the strategy based on data from the last 3 months, the last 6 months, Dec 2022, or some other specific dataset for the optimization process? Obviously, past performance doesn't guarantee future results, but I'm curious how should one optimize their strategy!

Also is there a method for integrating different data segments during the optimization phase (like optimizing the strategy based on December months from the past 10 years for example? Does NJ8 provides techniques like random sampling, k-fold cross-validation, etc. for strategy optimization?

Any insightful advice, recommended resources, or informative reads on this topic would be greatly appreciated. Thank you!

[NinjaTrader_BrandonH]

Hello rezamerik,

Thanks for your post.

So I may accurately assist, what exactly are you referring to when you say "while avoiding overfitting/underfitting"?

Yes, you could backtest a strategy in the Strategy Analyzer over 3 months or 6 months of historical data that you have downloaded. Historical data could be downloaded in the Tools > Historical Data window.

You simply need to set the Start Date and End Data to the dates you want to use in the Strategy Analyzer settings to test over that period of historical data.

No, you cannot backtest/optimize a strategy in the Strategy Analyzer over different data segments, like the December months from the past 10 years.

See the help guide documentation below for more information.

Downloading historical data: https://ninjatrader.com/support/help...8/download.htm

Running a backtest: https://ninjatrader.com/support/help...a_strategy.htm

Optimization: https://ninjatrader.com/support/help...a_strategy.htm

[rezamerik]

Thank you for getting back to me, Brandon. I'm curious about how people go about optimizing their strategies using historical data to achieve the best possible outcomes in the future. Specifically, how does one decide on the data they should use for optimization? If I aim for my strategy to perform optimally in the coming months, should I focus on optimizing the model using data from just the past few months, the past 6 months, or perhaps the past few years?

[NinjaTrader_BrandonH]

Hello rezamerik,

Thanks for your notes.

We do not have information to provide on what data should be used for optimization. It would ultimately be up to you to test optimizing the strategy using different parameters and timeframes to see what suits your overall goals the best as each strategy is different and results are dependent on how the strategy is programmed.

This forum thread will be open for other community members to share their insights on what they find useful for optimization settings.

[Peterkallas]

rezamerik , Everyone chooses something different.

From my journey, I am finding that what matters is it's a delicate art and science.

The first thing in my opinion - is your sample size large enough to provide statistical significance. Data scientists use t-test or Anova for this.

Once you determine your sample size is large enough, you can leverage walk forward analysis to help prevent over-fitting. This breaks the data into in sample and out of sample.

Keep in mind, the more degrees of freedom you optimize the more chance you can overfit, and the more precautions and data you need to ensure you don't

Sample sizes could range from 500-5000 trades to get a 95% confidence interval. This is very specific to your analysis and no one can "give you the answer", they can just talk about the process

When choosing a in-sample and out of sample period, this is also a delicate art/science. With 1 degree of freedom, you may want to use a 2:1 ratio in sample vs out of sample, where as a 3 degree of freedom optimization you may want to use a 4:1.

Hope this get's you started.

Take care,
Peter

[QuantKey_Bruce]

Additionally, good values for your continuous parameters are values for which the nearby values are also good. If the length of an MA (to stick with your example) has great results for 19 but poor loses money with 18 and 20, normally, you can reasonably conclude that this is not picking up on a fundamental feature of the market but rather is a statistical anomaly unlikely to be repeated.

Additionally to that, you can estimate your data mining bias by taking into consideration how many tests you have done and determining through analysis of the complexity of your rule set and the number of tests in your optimization how much confidence you have left.

There is indeed an art to optimizing in a way that "works" - just clicking "Run" will give you something, but it's something that's unlikely to be robust going forward.

[rezamerik]

Thank you Peterkallas and QuantKey_Bruce for sharing your insights! Can you suggest any books, courses, or other resources that offer a practical case study illustrating the use of k-fold cross-validation to optimize a multi-variable strategy? I'm specifically keen on learning how to fine-tune the number of variables in a strategy and then train it to achieve optimal future performance (preferably using NinjaTrader). Thank you once again!

[QuantKey_Bruce]

Bear in mind a time series is not like a big bag of data. K-fold cross validation isn't necessarily a great way to go about that. There are lots of resources online, but a quick Google yields this one which might be good for you to start with not least because it explains why: https://towardsdatascience.com/dont-...g-30b724aaea64

[rezamerik]

Thanks for sharing the link, Bruce! I was uncertain about whether a time-series split or a walk-forward approach would result in a better outcome, but what you provided is an excellent starting point Much appreciated!

[QuantKey_Bruce]

Yes, a walk-forward is what you want for testing any non-stationary time series. The reason is simple - the characteristics of the time series are changing, and k-fold and similar techniques get the time series data out of order which means you are no longer testing in a way that is reproducible. After all, you can't go back and trade past periods of time in the live markets... the market only moves forward and presumably, on average, the most recent periods of time are the most relevant in each case.

[rezamerik]

That's extremely helpful, thanks a bunch for your wisdom on this Bruce!