Backtesting Strategy and Optimization

[bjunaid]

Hi,

I am trying to Backtest my strategies and I am noticing few issues:

1. My strategy only runs up 01/01/20 to 02/17/20 and I get error message Failed to cancel orders. System.NullReferenceException: Object reference not set to an instance of an object. Not sure what is causing this error. Strategy seems to be running fine in real time. How should I got about testing this.

2. Optimization vs. Walk forward Optimization? Will Optimization change the code in my strategy will it just give me different scenarios? What is difference bewteen Optimization and Walf forward Optimization and Multi Objective Optimization?

Thanks,

[NinjaTrader_Kate]

Hello bjunaid,

Thank you for your post.

Does the strategy use Exit methods for stops/targets? If so, this could be a known issue wherein a historical overfill occurs. Would you be able to provide the strategy in question so we may test?

An optimization would not change the actual code in your strategy, no. The only Strategy Analyzer feature that would generate code would be the AI Generate feature which creates a strategy for you. Basically, in an optimization, you provide a range of values for the various user inputs in your strategy, and the optimizer tries the strategy with each potential combination of parameters, then ranks them for you. It's basically a way to help you find what input values work the best for the strategy on a particular instrument historically.

Here's a link to our help guide that goes into detail on Optimizing:

https://ninjatrader.com/support/helpGuides/nt8/optimize_a_strategy.htm

A walk forward optimization allows you to optimize strategy input parameters on a historical segment of market data, then test the strategy forward in time on data following the optimization segment using the optimized input values. The central idea is that you evaluate strategy performance data on the test data, not the data used in the optimization. This process is then repeated by moving the optimization and test segments forward in time.

Here's a link to our help guide on Walk Forward Optimization:

https://ninjatrader.com/support/helpGuides/nt8/walk_forward_optimize_a_strate.htm

A multi-objective optimization takes standard optimization a step further by allowing you to choose multiple objectives to test for. When results are returned instead of a singular list of best results ranked from best to least best instead you will be presented a graph. With multiple objective there is no single best result, instead it's up to the trader to choose what is the best tradeoff between two objectives.

Here's a link to our help guide that goes into more detail regarding multi-objective optimization:

https://ninjatrader.com/support/helpGuides/nt8/multi-objective_optimization.htm

Thanks in advance; I look forward to assisting you further.

[bjunaid]

Hi Kate,

Thanks for that explanation. It was really helpful.

1. For the first question: The strategy has trailing stops which changes the orders at market. I have looked at the trades my strategy is taking and it doesn't show any overfills because I have a code that catches any overfills and exits the position once there is an overfill. My Strategy pre-loads multi-time frames and multi instruments data series however I haven't programmed the strategy to utilize this additional data sets yet. I am not sure if that is causing the issue. I want to know if this is a data related issue? How can I be sure that my data is clean? My data provider is NT Brokerage and Kintect (Stocks). For example, if I am want to test the strategy on NQ 12-20 from 01/01/2020 to 11/30/2020.

[NinjaTrader_Kate]

Hello bjunaid,

Thank you for your reply.

I would doubt that this has to do with the data and more with the strategy hitting a condition that in a backtest causes the backtest engine to get jammed up, so to speak, and cancellation of orders to fail. Keep in mind that your RealTimeErrorHandling setting does not apply to a backtest, as that is considered historical data, so it's possible to still hit an issue with overfills in a backtest.

Does the strategy use the managed or the managed approach? Are you using two Exit orders to protect the position?

Thanks in advance; I look forward to assisting you further.

[bjunaid]

Hi Kate,

Thanks, you are right this error was not caused due to Data. There was an error in my code that I was able to fix last night.

Thanks Again,

[bjunaid]

Hi Kate,

1. When Optimizing results, should we optimize the results for the same amount of time frame as the backtest? For example, I backtested my strategy from 01/01/20 to 11/30/2020, should I use the same amount of time for optimization as well or should I use a shorter period?

2. What are the criteria for ensuring that the optimization results are accurate or being able to differentiate fake results from real results before I try to trade any strategy? For instance in the example below I see that my strategy looks profitable but when I look at the the actual trades, I see something different. Please attached files.

Backtest Results:



Monte Carlo Results:

[NinjaTrader_Kate]

Hello bjunaid,

Thank you for your note.

For compliance reasons, I can't give advice on exactly what you should backtest on or optimize over as that could be construed as trading advice - I can only tell you how to do so.

As far as your other question, I'm not sure exactly what you mean here. Your first screenshot shows the strategy was profitable in a backtest. The second shows a Monte Carlo simulation of the probability that your strategy would make x amount of profit from a sampling of the trades from the backtest.

Keep in mind that you should expect that a strategy running real-time (live brokerage account, live market simulation, Market Replay etc...) will produce different results than the performance results generated during a backtest. This difference may be more easily seen on certain Bars types (e.g. Point and Figure) than others due to their inherent nature in bar formation.

During a backtest you can select conservative or liberal fill algorithms which will produce different results. Fills are determined based on 4 data points, OHLC of a bar since that is the only information that is known during a backtest and there will be no intra-bar data. This means actions cannot happen intra-bar, fills cannot happen intra-bar. All prices and actions come from and occur when the bar closes as this is all the information that is known.

Because of this, OnBarUpdate will only update 'On bar close' as it does not have the intra-bar information necessary for 'On price change' or 'On each tick'.

Also, here is a link to the differences on real-time vs backtest (historical).
http://ninjatrader.com/support/helpG...ime_vs_bac.htm

Adding intra-bar granularity can help with this.

Intra-bar granularity adds a second data series such as a 1 tick series so that the strategy has finer granularity in the historical data in between the OHLC of the primary series. This allows for more accurate trades by supplying the correct price at the correct time for the order to fill with.

In NinjaTrader 8, there have been two new enhancements so that programmers do not have to manually add this secondary series and code the script to for high accuracy fills (Order Fill Resolution) and for intra-bar actions (TickReplay).

Here is a link to our forums that goes into depth on using Order Fill Resolution and Tick Replay to ensure your backtests are as close to real time results as possible:

https://ninjatrader.com/support/foru...mance?t=102504

High Fill Order Resolution and TickReplay cannot be used together. If it is necessary to have both, it is still possible to add intra-bar granularity to a script in the code itself for order fill accuracy and use TickReplay to update indicators with Calculate set to OnPriceChange or OnEachTick historically.

Please let us know if we may be of further assistance to you.

[bjunaid]

Hi Kate,

Thanks Kate! for the detailed explanation. Really appreciate it.

It would really help me if you can reference my question numbers.

1. When we add more than one data series, does the primary data series gets loaded first or does the secondary data series gets loaded first or does it all happen at the same time? For instance, wouldn't my order be submitted late, if my entry signal is based on primary-5 minute bar and I submit the order on secondary-1 minute bar since secondary bar is loaded after the primary bar?

2. Slippage: How can I account for slippage when back testing my strategies, specially when I submit market orders. Do I enter Ticks/$ value in backtest window

Thanks,

[NT-Roland]

Hello bjunaid,

Hi Kate,

1. When Optimizing results, should we optimize the results for the same amount of time frame as the backtest? For example, I backtested my strategy from 01/01/20 to 11/30/2020, should I use the same amount of time for optimization as well or should I use a shorter period?

2. What are the criteria for ensuring that the optimization results are accurate or being able to differentiate fake results from real results before I try to trade any strategy? For instance in the example below I see that my strategy looks profitable but when I look at the the actual trades, I see something different. Please attached files.

ad 1: I strongly recommend not to run optimizations for the entire period you have historical data for. Otherwise, your computer will almost always find something looking promising because you "trained" the strategy exposing ALL data. That's like filling in the lottery ticket after the draw. I myself leave at least 20% of the time frame out and then check if the settings from the (shorter) optimization period produce similarly robust results when applied to the control period (i.e. to "unkown" market data the optimization did not see).

ad 2: Can you share a graph of CumNetProfit (Analysis - Daily - Graph CumNetProfit)? I'd assume, it is not as nice as the Monte Carlo graph. Why?
You seem to cut removing outliers of both winners and losers in your Monte Carlo overview. I suggest to remove winning outliers, but not losers. How does this impact the curve?

Further, to be statistical relevant, you need a certain number of trades. The higher the better. 179 trades might perhaps just not be enough in this respect.
Some developers apply the following rule of thumb: Total number of trades / number of "degrees of freedom" >= 100. Thus, they want to have on average at least 100 trades per filter criteria applied in their strategy in a backtest. They don't necessarily throw everything away not meeting this criteria, but know the outcome is (even) less statistical relevant, if that's not the case.

Your strategy has a very low percentage of winners. It makes a huge difference to "trade" this in a backtest vs. live. You must be 100% convinced your are doing the right thing to be able to continue standing at the helm "no matter what" from a psychological point of view. A lot of traders have an issue with not being right most of the time. Even if you don't double up, compare the win rate of this strategy to a martingale scenario to get a feeling for the expected pain when traded live. How often do you dare to double your next bid if the losing strike continues, even "beyond" expected statistic odds? What's the max. consecutive losers in your Monte Carlo simulation? I'd assume, if it way higher than 40, but even if not, that's already a lot.
You must be cold as ice to trade something like being wrong 85% of the time and make sure to remember you are Mr. SuperCool when the cash drain continues.
Not my cup of tea, but if you can trade that and grow your account, great.
NT-Roland

[bjunaid]

Hi NT Roland,

I appreciate your detailed response. Thanks

I am not trading this strategy because the result seemed to good to be true. I am new to algo trading so I am just learning and educating myself before trading anything live.

When you backtest your strategy do you backtest it on the last 5 to 10 years of data first and then run optimization on 20% of that data then go back and run back test on the same 5 to 10 years of data again? My question is that what are some reasonable steps a person should take before going live for properly backtesting, optimizing, walkforward tesing, and analyzing graphs?







Thanks,

[NinjaTrader_Kate]

Hello bjunaid,

Thank you for your reply.

1. When we add more than one data series, does the primary data series gets loaded first or does the secondary data series gets loaded first or does it all happen at the same time? For instance, wouldn't my order be submitted late, if my entry signal is based on primary-5 minute bar and I submit the order on secondary-1 minute bar since secondary bar is loaded after the primary bar?

The difference in timing would be negligible. The secondary series would process right after the primary series. If you're worried about speed of submitting, use a more granular secondary series.

2. Slippage: How can I account for slippage when back testing my strategies, specially when I submit market orders. Do I enter Ticks/$ value in backtest window

When you're running a strategy, you have the option to add a set number of ticks of Slippage for market orders. The orders will have that amount of slippage applied if the bar allows for that amount of slippage. You may want to try several values for slippage to see how that affects your strategy performance.

https://ninjatrader.com/support/helpGuides/nt8/index.html?understanding_historical_fill_.htm#UnderstandingTheHistoricalFillAlgorithm

Please let us know if we may be of further assistance to you.

[bjunaid]

Thanks Kate!

[NT-Roland]

Hi bjunaid,

When you backtest your strategy do you backtest it on the last 5 to 10 years of data first and then run optimization on 20% of that data then go back and run back test on the same 5 to 10 years of data again? My question is that what are some reasonable steps a person should take before going live for properly backtesting, optimizing, walkforward tesing, and analyzing graphs?

ad Backtest/Optimization Period:
I myself reserve at least 20% of the time frame for control purposes only. This time frame is neither included in backtests nor in optimizations. Thus, if I have 10 years of data, 2 years are strictly left out as if I hadn't them. They only come into play once I think based on the other 8 years, this strategy is good enough to give it a shot. I'd like to trade it live, but I'm glad I can still do a dry run, before I have to ultimately pull the trigger.
I stricly limit the number of allowed "dry runs" in such control period. In other words, I don't revisit the 8 backtest/optimization period over and over again until I ultimately find a combination which also "performs" in the control period. You could perhaps wonder, why not, shouldn't this still be safe for as long as you don't include that control period in the backtest/optimization?
Well, I think, it's not. I carefully picked "my" preferred settings from the backtest/optimization period not knowing how they would perform in unknown periods. Perhaps a hand full of combinations which made most sense to me (quite stable, acceptable draw down, reasonable period to recover, combinations in the "neighbourhood" similarly good, etc.). If I throw them overboard and use another parameter set instead, only after I ran the check in the control period, I start to trick myself. I even put "my" preferred picks to paper before I perform the ultimate control check to avoid this. I might be throwing away far too much with this approach, but the stricter I am in this area, the more confidence I have when going live with a bot which really passed this last (blind) test.
ad WFO:
I don't like Walk Forward Optimization too much. The issue with WFO is that
i) you don't know in which time intervals you should roll (cannot be optimized in NT) and
ii) WFO only applies the "best" setting from one interval to the next.
Well, how can you be certain that i) you picked the right rolling interval at the start (e.g. every 1 months vs. every 3 months vs. ...), ii) that you started the "right" interval at the "right" date/time (Jan 10 + X thru Xn, might give you totally different results than Jan 17 + X thru Xn), iii) that picking the "best " setting for one interval gives you the best overall result? Other than by pure coincidence, that's never the case. Thus, you need to test way more combinations than you can even imagine.
WFO with 1) rolling start dates (for the first interval dominating all further), 2) interval duration which can be exposed to optimization and 3) a flexible number of "best" results from the preceding interval which are carried forward and tested further would be much more beneficial. Until this is available, I myself don't see much value in WFO.
ad Analyzing Graphs:
NT comes with a high number of very useful graphs.
Analysis;
1) Period; From Daily up to Years, Trades, Half-hour of day, Hour of Day, Weekday,
2) Direction; All vs. Long vs. Short,
3) Time base; Exit Time vs. Entry Time,
4) Grapht; CumNetProfit, CumMaxDD, etc.
...
I miss some items like cumulative overviews by same month (e.g. all January's, all Feb's, etc.), all same calendar weeks ( e.g. all CW 1, all CW 2, etc.), but what is available already gives you a bunch of info to study and evaluate.
You can easily curve fit a strategy eliminating whatever you don't like to see when looking at these graphs.
The question is, is there a specific reason why for example some days or times should be excluded (e.g. high volatility around news releases, Non-Farm Payroll, etc.)?
If not, I'd rather leave them in.
All of the above is just my own opinion. I'd love to hear from others how they approach backtesting and optimization.
NT-Roland

[bjunaid]

Hi,

I am in the process of back testing my strategy and I have few more questions:

Q1. I have a paid subscription for Kinetick and TD Ameritrade. When I Back Test my strategy on AAPL 2 minute bars for the period: 01/01/2019 to 12/31/2019 I don't get any results however when I backtest the same strategy with the same criteria for the period: 07/01/2020 to 12/31/2020 it works fine. I thought Kinetick provides 2 years of historical minute data and 10 years of historical daily data. [fyi I have already asked this question via email to Kinetick as well.]

Q2. I Back Tested the strategy on two months of data from July 2020 to August 2020. I noticed that all trades for most of July and August seemed reasonable except for the last trade which exited on session close instead of stop loss or profit target. Please see the attached file. I would like to know how to handle this type of trade when backtesting my strategy because this is skewing my summary and analysis report.

Please kindly reference the question numbers when answering the questions.

Thanks,

[NinjaTrader_Kate]

Hello bjunaid,

Thank you for your reply.

Q1. I have a paid subscription for Kinetick and TD Ameritrade. When I Back Test my strategy on AAPL 2 minute bars for the period: 01/01/2019 to 12/31/2019 I don't get any results however when I backtest the same strategy with the same criteria for the period: 07/01/2020 to 12/31/2020 it works fine. I thought Kinetick provides 2 years of historical minute data and 10 years of historical daily data. [fyi I have already asked this question via email to Kinetick as well.]

Are you able to open and view a chart for the time frame you've tried to test over?

I would recommend checking whether you have historical data for the period 1/01/19 to 12/31/19. You'll need to check in the Historical Data window. If you do not see data there for this time frame, you can try to download it from the Load tab of the historical data window while connected to your Kinetick connection.

https://ninjatrader.com/support/help...ta_manager.htm

Q2. I Back Tested the strategy on two months of data from July 2020 to August 2020. I noticed that all trades for most of July and August seemed reasonable except for the last trade which exited on session close instead of stop loss or profit target. Please see the attached file. I would like to know how to handle this type of trade when backtesting my strategy because this is skewing my summary and analysis report.

The Strategy Analyzer will automatically exit open trades at the end of the backtested period. There would not be a way to keep this from occurring, nor would there be a way to exclude that last trade from the backtest results.

We do have a current feature request for the ability to exclude the last trade from backtest results if you'd like to add a vote to that.

You could optionally trigger any open position to exit with an exit market order on the last historical bar (the last bar of the backtest), however, this would not be accurate to what would occur if you were actually running the strategy, since the position would remain open past the end of the session.

To find the last historical bar:

if (State == State.Historical && CurrentBar == Count — 2)

Please let us know if we may be of further assistance to you.