Backtesting Multi-Timeframe does not match real market data.

[MONTERO]

I have a Multi-Timeframe strategy based on two volume charts and one tick chart. It calculates on bar close of the primary timeframe. If I back test on ES or ER and compare the results with the strategy applied to a chart of the same time period they never match. This is very distressing because I do not know which data to believe so I can optimize the strategy intervals.

[NinjaTrader_Ray]

Can you clarify what you mean by comparing the strategy to a chart of the same time period?

- Do you mean comparing the Strategy Analyzer chart to a real-time chart of the same interval?

[MONTERO]

I run a backtest and then select the "Periods" and pick a day from the backtest periods. I then compare the total of that day to the real chart (not backtest chart) by manually calculating all of the trades in the real chart.

Whitmark has been helping me and gave me the idea to verify the trades on the real chart with the trades listed under the "Trades" tab. Those do not match the trades in the real chart either.

[NinjaTrader_Ray]

The trades in the trade tab does not necessarily reflect the backtest. The performance stats of in the "Performance" tab reflects the executions in the account. A backtest reflects executions within the strategy. The account could have executions generated outside of the strategy or, generated by multiple strategies.

Results can differ in backtest compared to real-time also. See information on this here.

http://www.ninjatrader-support.com/HelpGuideV6/DiscrepanciesReal-TimeVsBacktest.html

[MONTERO]

The trades in the trade tab does not necessarily reflect the backtest.

Then what is the purpose of the trade tab?

The performance stats of in the "Performance" tab reflects the executions in the account. A backtest reflects executions within the strategy.

I do not understand what you are explaining. The performance tab is for the live account?

The account could have executions generated outside of the strategy or, generated by multiple strategies.

I am only running one strategy that executes on bar close so why would there be executions outside the strategy if I only want the strategy to create the executions?

I have already read the document for backtesting. Since my strategy only executes on bar close there should be no issue.

Thanks for your help

[NinjaTrader_Ray]

Performance tab - Generates performance on an account
Strategy Analyzer - Generates performance on a strategy

- An account holds all executions for an account
- A strategy only holds executions for that strategy which can be a subset of the executions contained in an account

In NT 6.5 we have enhanced this so reports can be generated for an account or for a NinjaScript strategy.

Also, you can not compare a backtest results to real-time chart results. There are many variables especially if you are dealing with volume based charts that can render the results different.

[MONTERO]

So if you created a multi-timeframe strategy that executes on bar close and you wanted to optimize the time interval settings so you could have it auto trade for you and make a lot of money how would you verify your setup is optimal?

Thanks

[NinjaTrader_Ray]

For clarification on my end:

- Time interval --> Are you referring to bar interval such as 100 volume vs 1000 volume bar charts?

- If yes, how would you go about optimizing these values since NT does not support optimization on bar intervals.

Thanks in advance.

[MONTERO]

Yes, I have (for example) a strategy that trades off a 100vol and refers to a 200tick and a 1000vol chart. I have been trying to find the right combination of numbers like 110v 220t 1200v or would it yeild better with 120v 240t 1230v? I have been changing these variables in backtest and recording the results. One day I decided to compare one weeks of back test to the strategy applied to a regular chart with same setup. The results were grossly different.

I have re-proved this many many times with never a match. Sooo do I go with the results from manually calculating the trades in the chart or do I go with the results from the backtest to determine the best combination? So that in the future when I apply the perfected high yeilding strategy to a live trading chart it will continue the great trend.

If I do not use the results from the very convenient back test to help me tweak the numbers then what good is the back test feature for anyone if the data is not valid?

Thanks

[NinjaTrader_Ray]

I see, you manually changed interval settings and re-ran the backtest results.

Please refer back to my original link of why results can be different.

http://www.ninjatrader-support.com/HelpGuideV6/helpguide.html?DiscrepanciesReal-TimeVsBacktest

Look at the very bottom and you will see a link to "chart bars are built". Tick based bar intervals (Volume, Tick and Seconds) will produce different charts based on the range and session start/end times. This is where the discrepenancy likely comes into play.

[MONTERO]

So the data from the live (stand alone) chart is from Zen-fire and the data from back testing is not?

How would you perfect your strategy setup then???

The results from chart and backtest can vary as much as +$400 to -$322 for the same day. I can understand a minor discrepancy of some ticks carrying over to another bar but I just doesn't seem logical to have such a wide gap.

How would you, NinjaTrader_Ray, perfect your strategy setup then???

Help I'm sinking.

Montero

[NinjaTrader_Ray]

I can't comment if the data in real-time is different than what your backtesting against. If it is from the same data source then it should be the same.

I think you underestimate what a minor difference can do to a strategy. If your strategy is that sensitive, one could argue that its not robust. A minor difference in one bar can preclude one very profitable trade that can throw off your performance in a single day.

- If the underlying data set is different, you will get different performance results
- If the underlying data is identical (same number of bars, same start time and end time etc..) and the strategy produces different results ---> Then there is a NT logic issue that we will need to correct

[MONTERO]

I will do some research for a single day and compare each bar in back test and on the chart to see if there are differences bar by bar.

[richa61416]

Montero, I would recommend that you record realtime data, to test against any strategy tester results. Your hand recorded testing of your strategy, also fails to take into account slippage especially on the ER2.
Also, you can test you strategy on real time data, then run the backtester for that day, to see what it generates.

What is key however, is to ensure that the property setting on the chart, is the same as the session start and end times for the period you are backtesting. If you are using trailing stops also, I find that they appear much better in backtesting that in reality.

[NinjaTrader_Ray]

I should have added to my last comment that this is true for historical data only. A strategy running in real-time (Sim101 or a live brokerage account) will for sure produce different results than a historical backtest.