Reliability of Market Replay testing

[pbailey19]

I'm developing an auto-trade strategy for the e-mini futures using Renko bars. I'm well aware that strategies which look very good in backtesting (especially when using Renko bars) can behave much differently in real-life market conditions. That's why I verify all of my strategies using Market Replay data. This has led me to wonder a couple of things, though:

(#1) Is there any coding or strategy technique that minimizes the difference between backtest performance and test-forward performance (using Market Replay)? For example, using limit orders instead of market orders?

(#2) Would this technique work with a range bar or Renko bar based strategy?

(#3) Can Market Replay test results be affected, depending on the use of fast forward? For example, if I want to run a week's worth of tests on an automated strategy, and I bump the playback speed to 500x, are my results compromised somehow?

[NinjaTrader_PatrickH]

Hello pbailey19,

Thank you for your post.

The only option I am familiar with to help improve the results of backtesting is to use a smaller data series for the orders. This is detailed in our reference sample at the following link: http://www.ninjatrader.com/support/f...ead.php?t=6652

With Range or Renko bars this will not help to improve the results as the bars will still need to close to signal the order placement. And the Market Replay speed should not effect the results of the strategy.

[sledge]

I'm developing an auto-trade strategy for the e-mini futures using Renko bars. I'm well aware that strategies which look very good in backtesting (especially when using Renko bars) can behave much differently in real-life market conditions. That's why I verify all of my strategies using Market Replay data. This has led me to wonder a couple of things, though:

(#1) Is there any coding or strategy technique that minimizes the difference between backtest performance and test-forward performance (using Market Replay)? For example, using limit orders instead of market orders?

(#2) Would this technique work with a range bar or Renko bar based strategy?

(#3) Can Market Replay test results be affected, depending on the use of fast forward? For example, if I want to run a week's worth of tests on an automated strategy, and I bump the playback speed to 500x, are my results compromised somehow?

for what it is worth - I do find, even with 'immediate fill' (used to fight multiple contracts slippage) and using 1 contract, that market replay is still 1 tick better than live (with market order entries and using stops for exits).

You should use 1 contract to fight slippage multiple trade counts in summary reports for your strategy, and if you need to use more contracts, just multiple the results * # of contracts... else your charts and % profitable are totally skewed to who knows what. this is known issue that won't be addressed.

[koganam]

for what it is worth - I do find, even with 'immediate fill' (used to fight multiple contracts slippage) and using 1 contract, that market replay is still 1 tick better than live (with market order entries and using stops for exits).

You should use 1 contract to fight slippage multiple trade counts in summary reports for your strategy, and if you need to use more contracts, just multiple the results * # of contracts... else your charts and % profitable are totally skewed to who knows what. this is known issue that won't be addressed.

Unless, of course, you are pyramiding before or until exit.

[sledge]

Unless, of course, you are pyramiding before or until exit.

For 1 strategy, Market Order entry of say 3 contracts in Market Replay usually filled at same place. Say 1846.00. Scaled out 1847.00, 1848.00, 1849.00. Great! NT reports 1 trade over all and it's profitability/failure as one unit.

Next day, market order 3 contracts, 2 filled at 1846.00, 1 @ 1846.25. Fine. I accept this in normal life. But the reporting system considers this as 2 different trades?
Your win/loss ratio and win % are now skewed with 2 trades when it should be one.

The only thing I've seen to contain this was someone recommended here to use 1 strategy for each contract scale out, and treat them individually.

I have a post some where on here about it.