Strategy Analyzer vs. Market Replay Results

[troy_xp]

Hi,

I'm trying to understand why I'm getting different results when running a strategy using Market Replay vs. the Strategy Analyzer.

My strategy is using 3 min bars as well as tick data. The 3 min. bar data is used to generate trade signals and the tick data is used to manage the trade (moving stops, etc). In the OnBarUpdate() function I'm checking if 'BarsInProgress == 0' for the bar data and 'BarsInProgress == 1' for the tick data. I then branch off into different logic depending on what time frame I'm in.

When I run it using the Strategy Analyzer it seems as if the tick data is ignored. I'm assuming the reason is because the Strategy Analyzer doesn't allow the use of multiple time frames in the same manner as Market Replay or live data - is this correct? I have also noticed that the simulated stops don't work the same under each scenario.

I have read other posts in the forum and I think I understand what I need to do (including looking at the 'SampleEnterOnceExitEveryTick' sample strategy); but before I re-code my strategy I wanted to check that making these changes will help get similar results...

My understanding is that instead of adding another time frame to the strategy (ex. Add(PeriodType.Tick, 1) ) I need to set the CalculateOnBarClose = false while using 3 min. bars. I then should check if it's the 'FirstTickOfBar' and use that for my trade signals (actually looking back one bar). And if it's not the 'FirstTickOfBar' I then use my trade management logic (at the tick level).

Will making these changes get me similar results when testing the strategy using Market Replay and the Strategy Analyzer?

Any comments greatly appreciated.

[NinjaTrader_Bertrand]

troy_xp, first of all I would recommend to check into this link here explaining potential discrepancies you witness while evaluating your strategy - http://www.ninjatrader-support.com/H...sBacktest.html

The Strategy Analyzer would allow backtesting strategies on multiple series / timeframes, however for working with CalculateOnBarClose = false you need to use realtime or market replay data to notice effects (then your proposed logic is correct).

For backtesting intrabar fills, you would need to add a finer granularity to execute orders to and potentially also need to recode logic to simulate the higher timeframe conditions in a lower frame to be able to fire trades intrabar - http://www.ninjatrader-support2.com/...ead.php?t=6652

[tradejockey]

flaws

The problem is manifold..
1. backtest does not use spreads. slippage defaulting to 0 is very misleading and does no service to anybody. set it to 1 to simulate spread. This will set 1 tick to get in and 1 tick to get out. Setting slippage to 0 will give very rosy results but you will realize that it wont work in reality.
2. backtesting assumes that a tick with its date time stamp is part of a bar for that time frame. In reality, a tick for one bar may wind up in the next bar due to the internet latency and all that jazz especially the last tick of a minute.
3. replay testing is pretty much useless as it does not back fill for gaps. This is particularly annoying as you cant backtest if data is missing. Say you record for whole day starting at say 3 am.... It will not back fill for 12 am to 3 am causing your strategy to not calculate for data between that time. Say dax starts at 2 am, then the strategy wont consider what happened between 2 to 3.
4. historical data is a major nuisance especially if your provider is zenfire.

[NinjaTrader_Bertrand]

tradejockey,

1. Correct, you would need to enter your 'spread' manually. NinjaTrader 7 will offer access to historical bid / ask data (if provider supported) - http://www.ninjatrader.com/webnew/NT7/NinjaTrader7.html

2. Correct, that's why it's essential to use live forward testing, too.

3. Answered this in your other post, with NT7 you would be able to use continuous replay across days / session breaks

4. Sorry don't follow here...

Have a good weekend

[pbz1111]

The problem is manifold..
3. replay testing is pretty much useless as it does not back fill for gaps. This is particularly annoying as you cant backtest if data is missing. Say you record for whole day starting at say 3 am.... It will not back fill for 12 am to 3 am causing your strategy to not calculate for data between that time. Say dax starts at 2 am, then the strategy wont consider what happened between 2 to 3.

But if you have a whole day's worth of data, without gaps, would it work right?

[NinjaTrader_Bertrand]

But if you have a whole day's worth of data, without gaps, would it work right?

Yes, correct pbz1111