Fill ERROR in MarketReplay

[dom993]

Hi there,

I believe I am reporting 2 separate issues here - at least, this is my understanding.

Issue #1 : I am testing a strategy using MarketReplay. Even though I did specify a custom FillType ("BetterThanDefault"), Ninja apparently uses the "Default" FillType.
I "know" this because I created the "BetterThanDefault" FillType to always enforce that stops are executed at least at the specified stop-price + slippage, regardless of the H/L of the bar on which the stop is executed.

Note: I have previously backtested the strategy on ~3years of tick-data, using the "BetterThanDefault" FillType, and I know for a fact that it does exactly what it is supposed to do.

I have seen in MarketReplay several instances of stop executed without slippage, including the one which is the subject for the next issue.


Issue #2 : The trade displayed on screen is using MarketReplay with data downloaded from Ninja server. The main timeframe used is Volume-100. The strategy actually places all its orders on a secondary timeframe, which is set on Second-1. The trade is a long taken on CL 06-12 at 9:18am EST. The stop level is set at 97.16, and the stop is triggered at 9:43:51am. There are 3 bars on the main timeframe which all have the same 9:43:51am timestamp. It is the 3rd bar that should triggers the stop. However, the stop appears to be triggered by the 1st of these 3 bars, and executed on the 2nd one, resulting in a price execution which is plain wrong.

I believe that the following is happening: because the trades are executed on the 2nd timeframe, the stop is triggered after the 1st bar on the main timeframe (which it should, as the 1-second bar includes all 3 V100 bars). However, Ninja appears to use the 1st timeframe in the FillType method, and as a result the execution price is wrong (it would be correct if Ninja was using the 2nd timeframe H/L to compute the execution price).

I do believe this problem is specific to MarketReplay, as it doesn't happen when simply loading the strategy on a chart for CL 06-12 on 10/5/2012.


Please understand those 2 issues are important to me, I am still in the process of evaluating NinjaTrader, I have spent the last 4 weeks porting a complex strategy (~5000 lines) with backtesting result pretty much identical to my current platform (I understand exactly why a few trades have slightly different outcome), this is the next step - validating the real-time behavior of the strategy using MarketReplay.

Regards
Dominique

[NinjaTrader_Bertrand]

Hi Dominique, for sure differences can occur as in fill types are only applicable to backtesting, in Market Replay and live simulation you would trade against the simulation engine build into NT.

http://www.ninjatrader.com/support/helpGuides/nt7/simulation.htm

So it could be slippage would not be applied exactly as in your backtesting estimates as the simulator estimated it would have likely gotten you a better fill in this case. Of course a probability based fill estimation cannot account for all factors contributing in realtime, live market queues experienced, it's however the best estimate to arrive at given the underlying data.

The simulator does offer to change delay settings and also you could force partial fills to be returned to you, helping further testing how your script would react to worsening fill conditions.

[dom993]

Bertrand,

Thank you for confirming I had the correct perception of both issues.

Re. your last suggestion (using simulator settings), per NinjaTrader's documentation delay settings don't apply to MarketReplay. That said, delays shouldn't do anything for exchange-hosted stop orders ....

May I respectfully suggest you take 1 feature suggestion to PLM/R&D:

1) possibility of usage of custom FillTypes instead of NT built-in simulation engine for MarketReplay / live simulation.

May I also respectfully suggest you take issue #2 as a bug to R&D, as this is just what it is.

In the meantime, what am I supposed to to to validate the real-time behavior of my strategy, when fills in MarketReplay are incorrect ?

[NinjaTrader_Bertrand]

Correct, the delays are applicable in simulated real-time trading, so that would be the next step after conceptually testing your script on Market Replay and confirming the edges to be holding up.

Thanks for the suggestion here, I've added into our product management tracking system under # 1821.

Market Replay granularity is down to 1 second only, this is a known limitation and on our list to address as well - a higher granularity would be needed for your case here to timestamp the execution finer.

[dom993]

Bertrand,

What should I do to get the same fills in backtest that I get in MarketReplay ? The entire purpose of the MarketReplay step is to check the real-time behavior of the strategy, for this purpose I don't mind getting those wrong fills as long as I can see the same ones in backtesting - because I do not want to go manually over 1500+ trades manually & figure out if every difference between MarketReplay & Backtest is a strategy issue or a fill-issue in MarketReplay mode.

Thanks in advance
Dominique

[NinjaTrader_Bertrand]

Dominique, those 2 would not be expected to line up 100% - it's just different environments you're in. Backtesting can only go so far, but with the added 1 second granularity you have you're very close getting it as accurate as possible. A still higher step, would be executing in backtesting to a 1 tick series, this is the lowest granularity you could achieve.

For a discussion of expected discrepancies seen between Backtesting and Market Replay / Live trading please also see this helpguide article here - http://www.ninjatrader.com/support/h...ime_vs_bac.htm

For the underlying data used - the replay data is ZenFire recorded data from our server - what connection / data provider are you using for your backfills?

[NinjaTrader_Bertrand]

Another item Dominique would be submitting to the actual bid / ask series in backtesting for more accurate fills compared to Market Replay, per default you're working on the last series only here while replay would take the full recorded Level 1 market data for fill estimation.

[dom993]

Bertrand,

I just ran a demo strategy that goes long at 9:23am on May-5-2012 with a stop set at 97.16, on a 1-tick chart. The strategy uses CalculateBarOnClose = false (!), this is really all that can be done at my end.

The result is identical, ie. in MarketReplay the stop is executed at 97.21, way before (in the tick-stream) 97.16 is ever printed on the chart.

Did you already submit this *bug* to R&D, or are you going to, or can you provide me with a custom FillType that will replicate that odd behavior ?

If you tell me how to access NextHigh / NextLow / NextOpen for specific timeframes (it is not as simple as NextHighs[] / NetxLows[] / NetxOpen[] .. I just tried that) then I'll be able to replicate the MarketReplay fill error and from there I can compare backtest results to MarketReplay results.

Thanks in advance
Dominique

PS - re. your question about my historical data, it is from IQfeed ... but this has nothing to do with the issue at hand.

[NinjaTrader_Brett]

Hello,

Bertrand is out the rest of the week so I am jumping in here, I beliee I understand the issue and ave read the entire ticket.

There is no "Fill Type" for market replay.

Market replay is an advanced simulation engine that works with probability of fill based on where the bid and ask price is and what volume is at those points. Somehow innmarket replay you are getting the fill due to bid or ask price being at this value and it is filling you. You may have some bad bid/ask pricing on the Market Reply file you are playing.

There is no way to change how market replay will fill you as it always works this way in live market replay trades.

I believe you have already seen this but I will post it again as it directly relates to what you are working on:

http://www.ninjatrader.com/support/helpGuides/nt7/discrepancies_real_time_vs_bac.htm

-Brett