Level 1 sim engine differences: backtest vs mkt replay?

[bluelou]

I'd like to get a better understanding of the simulation engine differences between market replay and backtest.

1) For a backtest we can assume the following:
a.] periodicity is 1-tick bars and COBC = true.
b] For a long signal Close(1 tick, bid series) must be >= Close(1 tick, last series)

2) For market replay we can assume the following:
a] 1-tick data and COBC = true
b] For a long signal GetCurrentBid() must be >= Close(1 tick, last series)

Comments:
--Rules 1b and 2b are necessary conditions.
--However, for a market replay the fills and the strategy looks great. For a backtest, many of the expected fills are missing and the strategy is a poor performer.

Question:
What exactly is going on with the different Level 1 data series in the sim engine? Does backtest aggregate ticks but market replay does not? I'd like to understand exactly what NT does here.

Thx,
Lou

[NinjaTrader_MichaelM]

Hello Lou,

Thank you for writing in. To make sure I completely understand what you are saying, you essentially have two strategies (one for historical backtest and one for market replay). The historical backtest strategy has an added 1 tick data series of the bids which you use to do the calculation in 1b. The market replay strategy does not (because it uses GetCurrentBid() instead).

If this is not the case, please note: When accessed during a historical backtest, the close price of the evaluated bar is substituted when using the GetCurrentBid() or GetCurrentAsk() methods.

Furthermore, backtests have significantly different options and something that may be affecting what you are seeing is your historical fill processing algorithmn. Please see this page for more information on historical fill processing and other backtest properties: http://ninjatrader.com/support/helpG...a_strategy.htm

Please see this document in our help guide for more information on what happens when backtesting: http://ninjatrader.com/support/helpG...ime_vs_bac.htm

Please let me know if you have any further questions.

[bluelou]

Michael,
--There is only one strategy.
--The backtest version uses bid/ask/last data 1-tick data series since GetCurrentBid/Ask is not available for the Strategy Analyzer.
--The market replay version uses GetCurrentBid, GetCurrentAsk, AND a 1-tick Last data series.

Assumption:
For now, let's assume the fill algorithms are the same for both the backtest and market replay modes.

Question:
--What I'm trying to figure out is how a 1-tick Ask data stream in Strategy Analyzer is different from GetCurrentAsk using Market Replay.

--For instance, I'm assuming market replay will replay all trades and quotes in the sequence that they were recorded. Is that correct?

--But, what does Strategy Analyzer do? For instance if the Ask hits 3 different levels with a few milliseconds what does Strategy Analyzer assume?

--I'm fairly certain the simulation engines are different. With market replay I can see the sequence of trades and quotes using OnMarketData(). With Strategy Analyzer, using print statements the Level 1 data is not exactly the same as OnMarketData.

--I'm assuming there's some aggregation going on with Strategy Analyzer, I just can't tell what it is, what the difference is. Please explain.

Thx,
Lou

Hello Lou,

Thank you for writing in. To make sure I completely understand what you are saying, you essentially have two strategies (one for historical backtest and one for market replay). The historical backtest strategy has an added 1 tick data series of the bids which you use to do the calculation in 1b. The market replay strategy does not (because it uses GetCurrentBid() instead).

If this is not the case, please note: When accessed during a historical backtest, the close price of the evaluated bar is substituted when using the GetCurrentBid() or GetCurrentAsk() methods.

Furthermore, backtests have significantly different options and something that may be affecting what you are seeing is your historical fill processing algorithmn. Please see this page for more information on historical fill processing and other backtest properties: http://ninjatrader.com/support/helpG...a_strategy.htm

Please see this document in our help guide for more information on what happens when backtesting: http://ninjatrader.com/support/helpG...ime_vs_bac.htm

Please let me know if you have any further questions.

[NinjaTrader_MichaelM]

Hello bluelou,

For now, let's assume the fill algorithms are the same for both the backtest and market replay modes.

This is not a possibility. Backtesting uses historical fill processing and Market replay uses the simulation engine. Market replay will replay all trades and quotes in the sequence that they were recorded.

If you are seeing a difference in data, it is possible that you are not getting the data from the same data provider for your market replay data vs historical data. Different data providers provide different data, especially for tick data. Please see this document for more information: http://ninjatrader.com/support/helpG..._are_built.htm

Please let me know if you have any further questions.

[bluelou]

Michael,
Perhaps I can narrow the question a bit...

With Strategy Analyzer let's assume BIP = 0 is the 1-tick Last series.

Does that mean that my additional 1-tick bid and ask data series are only updated at each 1-tick Last event?

If so, then that would likely explain the difference b/t using GetCurrentAsk in market replay vs a 1-tick Ask data series in Strategy Analyzer. The difference being that GetCurrentAsk (using Market Replay) is updating on each new Ask event while Close(1 tick, Ask) would only update with Strategy Analyzer on each new 1-tick Last event. Does this sound right to you?

[bluelou]

No, it's the exact same data provider, IQFeed.

[NinjaTrader_MichaelM]

Hello bluelou,

IQFeed does not support historical bid/ask tick data. Please see this document for a breakdown of the differences between data providers: http://ninjatrader.com/support/helpG...rical_data.htm

What is possibly happening is that you have historical bid/ask data downloaded from the NT servers (or somewhere else) which is playing into your calculations and causing all this confusion.

As a general rule the ask or the bid should not be updated more often than the last, as the last is supposed to reflect the last traded price be it the bid or the ask.

Please let me know if I may be of further assistance.

[bluelou]

Michael,
Please allow me to offer 2 corrections to your reply:
1) IQFeed offers Level 1 and Level 2 data. It's true that you can't access (most of) the historical Level 1 and 2 data from IQFeed but you can record it and this is what I do using both NT and QCollector. I record the Level 1 data.

2) You wrote the following: "As a general rule the ask or the bid should not be updated more often than the last, as the last is supposed to reflect the last traded price be it the bid or the ask."

I think you're mixing trades and quotes. This goes to the very heart of the question that I'm asking regarding Strategy Analyzer. It's the other way around, bid/ask quote events are typically many times greater in number than trades. I'm trying to figure out what the Strategy Analyzer sim engine does given the circumstances I've described.

~Lou

Hello bluelou,

IQFeed does not support historical bid/ask tick data. Please see this document for a breakdown of the differences between data providers: http://ninjatrader.com/support/helpG...rical_data.htm

What is possibly happening is that you have historical bid/ask data downloaded from the NT servers (or somewhere else) which is playing into your calculations and causing all this confusion.

As a general rule the ask or the bid should not be updated more often than the last, as the last is supposed to reflect the last traded price be it the bid or the ask.

Please let me know if I may be of further assistance.

[NinjaTrader_MichaelM]

Hello bluelou,

Could you please confirm when you refer to quotes are you referring to Level II data?

If you have a 1 tick data series of ask prices and a 1 tick data series last prices, every time the ask price updates so does the last price, for level I data. The same thing goes for the bid price and the last price.

The strategy analyzer should have access to historical data in the sequence in which it occurred.

If you run the backtest version on the market replay connection, do you get the same ask prices as when you run it on the strategy analyzer? If you do not, then please send me the code that I can use to recreate the issue so I may investigate further.

If you do not wish to post it on the forums, please send it via email to platformsupport[AT]ninjatrader[DOT]com with the subject line "ATTN: Michael M #1372472".

Please let me know if you have any further questions.