Market Replay fill logic bug

[NinjaTrader_Matthew]

Thanks, received.

On the replay files you sent, 2013-12-02 4:04:12 ET the market is trading around 1.3586... The market does not get down to 1.3550's until about 4:36:00 ET (AM).

Can you also post a screenshot of your chart showing the execution marker for the problem scenario? Please try include the chart's time axis as well so I can investigate further.

[bluelou]

Matthew,
See the attached image. Sorry, I had the wrong time. It should be 2013-12-02 11:04:10 ET.

The bid never touched 1.3550 so the short limit order at 1.3550 was never marketable and should not have been filled.

Also, both "Enforce partial fills" and "Enforce immediate fills" are unchecked.

[bluelou]

Matthew,
I think I know what's going on. NT seems to be ignoring the bid/ask series, ignoring when resting limit orders become marketable, and instead the fill logic is if a trade (in any size) has occurred at your price then NT regards your order as a fill.

I hope I'm wrong or, at least, I hope there's a solution otherwise that would mean NT market replay and backtest fills don't represent real markets.

[bluelou]

Matthew,
Still there?

[NinjaTrader_Matthew]

I could not reproduce a fill at 11:04:10, but at 11:04:08, the bid was at the limit price for several ticks. Here is the OnMarketData calls around the time of the fill

Bid at Limit Price @ 12/2/2013 11:04:08 AM
Last price: 1.3550 Volume : 2
Bid at Limit Price @ 12/2/2013 11:04:08 AM
Last price: 1.3550 Volume : 4
Bid at Limit Price @ 12/2/2013 11:04:08 AM
Last price: 1.3550 Volume : 1

This screen shot using the replay data you provided shows the bid & last trade price at 1.3550 and the ask at 1.3551.



Based on these variables, I would expect a fill. Printing OnMarketData() events around the time of the fill show the sell limit order was marketable as the bid price was at 1.3550 at the time the order executes in my test environment. We should also keep in mind that is virtually impossible to accurately simulate the market, so the fill algorithm at any rate is going to be a 'best guess' and is based off of the core market data events received.

I would then suggest adding some prints to OnMarketData() in your strategy and checking the e.Price for the bid/ask updates around the time of the fill to see if this helps clarify what is occurring. The bid/ask data series added has limitations in term of granularity and is not used for the fill algorithm, it is the underlying market data events which is used.

[bluelou]

Matthew,
Okay, I will use OnMarketData instead. I wasn't aware that this was more accurate.

I have several other cases that are the opposite of this one. That is, the limit order is marketable but doesn't get filled. I'll try using OnMarketData and see if that gives me different results.

What is the difference between "core market data events" and Level 1-type functions such as GetCurrentBid? What is the best way to see these core events? using OnMarketData?

[NinjaTrader_ChelseaB]

Hi bluelou,

There is a difference with using GetCurrentAsk() and GetCurrentBid() historically vs real-time.

This is also true of NinjaTrader 7.

From the help guide:
"Note: When accessed during historical processing, the close price of the evaluated bar is substituted."

Below I am adding a link to the help guide on GetCurrentAsk().
http://ninjatrader.com/support/helpG...currentask.htm

With OnMarketData, this will always be the actual bid and ask provided that the method has data available to trigger. (Meaning tick replay is enabled)

[bluelou]

Chelsea,
Thanks for that, tho that wasn't a question I had Maybe you can help me figure out my current question?

I'm trying to compare Market Replay and Strategy Analyzer results. I'm getting very different results for the same strategy and I'm trying to understand why.

I don't think the issue is solely a matter of fill assumptions. However, while I can see each market event using OnMarketData with Market Replay I don't have the same level of visibility with Strategy Analyzer.

With Strategy Analyzer I only have Print() statements. I can't tell if the Print statements show a problem or just don't have sufficient granularity (I suspect that latter is the case). For instance, a Print statement will show a market as 1.3500 bid x 1.3500 ask (a locked market). But, OnMarketData will show something very close to the correct sequence of events, say 1.3499 bid x 1.3500 ask..

Question:
Given I'm working with 1-tick bars (bid/ask/last) in Strategy Analyzer and (GetCurrentBid/Ask + 1-tick last series) for Market Replay what are the differences in the 2 simulation engines, other than fill assumptions?

Hi bluelou,

There is a difference with using GetCurrentAsk() and GetCurrentBid() historically vs real-time.

This is also true of NinjaTrader 7.

From the help guide:
"Note: When accessed during historical processing, the close price of the evaluated bar is substituted."

Below I am adding a link to the help guide on GetCurrentAsk().
http://ninjatrader.com/support/helpG...currentask.htm

With OnMarketData, this will always be the actual bid and ask provided that the method has data available to trigger. (Meaning tick replay is enabled)

[NinjaTrader_ChelseaB]

Hi bluelou,

This was in response to:

Matthew,
Okay, I will use OnMarketData instead. I wasn't aware that this was more accurate.

I just wanted to clarify on why this may be more accurate.


The simulation engine is the same for Replay101 and Sim101 so the behavior will be the same.

The question is, is the data the same? Is the strategy making the same decisions because it has the same data at the same time?

You mention:

With Strategy Analyzer I only have Print() statements. I can't tell if the Print statements show a problem or just don't have sufficient granularity (I suspect that latter is the case). For instance, a Print statement will show a market as 1.3500 bid x 1.3500 ask (a locked market).

What print statement? What are you printing?

But, OnMarketData will show something very close to the correct sequence of events, say 1.3499 bid x 1.3500 ask..

So in the Strategy Analyzer backtest you are getting the correct data, but this is not what you are using for the logic that places orders?
Your print from above, what is this that you are comparing?

If you are comparing to GetCurrentAsk() or GetCurrentBid(), my previous post was trying to tell you that these do not work in the Strategy Analyzer. They return the close price when use historically.

[bluelou]

Chelsea,
I can answer your questions a bit later. but, I think you're already pointing me in the right direction. That's reassuring to know that the sim engine is the same. Now, onto the question you posed:

"The question is, is the data the same? Is the strategy making the same decisions because it has the same data at the same time?"

What is your answer to this? Is this how market replay and Strategy Analyzer differ? Is market replay a truer representation of the exact sequence of how quote and trade arrivals occurred in an actual market? Especially when using Market Replay in combination with GetCurrentBid/Ask?

[NinjaTrader_ChelseaB]

Hi bluelou,

I don't know what you are printing.

If you are comparing OnMarketData with real-time and historical with tick replay enabled, then I would expect the data to be the same in backtest or live.

If you are comparing GetCurrentAsk() with OnMarketData, these will not be the same historically.

So, again, it really just depends on you and what you are using in the script. Is tick replay on? What are you printing?

If you print the time, price, and type of each tick do these match between historical and real-time?

[bluelou]

Chelsea,
Let's try to consider one more angle. The strategy uses 100% resting limit orders; i.e., limits or stop limits.

Regarding fills what assumptions does market replay make? Using Strategy Analyzer, what would be the best approximation to a Market Replay fill?

Here's the issue: I'm seeing my limits become marketable but they're not always filled. So, if I have a long buy limit order resting at, say, 1.1000 and the market subsequently goes asked at 1.1000 then my limit order is marketable and I should get a fill.

Now, with market replay I appear to be getting most, but not all, fills as anticipated. But, with Strategy Analyzer over 1/3 of the Market Replay fills are missing.

What's throwing me off is that it's the exact same code except for the data series. Market replay uses GetCurrentBid/Ask, and Close(1-tick, Last). Strategy Analyzer uses 1-tick series for bid/ask/last.

What is accounting for the difference in fills between the 2 modes in your opnion?



Thx,
Lou

[NinjaTrader_ChelseaB]

Hello Lou,

To get the same behavior as Market Replay when testing in the Strategy Analyzer enable tick replay.

[bluelou]

I just added more to my last post. What is 'tick replay' in Strategy Analyzer? I'm already using 1-tick data series.

[bluelou]

Ah, you mean COBC = false, right? Yeah, I can't do that in this case. I need another volume bar series elsewhere in the code.

So, do you think that would account for the difference? If I set the Strategy Analyzer version with COBC= false then I would get fills much closer to what's happening w/market replay?