Market Replay fill logic bug

[bluelou]

In the link you posted the doc states the following regarding market replay "...the fill algorithm is dynamic...".

It would be helpful to know exactly in what way it is dynamic.

[bluelou]

Chelsea,
I think I found a possible answer to what is going on with my strategy w.r.t. market replay and strategy analyzer: the volume bars are different.

One part of the strategy uses a slower volume bar data series and this data is stored in a list for use in a historical lookback. I just realized that these bars are formed differently for mkt replay vs strategy analyzer; e.g., different closing prices.

Should that difference be expected? If so, then fine, at least I have a possible answer.

~Lou

[NinjaTrader_ChelseaB]

Hello Lou,

Just to clarify, you are using Volume[0] in your script and these volume prints for the primary series are different between realtime and historical when the bar closes?

Can you post prints of the volume with time stamps historically and then print the same time stamps with volume in market replay so I can see the difference?

[bluelou]

Hello Lou,

Just to clarify, you are using Volume[0] in your script and these volume prints for the primary series are different between realtime and historical when the bar closes?

Can you post prints of the volume with time stamps historically and then print the same time stamps with volume in market replay so I can see the difference?

1) No, not Volume[0]. In this case it's Volume = 1000 (i.e, 1000 contracts).
2) Yes, the market replay and historical bar closing prices are different in some cases. This is affecting the sim results.

Okay, I'll post the prints here.

[bluelou]

From market replay: Time 2013-12-03 11:10:54 volume_close 1.35940
From historical: Time 2013-12-03 11:10:54 volume_close 1.35930

The bars also look different. There are unchanged bars (open == close) in one volume bar series that are not in the other.

The difference in the print above was enough to cause a difference in the strategy results. Let me know if you need more data or in a different format.

[NinjaTrader_ChelseaB]

Hello bluelou,

I am asking you what you are printing in your code that is different.

Are you not printing Volume[0]?

Are you printing VOL()[0]?

What is the code in your print so I can try on my end?


I suspect that your PC clock is slightly off. This will cause your downloaded historical data to be shifted off of the market replay data.

If you go to time.gov, is your computer clock in exact sync to the very second with the time shown in time.gov?

[bluelou]

Very good, ChelseaB!

Yes, the clock is off by about 4-5 seconds from time.gov and is set to synchonize with time.nist.gov.

I recall having an issue like this a few years ago but I can't remember what the resolution was. What do you suggest I do to resolve this with respect to NT?

Hello bluelou,

I am asking you what you are printing in your code that is different.

Are you not printing Volume[0]?

Are you printing VOL()[0]?

What is the code in your print so I can try on my end?


I suspect that your PC clock is slightly off. This will cause your downloaded historical data to be shifted off of the market replay data.

If you go to time.gov, is your computer clock in exact sync to the very second with the time shown in time.gov?

[bluelou]

Regarding volume, what I've been looking at is an indicator that returns a volume bar Close price n bars ago.

This volume bar closing price differs slightly when going b/t market replay and backtest. Furthermore, the charts also reflect this difference. The historical volume bar chart has some differences from the market replay volume chart.

Having said that, I think you're onto something with the clock issue. Let me know what steps you recommend to resolve the clock issue.

[NinjaTrader_ChelseaB]

Hi bluelou,

Please use the steps below to change your Time Zone and sync the PC clock, note to set the server to time.nist.gov.

To sync the PC clock:

• Right-click the clock in the lower right corner of your desktop
• Select Adjust date/time
• Click on Internet Time tab
• In the Server: drop-down select time.nist.gov
• Click Update
• If the message that appears says successful your PC clock should now be updated
• If not, select a different server from the Server: drop-down and try again (repeat until one of the servers is successful)

http://mintywhite.com/windows-7/7maintenance/windows-seven-7-sync-system-clock-with-internet-time-how-to/

After updating the PC clock, please restart NinjaTrader, then right-click the chart and select Reload All Historical Data.

After this please let me know if this has not resolved the issue.

[bluelou]

Chelsea,
It's my understanding that if I put print statements in OnMarketData w/market replay that that's about as detailed as I can get in terms of bid/ask/last and size quoted AND it's probably the best representation of the sequence of those market events. Does that sound right?

Here's some findings:
1) The volume bar sync issue may be somewhat isolated - just a few instances.
2) The market replay results appear to match best with Strategy Analyzer when Fill Type = Liberal

3) However, in looking at prints in OnMarketData() it doesn't seem "liberal" at all with market replay. It looks like a fair representation of the sequence of market events - that's a good thing E.g., long limit posted at bid of 1.3500 then market subsequently goes asked at 1.3500 and I get filled. Very straightforward.

4) That leaves the mystery of why I need Strategy Analyzer fill type = liberal to get the results to match market replay. There's something different about the Strategy Analyzer sim engine that I can't figure out.

5) For now, my take-away is that market replay really is a good representation of the actual sequence of events. Also, that strategy analyzer seems to have some quirks that lead to missing events at a periodicity of a 1-tick bar.

Do you agree in general? I guess I can drop the issue for now as not 100% resolved - but at least I know the proximate source of any differences.

~Lou

[NinjaTrader_ChelseaB]

Hello bluelou,

Yes, the OnMarketData prints would be the finest granularity of data you can view and does have the correct sequence.


1) With the volume, did setting your clock and reloading the data not correct this?

2) The liberal fill type fills an order when touched. The default fills an order when passed through. As long as there's volume in a live market, it will most likely fill when touched. Thus the liberal fill type is more like a real connection.

3) I'm not quite understanding what you are saying by this. You are looking at the OnMarketData prints in real-time on the Market Replay, but how is this related to a historical fill algorithm?

4) The liberal fill type fills when touched. This is more like a real brokerage and more similar to how the Simulation engine fills orders. Thus you get trades that are more like real-time trades because they fill in a similar manner.

5) When comes to accuracy, there are some differences. With real time data, sell orders can only fill at the bid and buy at the ask. In historical data, when you add in tick series, it can fill with any of those tick series. Just based on where the prices are, likely buy orders will fill at the ask and sell orders will fill at the bid but really they will fill at the first price from any added series that has a price it can fill at.

However, the fill algorithm will make most of the difference here. The historical fill algorithm is not the same as real-time.

[bluelou]

Chelsea,
Yes, I reset the clock but this only affected 1 trade.

Regarding your #3), here's what I'm trying to do:
1) I run market replay and that generates a set of trades, n.
2) I run strategy analyzer, using a fill assumption that my limit orders must be marketable to be filled and that generates a set of trades (n - m)
3) I'm trying to figure out what's causing the subset " -m ". Why are these trades not getting filled?

In a effort to understand this I ran print statements with OnMarketData (mkt replay) in order to make sure that the limit orders really were marketable, which is the case, fortunately.

So, now I'm trying to figure out why marketable limits aren't getting filled in Strategy Analyzer. I understand that I'm comparing 2 different methods (backtest vs mkt replay), but I think a marketable limit order should be counted as a fill regardless of the fill algo.

Hello bluelou,

Yes, the OnMarketData prints would be the finest granularity of data you can view and does have the correct sequence.


1) With the volume, did setting your clock and reloading the data not correct this?

2) The liberal fill type fills an order when touched. The default fills an order when passed through. As long as there's volume in a live market, it will most likely fill when touched. Thus the liberal fill type is more like a real connection.

3) I'm not quite understanding what you are saying by this. You are looking at the OnMarketData prints in real-time on the Market Replay, but how is this related to a historical fill algorithm?

4) The liberal fill type fills when touched. This is more like a real brokerage and more similar to how the Simulation engine fills orders. Thus you get trades that are more like real-time trades because they fill in a similar manner.

5) When comes to accuracy, there are some differences. With real time data, sell orders can only fill at the bid and buy at the ask. In historical data, when you add in tick series, it can fill with any of those tick series. Just based on where the prices are, likely buy orders will fill at the ask and sell orders will fill at the bid but really they will fill at the first price from any added series that has a price it can fill at.

However, the fill algorithm will make most of the difference here. The historical fill algorithm is not the same as real-time.

[NinjaTrader_ChelseaB]

Hi bluelou,

I disagree with your statement.

I understand that I'm comparing 2 different methods (backtest vs mkt replay), but I think a marketable limit order should be counted as a fill regardless of the fill algo.

If the fill algorithm logic fills the orders and the requirements for the orders to fill is different, I would expect different fills.

If one algorithm requires a different price than another algorithm then they are not going to fill in the same place.

What would be the point of fill algorithms in the first place if they don't change how, if, and when orders fill?

[bluelou]

okay, true enough. In my particular use case w/Strategy Analyzer I'm expecting resting limit orders to fill when the limit order becomes marketable - pick whatever fill algo you want except "if-touched". It's not happening.

Hi bluelou,

I disagree with your statement.



If the fill algorithm logic fills the orders and the requirements for the orders to fill is different, I would expect different fills.

If one algorithm requires a different price than another algorithm then they are not going to fill in the same place.

What would be the point of fill algorithms in the first place if they don't change how, if, and when orders fill?

[bluelou]

It seems like our progress has stalled. I suspect that my conclusion is going to be that Strategy Analyzer can't be depended upon to fill marketable limit orders regardless of fill algo.