Market Depth Events : Position Indexes

[ScoobyStoo]

Hi,

It looks like the position indexing of market depth events has changed from zero based to one based with the release of NT7.0.0.16.

e.g. For instruments that have 5 published rows of market depth I am seeing data referring to positions 1,2,3,4,5 as opposed to positions 0,1,2,3,4.

Can you confirm if this is the case and whether it will remain one based for subsequent releases?

Thanks

[NinjaTrader_Dierk]

Nothing has changed with the position indexes for market depth events at least since NT5 (>4 years).

[ScoobyStoo]

Nothing has changed with the position indexes for market depth events at least since NT5 (>4 years).

OK. Hmmmm. Could the position indexing differ between data providers or is it internally calculated by NT?

[NinjaTrader_Dierk]

Nothing has changed there at least since NT5: NT always present you correct position indexes based on logic you went through with Josh and independent from provider.

[ScoobyStoo]

Have tracked down the problem. Doesn't relate to v7.0.0.16. Accidently downloaded some market replay data from the end of Jan 2010 and it looks to be corrupt. Will discard the data...

[NinjaTrader_ChristopherJ]

Thank you for reporting and please let me know if I may be of further assistance.

[ScoobyStoo]

I was a bit hastly earlier. It actually doesn't look like corrupt data because data that I know played correctly with v15 now exhibits different properties with v16.

Please take a look at something for me and see what you think.

1. Download the 6E 03-10 market replay data for the 7th Feb 2010 from your servers.

2. Plug in the attached market data dump indicator, replay the data from the start of the file until the market open and take a look at the output.

3. Then plug in the attached market data dump strategy, replay the data from the start of the file until the market open and take a look at the output.

Notice anything weird? The replayed data isn't the same (see attached dump files). A simple example of this is the first few Level 2 data messages:

In the indicator dump file they are...

Level 2 Market Data
Time : 22:59:59.836
Type : Ask
Operation : Insert
Position : 0
Price : 1.3657
Volume : 110


Level 2 Market Data
Time : 22:59:55.836
Type : Ask
Operation : Insert
Position : 1
Price : 1.3658
Volume : 44


Level 2 Market Data
Time : 22:59:54.836
Type : Ask
Operation : Insert
Position : 2
Price : 1.3659
Volume : 44


Level 2 Market Data
Time : 22:59:43.836
Type : Ask
Operation : Insert
Position : 3
Price : 1.366
Volume : 11


Level 2 Market Data
Time : 22:59:57.836
Type : Ask
Operation : Insert
Position : 4
Price : 1.3661
Volume : 40


Level 2 Market Data
Time : 22:59:59.836
Type : Bid
Operation : Insert
Position : 0
Price : 1.3655
Volume : 1


Level 2 Market Data
Time : 22:59:59.836
Type : Bid
Operation : Insert
Position : 1
Price : 1.3653
Volume : 11


Level 2 Market Data
Time : 22:59:54.836
Type : Bid
Operation : Insert
Position : 2
Price : 1.3652
Volume : 10


Level 2 Market Data
Time : 22:59:55.836
Type : Bid
Operation : Insert
Position : 3
Price : 1.3651
Volume : 159


Level 2 Market Data
Time : 22:59:59.836
Type : Bid
Operation : Insert
Position : 4
Price : 1.365
Volume : 20

In the strategy dump file they are...

Level 2 Market Data
Time : 23:00:00.836
Type : Ask
Operation : Insert
Position : 0
Price : 1.3655
Volume : 1


Level 2 Market Data
Time : 23:00:00.836
Type : Bid
Operation : Remove
Position : 0
Price : 0
Volume : 0


Level 2 Market Data
Time : 23:00:00.836
Type : Ask
Operation : Remove
Position : 5
Price : 0
Volume : 0


Level 2 Market Data
Time : 23:00:00.836
Type : Bid
Operation : Insert
Position : 5
Price : 1.3648
Volume : 16


Level 2 Market Data
Time : 23:00:00.836
Type : Ask
Operation : Remove
Position : 0
Price : 0
Volume : 0


Level 2 Market Data
Time : 23:00:00.836
Type : Ask
Operation : Insert
Position : 5
Price : 1.3661
Volume : 40


Level 2 Market Data
Time : 23:00:00.836
Type : Bid
Operation : Insert
Position : 0
Price : 1.3655
Volume : 1


Level 2 Market Data
Time : 23:00:00.836
Type : Bid
Operation : Remove
Position : 5
Price : 0
Volume : 0

Any ideas what is going on here?

[NinjaTrader_Ray]

Thanks for the information, please give us several days to look into the information you provided.

[ScoobyStoo]

Thanks for the information, please give us several days to look into the information you provided.

No problem Ray. Best of luck with the investigations.

Let me know if you need me to run any tests at my end.

[NinjaTrader_Bertrand]

Unfortunately I was not able to reproduce here ScoobyStoo, what exact chart setup / sessions are you using? I was testing both your scripts on Feb 7h 6E 03-10 freshly downloaded on Default 24 / 7 session and received exactly the same OnMarketDataDepth() events when comparing. Please delete the 6E data from the DB > Data folder, reload it fresh and then only connect to the Feb 7th and try on the same chart as I above and then rerun your scripts for the first minute of replay and compare the output.

[ScoobyStoo]

Hi Bertrand,

I perform my market replay tests using a 1 tick chart and the Default 24/7 session template.

Have deleted and redownloaded the data but am seeing the same results.

I have examined the issue further and think I've found the cause of the problem.

The data that is present in the indicator dump which is not present in the strategy dump is all timestamped before the 23:00:00 start of the session.

Take a look and you'll see what I mean. The Level 2 events building the initial order book are timestamped 22:59:59.836.

This data is being filtered from the strategy output but isn't from the indicator output.

[NinjaTrader_Bertrand]

Thanks ScoobyStoo, on a 1 tick chart I was able to see it as well - we'll take a look into.

[ScoobyStoo]

Thanks Bertrand.

Those order book events timestamped 22:59:59.836 are valid pre-open market data messages for the selected time period so the strategy shouldn't be filtering them.

Best of luck finding the cause...

[NinjaTrader_Bertrand]

You're welcome ScoobyStoo - just wanted to let you know that we're looking into the issue and will post here again once we've completed our analysis. Thanks

[ScoobyStoo]

Hi guys,

It's been a month since the last update to this thread.

Is there likely to be a fix for the order book problem when replaying data through a custom strategy?

Thanks