Market Replay: Level 2 Order Book Data Bug

[ScoobyStoo]

ScoobyStoo,

Can you please provide the indicator you are using for the dump. Thank you.

No problem.

See attached.

[NinjaTrader_JoshP]

1. Looking at your dump file the first L2 data was an update to position 0.

Level 2 Market Data
Time : 05:00:00.765
Type : Ask
Operation : Update
Position : 0
Price : 1.4289
Volume : 5

2. I was unable to reproduce. Did you play at 500x all the way through or were you fast forwarding and jumping around?

[ScoobyStoo]

1. Looking at your dump file the first L2 data was an update to position 0.

Level 2 Market Data
Time : 05:00:00.765
Type : Ask
Operation : Update
Position : 0
Price : 1.4289
Volume : 5

Yes. My mistake, that's the first one, it is the next L2 message that relates to an update to position 3. You get my point though? There should be a set of insertions to build both sides of the book before any updates occur.

2. I was unable to reproduce. Did you play at 500x all the way through or were you fast forwarding and jumping around?

When I first came across this problem I tried varying the speed of replay (e.g. x1, x10, x50, x500) and jumping around within the replay data. The output is always consistent.

If you can't reproduce this then I'm thinking it is environmental and specific to my machine. Is it likely to be an issue with me having NT6.5 and NT7 installed on the same box? I am now only using replay data downloaded from your servers with NT7.

I am at a loss as to what to try next with this one.

Check out the attached dump file. I've just generated this by fast forwarding to mid way through thr 5th Jan 2010. This time the order book is being built correctly at the beginning of the replay but look at some of the values.

e.g.

Level 2 Market Data
Time : 05:01:08.890
Type : Ask
Operation : Insert
Position : 0
Price : 1.4421
Volume : 93


Level 2 Market Data
Time : 05:01:06.890
Type : Ask
Operation : Insert
Position : 1
Price : 1.4422
Volume : 80


Level 2 Market Data
Time : 05:01:05.890
Type : Ask
Operation : Insert
Position : 2
Price : 1.4423
Volume : 58


Level 2 Market Data
Time : 05:01:24.890
Type : Ask
Operation : Insert
Position : 3
Price : -107339.4538
Volume : 8534548


Level 2 Market Data
Time : 06:19:30.640
Type : Ask
Operation : Insert
Position : 4
Price : -107339.4538
Volume : 0

I timestamps are jumping around all over the place and you'll also notice that after the book is built there are no more L2 events fired. This seems very weird. Are there definitely no exceptions being thrown which are getting suppressed somewhere?

In my experience, when data starts to come back with wild values such as this then it is sometimes down to the wrong chunk of memory being read. Are you guys using unsafe code blocks? Have you perhaps got an errant pointer in there which is mistyped?

Let me install NT7 on my spare laptop and see if that produces different results. Will get back to you...

[NinjaTrader_JoshP]

1. Why do you feel so? The book is already populated before you access it. You are not processing what was there beforehand you are simply processing new changes to the book. If the latest change is an update then that is the event you will get.

2. Perhaps you could try a different date and a different instrument? See if that makes any difference to ensure it is not that particular set that may be bad?

[ScoobyStoo]

1. Why do you feel so? The book is already populated before you access it. You are not processing what was there beforehand you are simply processing new changes to the book. If the latest change is an update then that is the event you will get.

Yep, got that. Obviously the replay data just contains the events in the sequence they were generated at the execution venue. However, NT6.5 does something clever and always rebuilds the book from scratch when you jump around in the time line. Not sure how it does this but it is very useful, as otherwise it's necessary to put in some code to trap row removal events that relate to rows that haven't actually been inserted.

2. Perhaps you could try a different date and a different instrument? See if that makes any difference to ensure it is not that particular set that may be bad?

I have installed NT7 on a different machine and downloaded several chunks of data for multiple instruments on multiple days. Unfortunately, I see exactly the same problem when I replay any of the data. Sometimes the L2 events starts replaying properly but then the same bizarre problems always begin to appear. I don't think the problem is down to the data. It can't all be corrupt.

If you can't reproduce the issue then let's try to figure out what's different between the setup on your machine and the setup on my two boxes. Both of mine have NT6.5 also installed. Could this be a problem?

[NinjaTrader_JoshP]

6.5 would not be a problem. What I suggest is you try on a ES replay. I tried on a 6E replay and did forward something to development for clarification in the meantime. For the ES replays I did not see any issues though.

[ScoobyStoo]

I'm afraid I see the same problem with ES data. I have examined the following contracts (6A, 6B, 6C, 6E, 6J, 6S, CL, GC, ES, NQ, YM, TF) and seen the problem exhibited in all of them.

I'm going to leave this one with you guys now. If you need me to do any further tests on my setup let me know. I'm sure your engineering dept will get to the bottom of it.

Thanks

[NinjaTrader_JoshP]

Please try using a date after January. Development has informed me that they made changes to the recording after January which would definitely impact the replay files you have downloaded before then.

[ScoobyStoo]

Please try using a date after January. Development has informed me that they made changes to the recording after January which would definitely impact the replay files you have downloaded before then.

Now we're cooking. The data I have tested from Feb works fine. Isn't that just typical. I spent hours downloading random samples from Oct, Nov, Dec 2009 and Jan 2010. Why I didn't try Feb 2010 I don't know...

One thing I should have asked previously is what source you use for your downloadable futures market replay data? Is it unfiltered? I have always assumed the downloadable data is recorded from an unfiltered Zen Fire feed because you guys have such a tight relationship with them. Is this the case?

Is all the data prior to Feb 2010 corrupt or is it simply as case of reformatting it? I guess what I'm asking is whether you are likely to provide usable data for 2009 in the near future?

Thanks.

[NinjaTrader_Dierk]

>> Is it unfiltered?
yes

>> I guess what I'm asking is whether you are likely to provide usable data for 2009 in the near future?
Unfortunately not, since the corrupted data is not "restorable".

[ScoobyStoo]

>> I guess what I'm asking is whether you are likely to provide usable data for 2009 in the near future?
Unfortunately not, since the corrupted data is not "restorable".

That's unfortunate. At least we've got to the bottom of this problem though.

Would be grateful if you/Josh could give me the date in January when the changes were made to the recording process so I don't attempt to download any corrupt data.

Thanks.

[NinjaTrader_Dierk]

Please stay with February. Thanks

[ScoobyStoo]

Please stay with February. Thanks

OK.

Is the data unfiltered?

[NinjaTrader_Dierk]

Please see post #25