Market Replay: Level 2 Order Book Data Bug

[ScoobyStoo]

Hi,

There is a problem with replaying L2 order book data that has been recorded using NT v6.5. Some replay files do not generate the correct L2 data when replayed.

The problem can be easily observed via either:

(a) A DynamicSuperDOM window.
(b) An API script invoking the OnMarketDepth method.

I have verified this bug on both v7.0.0.11 and v7.0.0.12. I have also tried a clean reinstall of v7.0.0.12 without any luck.

I cannot attached a zip file containing example market replay files because of the size limits imposed on uploads to this site.

Is this a known bug? I would appreciate your help in getting this sorted out.

Thanks

[NinjaTrader_JoshP]

Please send the file in for analysis to support [at] ninjatrader [dot] com. Thanks.

[NinjaTrader_Dierk]

In order to eliminate the possibility that it would be related to flaky NT6.5 (!) recorded files I suggest pulling fresh replay files from our NT7 server to see if issue persists. Thanks

[ScoobyStoo]

Please send the file in for analysis to support [at] ninjatrader [dot] com. Thanks.

Email has been sent.

[ScoobyStoo]

In order to eliminate the possibility that it would be related to flaky NT6.5 (!) recorded files I suggest pulling fresh replay files from our NT7 server to see if issue persists. Thanks

The data quality is fine. I can replay the data through NT6.5 without any problem. The problem occurs when the data is replayed through NT7. As I said, this problem is not exhibited by all the data recorded using NT6.5. The L1 data is always fine. It is just the L2 data that sometimes exhibits the problem.

[ScoobyStoo]

In order to eliminate the possibility that it would be related to flaky NT6.5 (!) recorded files I suggest pulling fresh replay files from our NT7 server to see if issue persists. Thanks

Hi Dierk,

It is my understanding that the replay files that can be downloaded from your server only include L1 data (i.e. transactions, best bid prices, best ask prices). Is this the case?

I need L2 order book data and I don't believe that is supplied.

Can you confirm?

Thanks

[NinjaTrader_Dierk]

>> I need L2 order book data and I don't believe that is supplied.
Incorrect. You would download L1 and L2 data. Again, I suggest you tried with data from the NT7 server, since there might be data incompatibilities between NT7 and NT6.5 data which we are not yet aware of.

[ScoobyStoo]

>> I need L2 order book data and I don't believe that is supplied.
Incorrect. You would download L1 and L2 data. Again, I suggest you tried with data from the NT7 server, since there might be data incompatibilities between NT7 and NT6.5 data which we are not yet aware of.

Thanks Dierk. I wasn't aware of this.

How do I go about accessing the downloaded L1 + L2 data?

I have used the Historical Data Manager to download the available 6E data for the past week. I selected the following parameters:

Start Date : 20/03/2010
End Date : 27/03/2010
Interval : Tick
Data Types : Ask, Bid, Last

As a result I now have a folder (C:\Documents and Settings\Admin\My Documents\NinjaTrader 7\db\tick\6E 06-10) which is populated with .ntd files.

However, when I try File > Connect > Market Replay Connection I get the error message 'Loginfailed: There is no recorded data to replay'.

What am I doing wrong?

[NinjaTrader_Dierk]

Please try File->Utilities->Download replay data

[ScoobyStoo]

Please try File->Utilities->Download replay data

OK, have got it working now.

A have a few questions:

1. What is the timezone for the data timestamps?
2. How is the quality of the data? Are there any breaks?
3. What instruments do you have data for?
4. How far back does the data go?

Thanks again

[NinjaTrader_Dierk]

1) Irrelevant as the data is replayed in your timezone. The date to select in the download form is in EST timezone.
2) I don't follow.
3) Same as our historical data server. Unfortunately there is no list available.
4) http://www.ninjatrader-support2.com/...ad.php?t=24114

[NinjaTrader_Dierk]

Please let us know as you could reproduce your issue with NT7 replay files. Thanks

[ScoobyStoo]

Please let us know as you could reproduce your issue with NT7 replay files. Thanks

Hi,

I still see the problem unfortunately.

I have downloaded data from your servers for the 6EH0 contract for the date 4th Jan 2010. I have replayed this and dumped out the market data events as the replay engine raises them (see attached file).

As you can see there are 2 problems:

1. The order book is not built correctly when the market replay session is started.

e.g. the first L2 event is an update to row position 3.

2. The L2 data gets wildly out of synch towards the end of the dump with the timestamps and order book prices diverging significantly.

e.g. these 2 events occur sequentially (look at the data and timestamps)

Level 1 Market Data
Time : 06:03:20.0
Type : Ask
Price : 1.4302
Volume : 16


Level 2 Market Data
Time : 05:15:28.765
Type : Ask
Operation : Update
Position : 0
Price : 1.4293
Volume : 6

[ScoobyStoo]

I've downloaded some more data and am seeing some truly bizarre results coming out. Things such as negative prices...(e.g. -0.0008)

I suspect that for performance reasons the market replay engine manipulates data at a binary level (using byte arrays etc). Strikes me that this problem is likely to be down to something simple such as:

1. If you are using unsafe code in the market replay engine then some code somewhere is addressing the wrong piece of memory.

2. The code that parses the market replay file is making a mistake in populating the in-memory data store.

Anyway, let me know if you need any further data from me to help fix this one.

Thanks

[NinjaTrader_JoshP]

ScoobyStoo,

Can you please provide the indicator you are using for the dump. Thank you.