How to Reconstruct an Order Book Using Only OnMarketDepth

[niko534]

Hello NinjaTrader Community,

I’ve been recording raw logs from NinjaTrader’s OnMarketDepth event (e.g., lines indicating "Add", "Remove", "Update" at certain positions and prices) and want to reconstruct a valid order book (DOM) from those logs alone. I’m aware NinjaTrader typically uses several events (OnOrderUpdate, OnMarketData, etc.) to get a complete picture, but I’m interested only in the OnMarketDepth approach.

Here are the main questions and uncertainties I have:

1. Shifting Logic
   • In some existing code examples, I’ve seen an array-based approach that “shifts up” the entire side when position=0 changes to a new best price.
   • Does the official NinjaTrader OnMarketDepth pipeline explicitly shift arrays behind the scenes, or is that purely a UI concept (e.g., re-sorting rows)?
   • Under which conditions should I forcibly shift levels in my code if I’m strictly using the (operation, position, price, volume) logs from OnMarketDepth?
2. Position vs. Price
   • The logs might say "Add, position=7, price=2891.25, volume=5". Do I store that in a dictionary keyed by position or do I store data keyed by price and treat position as more of a “visual rank”?
   • Which approach (array-based vs. dictionary-based) is recommended to remain consistent with how NinjaTrader’s DOM is updated in real time?
3. Add / Remove / Update
   • From the logs, I see lines like:Ask, Add, 7, 2891.25, 5
     Ask, Remove, 0, 2888, 0
     Ask, Update, 0, 2900, 15
     • How exactly does NinjaTrader handle these behind the scenes?
       • If I see "Remove" at position=0, do I zero out volume for that slot, or must I re-index subsequent positions?
       • If I see "Add" at position=7 with a new price, do I overwrite any existing price at 7 or do I shift the old price to position 8?
4. Data Stability
   • If my logs arrive in chronological order, can I assume all “Remove” operations for a level happen before a new “Add” for that same position?
   • What if the feed sends “Remove at pos=0, price=2891.25” and immediately “Add at pos=0, price=2892.00, volume=10”?
   • Which validations does NinjaTrader do internally to avoid a corrupt DOM if logs arrive slightly out of sequence?
5. Partial Fills / Volume Changes
   • Even though I’m ignoring OnOrderUpdate or OnMarketData, OnMarketDepth can reflect volume decreases (like “Update” from volume=8 to volume=3 at the same (side, pos, price)).
   • How do we handle partial volume changes in the official approach? Does NinjaTrader do anything special to track the “queue position,” or is it purely (new volume, old volume) differences?
6. Recommended “Best Practice”
   • Ultimately, I want a stable approach:
     1. Minimal resets—avoid discarding the entire book if a single line references a surprising price.
     2. Accurate shifting—only apply a “shift up/down” if that’s truly how NinjaTrader organizes the DOM.
     3. Valid final snapshot—my code should end up with a list/array/dictionary that matches the official DOM if I replay all the logs from OnMarketDepth.
7. Missing Classes
   • The partial code I’ve seen references classes like MarketDepth, LadderRow, or PriceRow. Could you clarify how they work with OnMarketDepth behind the scenes?
   • Which of these classes is primarily responsible for the re-sorting or shifting?
8. Sample or Reference Implementation
   • Are there official (or community) examples demonstrating how to store and update the DOM purely from OnMarketDepth logs?
   • Any “exact” logic on how NinjaTrader merges 'Add', 'Remove', and 'Update' lines for the same (side, pos)?

I’d greatly appreciate any guidance or clarifications from NinjaTrader staff or experienced community members on how to replicate a valid order book exclusively using the OnMarketDepth feed. Specifically, when to shift levels, how to store (pos, price, volume) consistently, and what checks to perform to keep the data stable.

Thank you in advance for your expertise and time!

[NinjaTrader_ChelseaB]

Hello niko534,

Welcome to the NinjaTrader forums!

Unfortunately, I cannot provide details on how NinjaTrader functions or is coded internally.

However, I can provide a link to the reference sample on building an order book you may find helpful.

https://developer.ninjatrader.com/docs/desktop/creating_your_own_level_ii_data_book_accessing_market_depth

"Does the official NinjaTrader OnMarketDepth pipeline explicitly shift arrays behind the scenes, or is that purely a UI concept (e.g., re-sorting rows)?"

I can't comment on how the internals of the core code works, however the SuperDOM is shifting up and down is a visual concept and not necessarily how the information is stored.
The reference sample uses a List of a custom class to store information for each level 2 row (e.Position) as new data comes in.

"Under which conditions should I forcibly shift levels in my code if I’m strictly using the (operation, position, price, volume) logs from OnMarketDepth?"

If you choose to shift values in a collection you might consider shifting these when a new price is reached.


"The logs might say "Add, position=7, price=2891.25, volume=5". Do I store that in a dictionary keyed by position or do I store data keyed by price and treat position as more of a “visual rank”?"

How you choose to store the data is up to you. The reference sample uses a List<LadderRow> collection where the keys are the e.Position level 2 rows, and the stored object is a custom LadderRow class that holds the price, volume, and market maker for that level 2 row.
I imagine a Dictionary may work as well.


"Which approach (array-based vs. dictionary-based) is recommended to remain consistent with how NinjaTrader’s DOM is updated in real time?"

The reference sample uses a List<LadderRow> collection. An array may work as well but you would need a way to map the array index to a specific e.Position.


"How exactly does NinjaTrader handle these behind the scenes?"

I can't comment on how the internals of the core code works, however in the reference sample for an add or update, the row position is added to the list if it doesn't exist, otherwise the list element is assigned a new LadderRow object.
For a remove the element is removed from the list.


"If my logs arrive in chronological order, can I assume all “Remove” operations for a level happen before a new “Add” for that same position?"

I have not seen any behavior that would suggest otherwise.


"What if the feed sends “Remove at pos=0, price=2891.25” and immediately “Add at pos=0, price=2892.00, volume=10”?"

The reference sample would remove the row then add the row.


"Which validations does NinjaTrader do internally to avoid a corrupt DOM if logs arrive slightly out of sequence?"

Unfortunately, I cannot provide this information.


"How do we handle partial volume changes in the official approach? Does NinjaTrader do anything special to track the “queue position,” or is it purely (new volume, old volume) differences?"

That I know of there isn't partial volume. Volume has to be 1 or greater for an order. The volume is number of contracts for all orders working at that level.
Contract volume is either removed or added. I'm not aware anything partial with this.
(Note, this volume is not filled orders, this volume is working orders)


"The partial code I’ve seen references classes like MarketDepth, LadderRow, or PriceRow. Could you clarify how they work with OnMarketDepth behind the scenes?"

Unfortunately, I cannot provide this information.


"Which of these classes is primarily responsible for the re-sorting or shifting?"

Unfortunately, I cannot provide this information.


"Are there official (or community) examples demonstrating how to store and update the DOM purely from OnMarketDepth logs?"

I've provided this above in this post.


"Any “exact” logic on how NinjaTrader merges 'Add', 'Remove', and 'Update' lines for the same (side, pos)?"

This would in the reference sample linked at the top of this post.

[niko534]

Hey Chelsea,

First, thanks for the quick response, your information has been invaluable.

Thank you for sharing the SampleLevel2Book indicator as a reference for building a custom Level II data book with OnMarketDepth().
I’ve reviewed the code and compared it to other possible approaches for managing the order book (DOM).

I’d like to clarify a few points and hopefully get advice on the best method for my scenario.

I'll start with some context. Given the raw market depth event logs I've recorded from OnMarketDepth(), I aim to create a new dataset that will represent a snapshot of the order book for each log from the raw data (full order book both bid/ask at each log).

This new dataset will later be used for ML and building features, so data integrity and granularity are important.

Without getting into NinjaTrader implementations, as I see it there are 2 valid approaches to take:

(A) Follow the Sample Code Exactly
(B) Store Data by “Price -> Volume”

(A) The SampleLevel2Book demonstrates how to build a Level II order book using OnMarketDepth events, storing bid and ask data in two List<LadderRow> collections keyed by e.Position (the ephemeral rank of the depth level). It handles Add, Update, and Remove operations to maintain the DOM but relies on position-based indexing, which mimics the visual ranking rather than the price-based organization as you said.

(B) Organizing the order book using price as the key, maintaining a Dictionary for direct price-based lookups, independent of position-based indexing.

My Concern:
Basically, my main concern is how do I ensure my order book remains accurate and logical?
Given that the order book at its core is simply a map of price and volume I can't find any benefits in using "e.Position" other than quickly accessing the "Best Bid/Ask", which can be achieved using min/max on the price-based approach. I've tried thinking of a scenario where the position is critical to driving some insights or keeping integrity, but couldn't find any reason to use it as the price itself can be used to drive the position while avoiding shifting/reconstructing the data structure (List) that hold the LadderRows.

Question:
Given my focus on data integrity, granularity, and ML readiness, which approach would you recommend for building the order book? Are there any additional edge cases or best practices I should consider to avoid corrupting the DOM when reconstructing it from raw logs?

Any guidance or insights on choosing the right path would be greatly appreciated!

[Skifree]

Chelsea,

Is this indicator UI coded using features not available to us? I've noticed the performance outshines my attempts to do something similar with Draw.Rectangle

[NinjaTrader_ChelseaB]

Hello niko534,

The reference sample is storing information in the List<LadderRow> objects using the e.Position as the key (index) to hold the volume for all orders at that position and remove orders at that position.

You can choose store the information using the price as the key, while tracking which volume needs to be removed or added at the price level.

I would recommend you give your idea a try, and compare with the original to see if properly suits your needs.

"Given my focus on data integrity, granularity, and ML readiness, which approach would you recommend for building the order book?"

I would not expect the reference sample to have issues with this integrity or granularity and would recommend using the approach where this is a working example, while also encouraging you to explore other avenues to see if there may be a better approach.

"Basically, my main concern is how do I ensure my order book remains accurate and logical?"

How to develop the logic would be beyond the scope of our support staff, but as you develop your custom logic we can provide direction on how the tools work.

This thread will remain open for any community members that would like to provide advise as to what logic to use.


"Are there any additional edge cases or best practices I should consider to avoid corrupting the DOM when reconstructing it from raw logs?"

Using locks can be dangerous if the lock is held for anything more than a very short time.

I would recommend within the lock only assigning the updated values to storage (collection variables) and doing any other work with the data outside of the lock.



Skifree,

"Is this indicator UI coded using features not available to us? I've noticed the performance outshines my attempts to do something similar with Draw.Rectangle"

Drawing a lot of objects will take a high CPU and memory resource demand.

From the documentation:
"Using DrawObjects vs custom graphics in OnRender()
When using Draw methods, a new instance of the Draw object is created including its custom rendering and calculation logic. These methods are convenient in many situations, but can quickly introduce performance issues if used too liberally. In some situations, you may see better performance for rendering via SharpDX in OnRender()."

https://developer.ninjatrader.com/docs/desktop/ninjascript_best_practices

It would be recommended to custom render large amounts of rectangles, text, and other objects in OnRender().

Included with NinjaTrader is the SampleCustomRender() which demonstrates custom rendering in OnRender().
Also, below is a link to an example of re-using brushes in OnRenderTargetChanged().

https://forum.ninjatrader.com/forum/ninjatrader-8/indicator-development/98639-performance-optimization#post789606

The SampleLevel2Book does not do any high resource use within OnMarketDepth so does not have performance issues.