OnMarketData vs. OnBarUpdate Price Level Question

[NinjaTrader_PatrickH]

Hello iantg,

Thank you for your patience.

We got further clarity on this item from development and you were correct.

GetCurrentAsk() and GetCurrentBid() can peak ahead to whatever the last updated price was for the instrument. Due to the multi-threaded nature of NinjaTrader 8 there is the possibility for the GetCurrentAsk() or GetCurrentBid() to see ahead of the instrument thread that updates MarketDataType events such as MarketDataType.Ask.

Please let me know if you have any questions.

[iantg]

Hi Patrick,

Thanks for getting back to me on this. from my testing GetCurrentAsk() and GetCurrentBid() will typically update first, so you comments from develop align with my experience on this point. So this leads me to my real questions from post 14 now. I will try to frame this into 2 very concise questions.

1. When the price does change, we will often see GetCurrentAsk() and GetCurrentBid() update first to the correct new price level. Then it can be expected that the prices coming from MarketDataType.Last will have a lag because this feed is reporting transacted volume and often won't change to the GetCurrentAsk() and GetCurrentBid() until around 50% of the level is complete. I would expect that this feed has a lag but when it does change to the correct new price it stays on this price level until the next price level is created and then it follows the same process (updating with a lag, but once the price changes it holds on that price). What I observe however is not this behavior. I see the lag in changing to the right price level as defined by the GetCurrentAsk() and GetCurrentBid(), but once the price hits this level it will flip back and forth between 2 or 3 different prices. This is the issue. If you look at the excel spreadsheet enclosed is post 14, I have this illustrated. Why does the price flip back and forth between multiple prices from MarketDataType.Last when we use the following logic to call the price:

if (marketDataUpdate.MarketDataType == MarketDataType.Last )

{

if (marketDataUpdate.Price <= marketDataUpdate.Bid)
{
transbid = marketDataUpdate.Price;

}




2. Question 2 is similar to question 1 but this addresses the feeds
if( marketDataUpdate.MarketDataType == MarketDataType.Ask)
if( marketDataUpdate.MarketDataType == MarketDataType.Bid)

My understanding of these feeds are that they should represent the level 1 price level for bid and ask respectively at any point in time. They provide a second perspective to GetCurrentAsk() and GetCurrentBid(), but ultimately they are attempting to show the current price levels. I understand that as your development team has confirmed, these will often lag GetCurrentAsk() and GetCurrentBid() and update second. But just as with the feed from question 1, I also see this feed flip prices back and forth multiple times during a single price level where GetCurrentAsk() and GetCurrentBid() has a single unabridged price the entire time. Again if you look at the spreadsheet from post 14, and look at the last 2 columns this shows the prices from this feed against the GetCurrentAsk() and GetCurrentBid() prices and often times we will see 3 distinct prices occuring during a single price defined by the GetCurrentAsk() and GetCurrentBid() price level. So in my use case here is an example of the problem statement: I am trying to figure out what the current bid price is, as a HF scalper, I need to know the price level as fast as possible and I need to capture it every time it changes. I can not miss a price level change because I was looking in the wrong place at the wrong feed. We have already established that GetCurrentAsk() and GetCurrentBid() will typically update the price level first.... But the level 1 data from if( marketDataUpdate.MarketDataType == MarketDataType.Ask) and
if( marketDataUpdate.MarketDataType == MarketDataType.Bid) contain 2 to 3 price levels per every 1 price level from GetCurrentAsk() and GetCurrentBid() so which version of the truth is right, and what should I be looking at? We only have two possible options here. Either GetCurrentAsk() and GetCurrentBid() is the sole version of the truth. When ran on a 1 tick data series this captures every price level change and the OnMarketData feed using MarketDataType.BId or MarketDataType.Ask is often wrong. When it reports 2 to 3 distinct prices when GetCurrentAsk() and GetCurrentBid() only shows 1 unabridged price, this is a bug and needs to be fixed. Or option 2 is that OnMarketData is actually capturing real price levels and GetCurrentAsk() and GetCurrentBid() is actually missing these. So in my use case I would be blind to real price level changes if I only use these.

So which is? Which scenario accurately describes what is going on?

I appreciate your looking into this for me.

Thanks,

Ian

[NinjaTrader_PatrickH]

Hello iantg,

Thank you for your response.

GetCurrentAsk() and GetCurrentBid() are intended to be used in OnBarUpdate() to pull a snapshot of the data at that time from the core process.

OnMarketData() is the raw Level 1 event. Do not use GetCurrentAsk() or GetCurrentBid() within OnMarketData().

The price "flipping" back and forth is quite literally a change in the price in OnMarketData().

If you need the snapshot of the current bid or ask use GetCurrentBid() or GetCurrentAsk() inside of OnBarUpdate().

Please let me know if you have any questions.

[iantg]

HI Patrick,

Thanks for helping me understand everything. I have always been of the impression that calling GetCurrentAsk() and GetCurrentBid() from OnBarUpdate with a 1 tick time series would be the most accurate approach. So basically I am just wanting to validate this assumption.

The conflict that I see in the data, can be illustrated in this example.


1. I have a 1 tick time series and for every trade that occurs I call GetCurrentBid() from the on bar update method. On every trade that occurs I run a print statement to get the price.

2. At the same time I am running the level 1 bid prices from the OnMarketData / MarketDataType.Bid method. I am also printing every price that occurs from this method.

After 100 trades and 100 prints from the GetCurrentBid() method I observe the following distinct prices occurring at the same time from the two methods.

From OnBarUpdate GetCurrentBid() the bid price moves like this.

2000
2000
2000
2000
2000
2000.25
2000.25
2000.25

From OnMarketData calling the MarketDataType.Bid the price moves like this.

1999.75
2000
2000
2000.25
2000.25
2000
2000
2000.25

So on the 4th update in this example the OnBarUpdate / GetCurrentBid() tells me the price is at 2000 and the OnMarketData / MarketDataType.Bid tells me the price is at 2000.25. So which one is right?

I see this type of example often, where the GetCurrentBid() called from OnBarUpdate on a 1 tick time series does not pick up this type of small price movement. I don't know if it is missing real price levels, or if OnMarketData / MarketDataType.Bid is picking up noise that isn't real.

So what I would like to know is in this scenario, which data feed is hitting the real price and which one is off?

Scenario 1: If the answer is that GetCurrentBid() called from OnBarUpdate on a 1 tick time series is right, then I will be fine. Because we already established that this method updates the price ahead of OnMarketData often. So using this exclusively would be ideal.

Scenario 2: However if we establish that the OnMarketData / MarketDataType.Bid is hitting real price levels and they are not being registered with OnBarUpdate / GetCurrentBid() even with a 1 tick time series then we have huge problem. Because a user will need to use both feeds in different ways to always get to the right price level in the fastest possible way.

So solve for Scenario 2, I would have to track any change from OnBarUpdate / GetCurrentBid() first since this often leads... but then I would have to also track if and when any change occurs to the OnMarketData feed to see additional price levels. But because these two feeds are so out of sync, it would be very challenging to program any solution to know what the price level was at any point in time. If you see the excel spreadsheet I enclosed on post 14, you will see how different the prices derived from these two are at any moment. I don't think any programmer should have to ever build a workaround to just define the right price. So if scenario 2 is in play, then we need a solution from NT to give us a single source of the truth.

If I was to try to solve for scenario 2 here is what I would be having to code and logically solve for.

1. The first real price is 2000 not 1999.75, because 2000 hits first from the OnBarUpdate method.
2. The 4th update, the price actually moves to 2000.25, and OnBarUpdate misses this for some reason.
3. The 5th update is also really on 2000.25, and again OnBarUpdate misses this.
4. The 6th update is tricky, because the OnBarUpdate is once again leading and moves up to 2000.25. But at the same time the OnMarketData feed moves down to 2000.
5. Finally at the very end, but feeds get in sync and are on 2000.25

Try to figure out programming logic to solve for this.... assuming scenario 2 is the one that is accurate. This would be a nightmare and is what I am trying to avoid at all cost.


Hopefully you can understand what exactly I am trying to solve for and understand and what the implications of this would mean. So in the context of my question and examples what do you think is going on in the data? Hopefully scenario 1 correctly describes the data and we just have a bug in the OnMarketData feed.

Ian

[NinjaTrader_Brett]

GetCurrentAsk() and GetCurrentBid() from OnBarUpdate with a 1 tick time series would be the most accurate approach

This is NOT the most "accurate", its just different use cases.

The intent of GetCurrentAsk() and GetCurrentBid() is if you don't want to process the RAW L1 feed. You then could take a peak at the current bid or current ask price at the time you call GetCurrentAsk(). If you wire up a timer to call GetCurrentAsk() every 1 second, you would then access the current ask price every 1 second. If you call GetCurrentAsk() during OnBarUpdate on 'last' you'll peak at the current bid/ask as a last event is processed.

Sounds like what you need/want is to process the entire RAW L1 feed. You want to use OnMarketData for that. Which is the 'true' sequence of bid/ask/last events and all events processed as received.

[iantg]

Hi Brett,

Thank you for offering to help me understand everything.

For my specific use case I need to see every price level change as quick as possible. I am operating in a HF / scalping capacity so every tick matters to me. I am on a rig with < 10 MS execution speed and paying to co-locate to help me get my orders out as quick as possible (at the retail level anyway) but for all this effort to optimize my speed I am struggling with the NT data structure a bit. Here is my issue in a nutshell.

After testing the live data-feed from CQG / Continuum with live trades for a few months now I have come to see that in most cases the GetCurrentBid() / GetCurrentAsk() prices called from OnBarUpdate using a 1 tick data series captures the price level changes ahead of the OnMarketData / MarketDataType.Bid / MarketDataType.Ask prices by anywhere from 10 MS to 500 MS. But alternatively, the level 1 feed from OnMarketData / MarketDataType.Bid / MarketDataType.Ask is able to capture price level changes that are never picked up by the OnBarUpdate GetCurrentBid() / GetCurrentAsk(). So I am in kind of a no win situation.

If I go exclusively with the OnBarUpdate / GetCurrentBid() / GetCurrentAsk() prices called on every single tick, this will capture price level changes the fastest but only when it changes in the first place. There seems to be price levels that occur with the level 1 feed in OnMarketData that are completely missed here.

But by contrast if I go with the OnMarketData level 1 feed exclusively, then on half of the price level changes I will be experiencing a lag of 10 ms to 500 MS as the OnBarUpdate / GetCurrentBid() / GetCurrentAsk() methods capture the price level change events first. I may pick up more price levels this way, but for the price levels that overlap between the two methods, this feed appears to constantly update slower.

So do you see the issue? When your Q/ A team put together the test script, and I ran it, it confirmed this issue (That OnBarUpdate GetCurrentBid() / GetCurrentAsk() will update the price level ahead of the level 1 feed from OnMarketData. I can't afford to lose a few hundred milliseconds on every other price level.

So what do you recommend I do here? Are there any work arounds, or settings that I can change that would either give me visibility to every price level change from OnBarUpdate, or speed up the OnMarketData feed to match the GetCurrentBid() / GetCurrentAsk() called from OnBarUpdate with a 1 tick data series?

What I am looking for is the fastest possible way to capture every price level change from a single source. But after researching this, it appears that in some cases OnMarketData is the route I will need to take, but in other cases OnBarUpdate is the route I will need to take.

Let me know what you think I should do with my specific use case. If there is not a single structure within NT that will accommodate this requirement currently, may I recommend a feature request to address this?

Thanks,

Ian

[NinjaTrader_Brett]

I suspect what your "getting a peek at" is the impact of NinjaTrader internal design. Where there currently does exist an internal processing queue that will distribute market data to all consumers across all threads. This logic does introduce some overhead but allows NinjaTrader to distribute processing across multiple CPU cores. By using GetCurrentAsk/GetCurrentBid your 'sidestepping' that internal processing overhead but its only a peak at the current price not the full L1 feed that you need.

Currently if you want the full feed there is no faster method in NinjaTrader then processing OnMarketData.

A word of warning though Shaving those last few milliseconds off and trying to reduce latency at these levels gets costly and complex. Since NinjaTrader internal design is a factor it is likely be a smaller percentage of your total worries. Colocating for sure handles the largest source for latency but after that each millisecond gain on handling becomes increasingly difficult often requiring:

* Writing direct to exchange
* Fiber NICS instead of 10BaseT.
* Operating System bypasses for traditional network/packet handling to reduce OS lag
* Highly specialized and streamlined application logic
* I've even heard of people bypassing CPU's and utilizing utilizing FPGA's

[iantg]

Hi Brett,

I appreciate the insight and the feedback from everyone over there that have contributed to this thread and in helping me with this query.

So I think you have more or less confirmed my concern which is that I will need to work with both the OnMarketData feed as well as the GetCurrentBid() / GetCurrentAsk() from OnBarUpdate running a 1 tick time series for my solution.

I wish I could just use the OnMarketData feed 100%, but honestly it is too laggy in a lot of cases. And I would ideally love to be able to use the OnMarketData feed for all the volume update but it's too slow to be useful for what I am doing. For example when the OnBarUpdate runs and I get a new bid price and bid volume using GetCurrentBidVolume() I might see a volume of 100 for example... at the same time the OnMarketData feed for that price level might have the volume at 500.... And it can literally take a couple hundred updates before it moves down to the volume of 100 that was already reported by the OnBarUpdate. I suppose your explanation about overhead and bypassing the overhead by peaking at the GetCurrentAsk / GetCurrentBid explains that.

So I don't think the OnMarketData will be fast enough for me to use it to track volumes, so at this point I am only looking to track the price levels. But the issue that still persists is that using only OnBarUpdate / GetCurrentBid() / GetCurrentAsk(), it appears that I am missing price levels that occur from within the OnMarketData feed. These seem to be real price levels that are somehow missed... even calling this update every single tick. Can you confirm this? For example in post 14 I have an excel spreadsheet where I pulled in distinct price levels from both methods and you can see how multiple price levels are showing up from the OnMarketData feed during a single price level from the OnBarUpdate / GetCurrentBid() / GetCurrentAsk() feed. So this is the first item I just need to clear up. Are these real price levels or noise, and if they are real levels, why do I not see them on the OnBarUpdate / GetCurrentBid() / GetCurrentAsk() feed running a 1 tick time series? Is there anything else I can do to see these price levels from the OnBarUpdate / GetCurrentBid() / GetCurrentAsk() feed or is this just a no go.

If the answer to above is there is nothing I can do to capture every price level from the OnBarUpdate / GetCurrentBid() / GetCurrentAsk() feed, then I have to build a workaround. Here is about all I can think of.

1. If the price level ever changes from the OnBarUpdate / GetCurrentBid() / GetCurrentAsk() feed, I will use it as the live price.
2. Once the OnMarketData and OnBarUpdate methods show the same price, if the price level changes from the OnMarketData feed I will use it. I will continue to use it exclusively until the OnBarUpdate method updates a new price level and then I will switch back to it.

So that would be my workaround... what do you think?

Ian

[NinjaTrader_PatrickH]

Hello Ian,

Thank you for your response.

The fact is still that the GetCurrentAsk() or GetCurrentBid() is grabbing the price at the moment of the OnBarUpdate() call. This means when that Last Price for that bar type has updated it is calling the Bid or Ask at that moment.

OnMarketData() is the sequence of events for the market data as it occurs. So you will likely see this "raw" market data update to different levels that the OnBarUpdate() for the Last Price did not see.

Your workaround is still built on the idea that the OnBarUpdate() call is going to occur for each "raw" market data event when it will only update for the Last Price bar event. I would suggest at the very least testing your workaround to get a better idea of this.

Please let me know if you have any questions.

[iantg]

Patrick,

Thanks for the explanation. I will of course have to build and test extensively before I integrate this into my live trading. But at this point it is clear to me that based on what you and Brett have shared, I will need a solution that utilizes both the OnMarketData feed as well as the OnBarUpdate feed. The issue, is that either feed can at different moments provide the next new price level while the other feed lags behind.

I would suggest perhaps NT could try to build a new feed that only captures the price level changes. I think the resting volume and transacted volume takes up too much bandwidth and bottlenecks the OnMarketData feed at times. For example there are cases where the OnBarUpdate feed can front run OnMarketData by as much as 1 to 2 seconds. For anyone in the scalping world, they will get killed over time by this lag. They will need a work around similar to what I am building. I am not suggesting that you change the OnMarketData feed, I know that we all benefit from the volume data being delivered, but this alone is not an optimized speed solution for knowing the price level.

So in light of everything, and speaking as a guy in the high speed scalping space pointing out this issue, do you think you could design a new feed, or some other type of enhancement just to deliver the price level changes faster?

Thanks,

Ian

[NinjaTrader_PatrickH]

Hello iantg,

Thank you for your response.

I will forward a request for such High Frequency functions, however I cannot guarantee any implementation of this and extreme High Frequency Trading based features and functionality are not a focus at this time.

Please let me know if you have any questions.

[NinjaTrader_PatrickH]

Hello iantg,

Thank you for your patience.

This request has been assigned the internal tracking id of SFT-3279.

Please let me know if you have any questions.

[neo302]

This is a very useful thread. If this is still the behavior and the only options available, it seems the possible lag and other key findings should be called out in the notes of the documentation.

[SamSamIAM]

is there any updates on this - where can i find the link for tracking id SFT-3279.

[NinjaTrader_Kennedy]

Hello SamSamIAM,

Thank you for your post!

When a feature request is implemented, you'll find a description of the new feature in the release notes:

• https://ninjatrader.com/support/help...ease_notes.htm

Please let me know if further assistance is needed.