Inherent problem (incompleteness) in provided Tick data

[ttodua]

This issue is quite similar as asked here, however the answers inside them are quite irrelevant to the problem and moreover it only applies to REALTIME EVENTS, while historical functionality is completely ignored, which should have been prioritized also. I am quite surprised this was not reported earlier and taken seriously, as NT lacks this important functionality for trading.

OnMarketData emits values which is one of the three MarketDataType: Last, Bid, Ask.

protected override void OnMarketData(MarketDataEventArgs e)
{
    Print("Datetype: " +e.MarketDataType + "| Volume: " + e.Volume + " | Price:" + e.Price);
}

To again explain in layman words (for whoever it is not fully clear) Ask and Bid prices there are meant to be the level-1 data updates (the change of the best bid-ask from orderbook of that symbol) and it can change (causing MarketDataType.Bid/.Ask event in OnMarketData) anytime independently whether there happens any trade (.Last ) in market or not.
And samelike, trade might happen (.Last event) anytime, independently what happens to orderbook (has no connection with .Bid/.Ask events) . So, trades might happen and you might get any number of .Last events, without any event of .Bid/.Ask change (if the bid-ask prices stays unchanged in that market/orderbook).

So, we agree that .Last -and Level1 (.Bid/.Ask) events are totally independent events, right?

then the problem lies within .Last events itself. In any traditional/stock/etc trading, there does not exist a generic "last" trade without one more detail - the side. Whenever trade happens, either buyer buys at someone's ask price, or seller sells on someone's bid price. So, trade happens from initiator (a.k.a. TAKER) always. it's impossible (unless forcemajour cases) that the trade didn't have a side. Someone always have pending orders (MAKER) in orderbook, and trade happens only when another side initiates order which ends up being as MARKET Order (even if it's limit order, if the price crosses opposite bid/ask price, then it becomes MARKET order).

So, ninjatrader emits MarketDataType.Last event, however, there is no way to know (from the adapter connection) whether it was buyer or seller.
What does happen in the backgrounds of NT, when you open the chart with i.e. 1-tick timeframe , with DataType - ASK?
it fetches (from adapter connection) the trades that happens on the ASK side, right? (so, the trades which were initiated by BUYERS)... and similarly logic applies when you open BID datatype chart.

However, we (or any trader) can't reliable open a chart, where is both (BID & ASK) historical data merged. You might say that we can open .LAST based chart, but if we do so, then it's impossible to look on historical bars and say whether which of them was BID or ASK bar. You are still forced to additionally load BID & ASK dataseries (thus, ending up loading 3 dataseries from adapter connection).

The solution is simple - whenever users opens chart i.e. XYZ symbol, with tick timeframe and .Last datatype, then Adapter Connection (developer) can just add the this within RequestBars:

for (var i=0; i< records.length; i++)
{
    ...
   ...
    var tradeSide = record.XyzValue ==="ask" ? TradeSide.Ask : record.XyzValue ==="bid" ? TradeSide.Bid : TradeSide.Unknown;
    .Add(price, price, price, price, time, volume, double.MinValue, double.MinValue, tradeSide)
    // instead of just
    //.Add(price, price, price, price, time, volume, double.MinValue, double.MinValue)
   ...
   ...
}

So, it will fil ninjaTrader with the information needed, so :

1) in REALTIME ONMARKETDATA events, when you get .Last event, you will be able to identify which `TradeSide` it was (either BID, ASK or unknown), and you wont be dependent to manually calculate/store what was the last bid/ask price (moreover, due to asynchronous nature from broker connections or due to many other events, it's not fully reliable way, as BID/ASK price might change and be out of the range when the .Last event comes in)

2) more importantly, not just realtime , but in HISTORICAL(which is why I open this topic) 1-TICK chart, you can identify which tick was which TradeSide, thus you dont need to additionally load another 2 dataseries ( bid/ask) and we can have something like .TradeSide property in OnBarUpdate so we can program strategies and indicators, which can be applied to .LAST datatype charts, and could identify whether bar is TradeType.Bid or TradeType.Ask.

3) additionally to above, it will open a huge door to ONRENDER functionality, to use that data, as a this moment, it's impossible and we can only get i.e. ...bars.GetClose(idx), but we will be able to know the the side of that tick-bar too, by having i.e. bars.GetBarTradeSide(idx) [of course, it will return 'TradeSide.Unknown' for any scenario other than 1-tick chart].

Thus, people can create more advanced logic by using `1-TICK .LAST` series, from either ninjascripts or even merely using a chart for analysis.

[bltdavid]

So, ninjatrader emits MarketDataType.Last event, however, there is no way to know (from the adapter connection) whether it was buyer or seller.

Hmm, but you can infer whether it is a buyer or seller, can't you?

Watch for Last >= Ask and when that occurs call it a 'buy' event,
since obviously the price generally reflects an aggressive buyer
who is usually buying with a market order, right?

Same with the sell event, except it is opposite.

Watch for Last <= Bid and when that occurs call it a 'sell' event, since
obviously the price generally reflects the aggressive seller who is
usually selling with a market order, right?

Something like this,

double AskPrice, BidPrice;
int AskHits, BidHits;
protected override void OnMarketData(MarketDataEventArgs e)
{
    if (e.MarketDataType == MarketDataType.Ask)
        AskPrice = e.Price;

    else if (e.MarketDataType == MarketDataType.Bid)
        BidPrice = e.Price;

    else if (e.MarketDataType == MarketDataType.Last && AskPrice > BidPrice)
    {
        if (e.Price >= AskPrice)
            ++AskHits;   // -- aggressive buyer event
        else if (e.Price <= BidPrice)
            ++BidHits;   // -- aggressive seller event
    }
}

[ttodua]

bltdavid thanks for response, but i doubt, you might have looked through the specific parts of the post, but might not noticed other parts, where i explain how that answer (which i had already referenced) is almost unrelated. but if you dont have enough time to re-read, i'll give it in two words: 1) it works only for realtime, and the main emphasize in the post was historical part. 2) neither in realtime it is reliable, because adapters tend to get live updates asynchronously, and in HTF, there are many occasions (myself have observed) when the level-1 data updates and trades are feeded in an non consistent manner (i.e. getting trade earlier than level1 data change, or in opposite sometimes) which makes it impossible to correctly react with algorithms, because of false "bid/ask" detection.

actually, nt could have fixed that easier, by providing an additional property. just to ask a funny question - lets say you start your strategy and the first OnMarketData event what you get, is .Last - can you tell, which side it was, bid or ask ? : ))) but even to give up this question, historical argument is much important.

[bltdavid]

Yeah, you're right, I had no idea my quick response was almost
an exact code rehash from the thread you cited. My apologies.

[ttodua]

You are more than welcome! Your posts have always helped me, not less than NT support itself.

[NinjaTrader_Kate]

Hello ttodua,

Thank you for your posts.

If I'm understanding correctly, you're wanting to identify buys and sells with historical data. Keep in mind that Bid/Ask do not mean completed trades. MarketDataType.Last is a completed tick. To determine if it was a buy/sell, we would compare the Last tick price against the bid/ask price.

This was demonstrated in the thread by bltdavid, however, you mention rightfully that does not work with historical.

BuySellVolume can be used with Tick Replay and that works with historical and would be a good starting point to look at.

We also have a BuySellVolumeOneTick example that shows how to tally buys and sells from 1 tick data series. (This uses BarsArray[1].GetBid()/GetAsk() to get the best ask and bid from the Last tick)

I'm attaching that BuySellVolumeOneTick script below that may be helpful for you.

Please let us know if we may be of further assistance to you.

[ttodua]

NinjaTrader_Kate
thanks for answer, but I don't think your mentioned workarounds are solutions. I'll give arguments:
1) Tick replay - we should exclude that scenario, it's quite far from topic. it's disconnecting from adapter, , jumping into other simulation engine and etcetera. Why such complexity?
2) I say, the data property should hold the value (if it was either bid or ask), and BuySellVolume Indicator is user-side implementation to recognize it. (and quite complex script to get the simplest information, which is native information of a trade)
3) as mentioned, BUySellVolume Indicator is complex (extra calling 1tick data series itself). Users, who want to apply an indicator on top of the chart data, instead of it, they need to apply/code their indicator on top of BuySellVolume indicator, to find out whether trade was buyer or seller. again, the data is itself the container to tell, whether trade was from buyer or seller.

in my opinion, all of those proposed approaches are workarounds (not solutions) and that was my suggestion to implement an additional parameter (as we work with numerous exchanges data, with many traders and we know what we are about) and all individual tick (trade) has their flag which says whether it was seller or buyer.
NinjaTrader programmers and ecosystem could use `AddDateSeries( 1 tick) ` and then in BIP===0 they would easily access the property of bar (whether it was buyer or seller) and so, programmers could build much accurate algorithms and strategies, without "possible guessing" whether it might have been on bid or ask (and checking multiple events, like OnMarketData and etc, which themselves, have their weaknesses on their own, like synchronisation issues mentioned by me above, as they are not reliable and you have to manually store the last bid ask price and compare the '.Last' and to 'guess' whether it was made on bid or ask.... ) why all such complexity, as most of the modern engines and exchanges provide tick data with the flag about buyer /seller and you could have implemented that additional flag in ninjatrader , like proposed above...
well, just our 2 cents, and do as you wish. thanks for time.