Tick Replay

[nedlloyd]

In NT 8.0.0.9, I would like Tick Replay to be on during historical data, but off when real-time (so that I get Market Data Types Bid and Ask as well as Last). I am brand new here and I am a little fuzzy about what data I am receiving in these states, so please give as much detail as you like. I believe I am getting only Last data when Tick Replay is on (regardless of state). I have found some discrepancies when looking at NTs Volume Profile indicator in Tick Replay or No Tick Replay. It seems the Bid and Ask volume decision can be different in the two modes. I have seen orders coming in where the bid and ask price were the same, so the order could be classified incorrectly (should never happen). I have an NT Continuum feed. I only looked into this because it seemed odd that the Volume Profile indicator would use two different ways for deciding if the order was bid or ask based on the state of Tick Replay. Why? Thanks for your help.

[NinjaTrader_ZacharyG]

Hello nedlloyd,

Thank you for writing in.

Tick Replay is a property that can be optionally enabled on NinjaScript indicators and strategies which will ensure that the market data (bid/ask/last) that went into building a bar is loaded in the exact sequence of market data events.

Having Tick Replay enabled on your chart should not be affecting your live data.

I would suggest taking a look at the following two links below for further information about Tick Replay and utilizing it in your scripts:

https://ninjatrader.com/support/helpGuides/nt8/en-us/?tick_replay.htm

https://ninjatrader.com/support/helpGuides/nt8/en-us/?developing_for__tick_replay.htm

Tick Replay events only occur on the "Last" market data type, so to check for ask and bid...

protected override void OnMarketData(MarketDataEventArgs marketDataUpdate)
{
	if(marketDataUpdate.MarketDataType == MarketDataType.Last)
	{
		Print(marketDataUpdate.Ask);
		Print(marketDataUpdate.Bid);				
	}
}

Can you please describe the discrepancy that you see between the chart with Tick Replay on vs. without Tick Replay?

As the volume profile indicator works on real-time data, the historical bars on your Tick Replay chart are essentially "real-time" and builds the volume profile indicator output. With Tick Replay off, it is only running on that current real-time data.

[nedlloyd]

Thank you for your response. If I run the following code:

       protected override void OnMarketData(MarketDataEventArgs marketDataUpdate)
        {
            if (marketDataUpdate.MarketDataType == MarketDataType.Last)
            {
                if (marketDataUpdate.Bid == marketDataUpdate.Ask)
                    Print(marketDataUpdate.ToString());
            }
        }

The output contains many of these:

instrument='CL 04-16 Nymex' type=Last price=31.92 volume=1 time=2/24/2016 12:04:11 PM bid=31.92 ask=31.92 isReset=False

Now, since Volume Profile is making a decision like this: (logic here seems weird, but I could be missing something)

           [COLOR=Blue] if (Bars.IsTickReplay)[/COLOR]
            {
                if (e.MarketDataType == MarketDataType.Last)
                {
                    if (e.Price >= e.Ask)
                        askPrice = e.Price;
                    else if (e.Price <= e.Bid)
                        bidPrice = e.Price;

                    price    = e.Price;
                    volume    = e.Volume;

                    if (!cacheDictionary.ContainsKey(price))
                        cacheDictionary.Add(price, new VolumeInfoItem());

                    volumeInfoItem = cacheDictionary[price];

                    if (price >= askPrice)
                        volumeInfoItem.up        += volume;
                    else if (price <= bidPrice)
                        volumeInfoItem.down        += volume;
                    else
                        volumeInfoItem.neutral    += volume;
                }
            }
            [COLOR=Blue]else[/COLOR]
            {
[COLOR=Blue]                if (e.MarketDataType == MarketDataType.Ask)
                {
                    askPrice = e.Price;
                    return;
                }

                if (e.MarketDataType == MarketDataType.Bid)
                {
                    bidPrice = e.Price;
                    return;
                }[/COLOR]

                if (e.MarketDataType != MarketDataType.Last || ChartControl == null || askPrice == 0 || bidPrice == 0)
                    return;

                if (Bars != null && !SessionIterator.IsInSession(Core.Globals.Now, true, true))
                    return;

                price    = e.Price;
                volume    = e.Volume;

                if (!cacheDictionary.ContainsKey(price))
                    cacheDictionary.Add(price, new VolumeInfoItem());

                volumeInfoItem = cacheDictionary[price];

                if (price >= askPrice)
                    volumeInfoItem.up        += volume;
                else if (price <= bidPrice)
                    volumeInfoItem.down        += volume;
                else
                    volumeInfoItem.neutral    += volume;

            }

How do we know (because ask == bid == price) if somebody bought or sold? It is like an order was executed and then the bid or ask price moved before the data was sent. But, the question is why does the Volume Profile code parse the data differently based on the state of Tick Replay? In the 'else' from above , bid and ask price are modified as Bid and Ask market data updates come in which seems to result in different outputs.

Buy Sell Pressure indicator also ONLY uses the 'Last' type market data updates, so how does it determine correctly what type of order came in (when ask == bid == price)?

When Historical data is loading and IsTickReplay is true, the data that comes in is all 'Last' data. So, if bid == ask we have no idea what the order actually was.

[NinjaTrader_ZacharyG]

Hello nedlloyd,

I'll be looking into this further and will return with my findings.

Thank you for your patience.

As a reminder, if you haven't already updated to the latest version of NinjaTrader, please note that all NinjaTrader users MUST update to NinjaTrader 7.0.1000.31 by Saturday, February 27th if they wish to continue using NinjaTrader; older versions of NinjaTrader will no longer work after the 27th. We encourage everyone to update now to avoid interruption in service.

None of your settings, work spaces, indicators, etc. will be affected if you follow the directions below.

To install the update please follow these instructions:

1. CRITICAL Shut Down NinjaTrader
2. Download the update HERE
3. Double click the file once it has completed downloading and follow the installation prompts on your screen

For more information about this critical update please click HERE.

[NinjaTrader_ZacharyG]

Hello nedlloyd,

Please take a look at the example code in the Developing for Tick Replay section of the help guide for information on how to access historical bid and ask prices with Tick Replay enabled: http://ninjatrader.com/support/helpG...ick_replay.htm

protected override void OnMarketData(MarketDataEventArgs marketDataUpdate)
{
     if (marketDataUpdate.MarketDataType == MarketDataType.Last)
     {
          if (marketDataUpdate.Price >= marketDataUpdate.Ask)
               Print(marketDataUpdate.Volume + " contracts traded at asking price " + marketDataUpdate.Ask);
          else if (marketDataUpdate.Price <= marketDataUpdate.Bid)
               Print(marketDataUpdate.Volume + " contracts traded at bidding price " + marketDataUpdate.Bid);
     }
}

The Volume Profile indicator is using similar logic to figure if it's ask or bid as does the Buy Sell Pressure indicator.

[nedlloyd]

Zachary,
Your response:
"Please take a look at the example code in the Developing for Tick Replay section of the help guide for information on how to access historical bid and ask prices with Tick Replay enabled:"

I understand how to access the bid and ask prices. In fact, the code that I put in the reply included ACCESSING the bid and ask prices (marketDataUpdate.Bid == marketDataUpdate.Ask)). I believe the DATA is wrong. What I am trying to determine is how to resolve the ambiguities in the data when Bid price == Ask Price == Price. In my previous reply I posted this code:

      protected override void OnMarketData(MarketDataEventArgs marketDataUpdate)  
      {              
         if (marketDataUpdate.MarketDataType == MarketDataType.Last)             
         {                 
               if (marketDataUpdate.Bid == marketDataUpdate.Ask)   
                   Print(marketDataUpdate.ToString());             
          }         
       }

So, if anything prints, Bid == Ask. Now, from your code:

protected override void OnMarketData(MarketDataEventArgs marketDataUpdate) 
{      
    if (marketDataUpdate.MarketDataType == MarketDataType.Last)      
    {           
        if (marketDataUpdate.Price >= marketDataUpdate.Ask)                      
            Print(marketDataUpdate.Volume + " contracts traded at asking price " +marketDataUpdate.Ask);           
        else if (marketDataUpdate.Price <= marketDataUpdate.Bid)                    
            Print(marketDataUpdate.Volume + " contracts traded at bidding price " + marketDataUpdate.Bid);      
    } 
}

If Price == Bid == Ask:
(example output from my code)
instrument='CL 04-16 Nymex' type=Last price=31.92 volume=1 time=2/24/2016 12:04:11 PM bid=31.92 ask=31.92 isReset=False

then the volume will always be classified as Ask volume. But, there is absolutely no way to tell if the volume was truly Ask or Bid because the Ask and Bid price are the same. The BEST we could do is count it as neither Ask nor Bid, which makes no sense (the contract was either bought or sold).

Run Volume Profile on the same symbol over the same day in two configurations. 1) Using Playback Connection on Market Replay data with Tick Replay off. 2) Then, do it with Tick Replay on and you will get different outputs. The total volume will be the same, but the Bid and Ask volume will be different.

This is crucial. It effectively renders Tick Replay useless because we cannot be sure that the volume is accurately classified as Bid or Ask.

[NinjaTrader_ZacharyG]

Hello nedlloyd,

Thank you for the additional information.

I'm looking into this further and will return with my findings.

[NinjaTrader_ZacharyG]

Hello nedlloyd,

It would not be expected for the bid and ask to be at the same price.

I have enabled Tick Replay on a CL 04-16 chart in Market Replay and created a a simple script that will only print the historic calls of OnMarketData(). This was tested between February 23, 2016 22:00 to February 24, 2016 21:45 Mountain Time.

I was unable to find an instance of a matching ask and bid price with the script.

protected override void OnMarketData(MarketDataEventArgs m)
		{
			if (State == State.Realtime) return;
			if (m.MarketDataType == MarketDataType.Last)
			{
				if (m.Bid == m.Ask)
					Print(m.Time + " match " + m.Bid + " " + m.Ask);
				else
					Print(m.Time + " no match");
			}
		}

Can you please delete your Market Replay data and download again to see if this still occurs?

[nedlloyd]

Zachary,

We have been doing this for three weeks now. I would really just like to solve the problem. Do you think that I manually created the output in a previous post where the bid and ask are equal? Are you telling me that my data feed is bad? It is a Ninja feed. Are you telling me the historical data was downloaded incorrectly? Is there more than one way? Are you telling me the code is wrong? I ran your code, except I commented out the else if there is no match:

        protected override void OnMarketData(MarketDataEventArgs m)
        {
            if (State == State.Realtime) return;
            if (m.MarketDataType == MarketDataType.Last)
            {
                if (m.Bid == m.Ask)
                    Print(m.Time + " match " + m.Bid + " " + m.Ask);
                // else
                    // Print(m.Time + " no match");
            }
        }

For some reason, I was having trouble getting the Playback Connection to work on CL. It keeps hanging when I put the indicator on the chart (no error is thrown in the logs). So, I just connected to NT Continuum with Tick Replay on and loaded 30 days of CL 04-16 (this is really what I want to be correct anyway, because this is the data I will be actually trading on - not the Playback Connection). Do I have to have Market Replay data downloaded for this to work? I downloaded (again) the Market Replay data from 2/15 - 2/26. The following is a portion of the output:

2/23/2016 9:09:08 AM match 32.9 32.9
2/23/2016 9:12:17 AM match 32.81 32.81
2/23/2016 9:15:42 AM match 32.75 32.75
2/23/2016 9:15:42 AM match 32.75 32.75
2/23/2016 9:47:20 AM match 32.57 32.57
2/23/2016 11:05:09 AM match 31.78 31.78
2/23/2016 11:27:22 AM match 31.75 31.75
2/23/2016 11:31:07 AM match 31.76 31.76
2/23/2016 11:34:54 AM match 31.79 31.79
2/23/2016 1:58:54 PM match 31.83 31.83
2/23/2016 2:00:42 PM match 31.9 31.9
2/23/2016 2:27:56 PM match 31.99 31.99
2/23/2016 3:11:22 PM match 31.83 31.83
2/23/2016 4:54:48 PM match 31.34 31.34
2/23/2016 6:00:00 PM match 31.28 31.28
2/24/2016 10:30:08 AM match 31.23 31.23
2/24/2016 10:41:31 AM match 31.53 31.53
2/24/2016 10:43:40 AM match 31.4 31.4
2/24/2016 10:47:49 AM match 31.2 31.2
2/24/2016 10:47:59 AM match 31.22 31.22
2/24/2016 11:36:50 AM match 31.04 31.04
2/24/2016 12:04:11 PM match 31.92 31.92
2/24/2016 12:34:51 PM match 31.88 31.88
2/24/2016 1:58:05 PM match 31.91 31.91
2/24/2016 2:40:47 PM match 32.23 32.23
2/24/2016 3:15:12 PM match 32.18 32.18
2/24/2016 4:48:30 PM match 32.2 32.2
2/24/2016 6:00:00 PM match 32.19 32.19
2/24/2016 6:59:29 PM match 32.2 32.2
2/24/2016 8:34:15 PM match 32.05 32.05


So, where am I going wrong, if you cannot produce this output? Can anybody produce this output, or am I the only one? If I am the only, how in the world is that happening?

[NinjaTrader_ZacharyG]

Hello nedlloyd,

Thank you for your patience.

I have consulted with the product management team. This behavior can occur from time to time and is expected.

What you do with the matching bid and ask ticks would be up to you. As it cannot be proven if it is a bid or ask, then perhaps the best thing to do is ignore these ticks.

With the volume profile indicator, the ticks would be assumed as ask due to the logic.

[nedlloyd]

Zachary,
Thank you for all of your help. It is really strange that the answer would be "this behavior is expected." It is expected to not know if the contracts were bought or sold? As I first stated, I have not seen this problem when the bid and ask market data types are being used to update the bid and ask price (ie, no Tick Replay). Can there not be an extra field in the market data update that tells us if the contracts were bought and sold? Surely at some point in time somebody (or some machine) knew! Or, and I understand that this would require huge memory, Ninja could save the bid and ask market data types as well as the last types. I mean, why have Tick Replay at all if we are uncertain of the accuracy? What if a 500 contract order goes through and we cannot determine if it is bid or ask?

Again, Thanks.

[NinjaTrader_ZacharyG]

Hello nedlloyd,

Tick Replay only replays the Last market data event and only stores the best inside bid/ask price at the time of the last trade event.

Historical bid/ask market data events (such as bid/ask volume) do not replay during Tick Replay. To obtain those values, you need to use a historical bid/ask series.

The Developing for Tick Replay section has recently been updated and provides much more information about how to utilize Tick Replay in a script: http://ninjatrader.com/support/helpG...ick_replay.htm

[quicktrick]

Hello NinjaTeam,

Could you please clarify real time market data?

If I download historical data using 'Tick' Intervals and 'Last' Data Type and then export them with the same parameters, I get the data in the following format:

yyyyMMdd HHmmss fffffff;last_price;bid_price;ask_price;volume

(the timestamp is in UTC). My data feed is Continuum.

1. The question is:

If I use OnMarketData event handler in real time, how can I get exactly the same data sequence as above? I mean, do I need to filter out any real time data and what conditions I have to check?

In the Developing for Tick Replay section of NT8 manual I can see:

Accessing the current best bid and ask at the time of a trade

NinjaTrader stores the best bid price and best ask price as the last trade occurs during the MarketDataType.Last event and provides it per the table below:

So, do I have to filter out any data other than matching the following condition:

if (marketDataUpdate.MarketDataType == MarketDataType.Last)
{
}

Will I get all the ticks with all their volumes in this case?

2. The second question is:

Is it possible to get the real time data timestamps in UTC, not in my local time, without any additional conversion? I guess, the data provider stores all the historical data with UTC stamps, isn't it?

[NinjaTrader_ZacharyG]

yyyyMMdd HHmmss fffffff;last_price;bid_price;ask_price;volume

Hello quicktrick,

Thank you for writing in.

As you're only wanting to obtain information from the Last MarketDataType, you'll just need to filter for that:

protected override void OnMarketData(MarketDataEventArgs m)
{
     if (m.MarketDataType == MarketDataType.Last)
          Print(m.Time.ToString("yyyyMMdd HHmmss") + ";" + m.Price + ";" + m.Bid + ";" + m.Ask + ";" + m.Volume);
}

You will get each real-time tick as this method is called on every change in level one market data.

This will output the time according to your local timezone, though. You would need to convert to UTC in your code. This can be done with the ToUniversalTime() method: https://msdn.microsoft.com/en-us/lib...=vs.110).aspx6

m.Time.ToUniversalTime().ToString("...")