Custom tick indicator receiving Market Replay Data before play is started

[michelm]

I've created a pretty simple vwap indicator that is reset for each session.

I accumulate volume and price in OnMarketData:

protected override void OnMarketData(MarketDataEventArgs marketDataUpdate)
{
            // Reset for every session
            if (sessionIterator.IsNewSession(marketDataUpdate.Time, false))
            {
                Print("NewSession: " + marketDataUpdate.Time.ToString());
                totalShares = 0;
                totalDollars = 0;
                sessionIterator.GetNextSession(marketDataUpdate.Time, false);
            }

            // Check session time
                        if (!sessionIterator.IsInSession(marketDataUpdate.Time, false, true))
            {
                return;
            }


            //Print(marketDataUpdate.Time.ToString() + totalShares.ToString());

            // TickReplay events only occur on the "Last" market data type
              if (marketDataUpdate.MarketDataType == MarketDataType.Last)
            {
                totalShares += marketDataUpdate.Volume;
                totalDollars += marketDataUpdate.Volume * marketDataUpdate.Price;
                Vwap[0] = totalDollars / totalShares;
            }

}

Works as expected when displaying an intraday chart with the instrument tick replay enabled.

However, when I attempt to use market replay, the indicator receives the full session's replay data before I press the "play" button in the market replay window.

Once I press the play button, the indicator receives the replay data again without an IsNewSession reset, and this causes the replay days vwap data to be double-counted.

So here is the setup:
Nt 8 version: 8.0.19.0

Chart Settings
Instrument: Es 06-19 (Downloaded from Ninjatrader market replay server)
Tick Replay: Enabled
Load Data based on: Days
Days To Load: 5
End date: 06-03-2019
Trading Hours: CME US Index Futures RTH
5 min candlestick chart based on last price.

Chart is as expected. Vwap is drawing, resetting on each new session.

At this point I connect to "Playback Connection" select start date of 06-04-2019. I drag the playback control to start the playback just before the 9:30 open, usually something like 9:29:04.

I have NOT pressed the play button, but my indicator OnMarketData has hit the sessionIterator.IsNewSession()(for 06-04-2019 session) condition(perhaps to be expected, not sure, but either way not a problem) and OnMarketData receives all the trades for the entire replay session from 06-04-2019 9:30:00 am until 06-04-2019 16:15:00(not expected). I'll repeat the above, the replay start time is like 9:29:04, I have not started the replay, but OnMarketData receives all the replay's session data. None of the 06-04-2019 candle sticks or vwap indicators are rendered.

Now I press play, I never get a IsNewSession() reset, and OnMarketData gets the 06-04-2019 replay data AGAIN, and the vwap is off because it's not starting out with zeroed variables once the replay is started like it should.

Why am I getting the replay data before the replay is started and then getting same data after the replay is started ?

Just hit me, perhaps the data is not coming from the replay file but it's coming from my tick history data ? I'll try to figure that out.

Thanks.

Full code:

#region Using declarations
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.ComponentModel.DataAnnotations;
using System.IO;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Windows;
using System.Windows.Input;
using System.Windows.Media;
using System.Xml.Serialization;
using NinjaTrader.Cbi;
using NinjaTrader.Gui;
using NinjaTrader.Gui.Chart;
using NinjaTrader.Gui.SuperDom;
using NinjaTrader.Gui.Tools;
using NinjaTrader.Data;
using NinjaTrader.NinjaScript;
using NinjaTrader.Core.FloatingPoint;
using NinjaTrader.NinjaScript.DrawingTools;
#endregion

//This namespace holds Indicators in this folder and is required. Do not change it. 
namespace NinjaTrader.NinjaScript.Indicators
{
    public class DailyVwap2 : Indicator
    {
        protected double totalDollars = 0;
        protected double totalShares = 0;
        protected SessionIterator sessionIterator;
        private StreamWriter sw;

        protected override void OnStateChange()
        {
            if (State == State.SetDefaults)
            {
                BarsRequiredToPlot                             = 0;
                Description                                    = @"Enter the description for your new custom Indicator here.";
                Name                                        = "DailyVwap2";
                Calculate                                    = Calculate.OnEachTick;
                IsOverlay                                    = true;
                DisplayInDataBox                            = true;
                DrawOnPricePanel                            = true;
                DrawHorizontalGridLines                        = true;
                DrawVerticalGridLines                        = true;
                PaintPriceMarkers                            = false;
                ScaleJustification                            = NinjaTrader.Gui.Chart.ScaleJustification.Right;
                //Disable this property if your indicator requires custom values that cumulate with each new market data event. 
                //See Help Guide for additional information.
                IsSuspendedWhileInactive                    = false;
                AddPlot(Brushes.DarkTurquoise, "Vwap");

            }

            else if (State == State.DataLoaded)
            {
                sessionIterator = new SessionIterator(Bars);
            }

            else if (State == State.Configure)
            {
            }
        }

        protected override void OnBarUpdate()
        {    
        }

        protected override void OnMarketData(MarketDataEventArgs marketDataUpdate)
        {


            // Resset for every session
            if (sessionIterator.IsNewSession(marketDataUpdate.Time, false))
            {
                Print("NewSession: " + marketDataUpdate.Time.ToString());
                totalShares = 0;
                totalDollars = 0;
                sessionIterator.GetNextSession(marketDataUpdate.Time, false);
            }

            // Check session time
                        if (!sessionIterator.IsInSession(marketDataUpdate.Time, false, true))
            {
                return;
            }


            //Print(marketDataUpdate.Time.ToString() + totalShares.ToString());

            // TickReplay events only occur on the "Last" market data type
              if (marketDataUpdate.MarketDataType == MarketDataType.Last)
            {
                totalShares += marketDataUpdate.Volume;
                totalDollars += marketDataUpdate.Volume * marketDataUpdate.Price;
                Vwap[0] = totalDollars / totalShares;
            }

        }

        #region Properties

        [Browsable(false)]
        [XmlIgnore]
        public Series<double> Vwap
        {
            get { return Values[0]; }
        }
        #endregion

    }
}

#region NinjaScript generated code. Neither change nor remove.

namespace NinjaTrader.NinjaScript.Indicators
{
    public partial class Indicator : NinjaTrader.Gui.NinjaScript.IndicatorRenderBase
    {
        private DailyVwap2[] cacheDailyVwap2;
        public DailyVwap2 DailyVwap2()
        {
            return DailyVwap2(Input);
        }

        public DailyVwap2 DailyVwap2(ISeries<double> input)
        {
            if (cacheDailyVwap2 != null)
                for (int idx = 0; idx < cacheDailyVwap2.Length; idx++)
                    if (cacheDailyVwap2[idx] != null &&  cacheDailyVwap2[idx].EqualsInput(input))
                        return cacheDailyVwap2[idx];
            return CacheIndicator<DailyVwap2>(new DailyVwap2(), input, ref cacheDailyVwap2);
        }
    }
}

namespace NinjaTrader.NinjaScript.MarketAnalyzerColumns
{
    public partial class MarketAnalyzerColumn : MarketAnalyzerColumnBase
    {
        public Indicators.DailyVwap2 DailyVwap2()
        {
            return indicator.DailyVwap2(Input);
        }

        public Indicators.DailyVwap2 DailyVwap2(ISeries<double> input )
        {
            return indicator.DailyVwap2(input);
        }
    }
}

namespace NinjaTrader.NinjaScript.Strategies
{
    public partial class Strategy : NinjaTrader.Gui.NinjaScript.StrategyRenderBase
    {
        public Indicators.DailyVwap2 DailyVwap2()
        {
            return indicator.DailyVwap2(Input);
        }

        public Indicators.DailyVwap2 DailyVwap2(ISeries<double> input )
        {
            return indicator.DailyVwap2(input);
        }
    }
}

#endregion

[NinjaTrader_Jim]

Hello michelm,

Thanks for your post.

When a live license key is entered in the platform, NinjaTrader will download historical data to satisfy the chart's Days To Load when connected to Playback. If Tick Replay is enabled, historical tick data will be fetched and will be used to drive OnMarketData and OnBarUpdate with Calculate.OnEachTick/Calculate.OnPriceChange operations. It would then be expected to see OnMarketData operations before the Play button is pressed.

You could use a check for if (State == State.Historical) return; to skip historical processing if you would like to test from "realtime" data or data that the Playback Connection uses after the Play button is pressed.

I could not test the script with ES 06-19 Replay data as it is no longer available on our servers. I tested with Market Replay data from last week, and the indicator appeared to work as it should. If something is still sticking out, could you give me some steps to test with more recent data? Screenshots can help as well.

I look forward to being of further assistance.

[michelm]

Hi Jim,

Thank you for getting back to me.

Using State, I was able to confirm using logging that during replay, I'm receiving the same trades twice, once from history before I actually start the replay, and again from the actual replay.

I don't have a live license. Obviously, I do have tick level history saved for the same day I'm running the market replay on.

IMHO, it does make sense to backfill replay with historical data, but I think the history backfill should stop at the replay start time, otherwise an indicator sees the same data twice and on accumulator indicators like vwap that leads to the same trades being counted twice.

I think the backfilling past the replay start time is a bug. Not an expert here, but I can't think of a reason backfilling past the replay start time and causing data duplication is useful. This is timeseries data and in this case NT is jumping around in time. To me it causes unexpected issues and off the top of my head I can't think of a generic workaround.

I added code to detect the first "realtime" tick and reset the accumulator vars to 0, but this only works in the situation where I always start the replay just before the instruments session start time(in this case 9:30 am est).

But if I started the replay at 10 am, the code would detect the first realtime tick at 10 and reset the accumulators so I would lose the first 30 minutes of data and the vwap would be incorrect for the first 30 minutes of the replay. So the correct session vwap would never be calculated.

IF I could somehow detect the replay start time before the historical backfill is started I could use your suggestion above(ie ignore historical backfill after replay start time).

To reproduce, follow the pretty specific steps above( i go through my chart set up and my replay set up in detail i think) but with an instrument you have both a replay file and historical tick data stored for on the target replay day. To see the issue happen, look at the script output window. You will see the same data twice on the replay day.

Here is the output. Replay Starts on 6-4-2019 at 9:30 am. The log show the last few backfill trades being loaded from history at the end of the 6-4-2019 replay sessions.
Then it show the first few realtime replay trades at 9:30. The first replay trade is for 4 shares and the total shares displayed in the log should be 4. But it's not, since the entire replay session has already been backfilled, it's the entire replay days session volume of 1428832 + 4. The last realtime trade results in an accumulated volume of ~2 x 1428832, which is double the actual volume.


6/4/2019 4:14:59 PM Session total shares: 1428829 Historical
6/4/2019 4:14:59 PM Session total shares: 1428832 Historical
6/4/2019 9:30:00 AM Session total shares: 1428836 Realtime
6/4/2019 9:30:00 AM Session total shares: 1428836 Realtime
......
6/4/2019 4:14:59 PM Session total shares: 2857664 Realtime

Same code but with a little more logging enabled:

#region Using declarations
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.ComponentModel.DataAnnotations;
using System.IO;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Windows;
using System.Windows.Input;
using System.Windows.Media;
using System.Xml.Serialization;
using NinjaTrader.Cbi;
using NinjaTrader.Gui;
using NinjaTrader.Gui.Chart;
using NinjaTrader.Gui.SuperDom;
using NinjaTrader.Gui.Tools;
using NinjaTrader.Data;
using NinjaTrader.NinjaScript;
using NinjaTrader.Core.FloatingPoint;
using NinjaTrader.NinjaScript.DrawingTools;
#endregion

//This namespace holds Indicators in this folder and is required. Do not change it. 
namespace NinjaTrader.NinjaScript.Indicators
{
    public class DailyVwap2 : Indicator
    {
        protected double totalDollars = 0;
        protected double totalShares = 0;
        protected SessionIterator sessionIterator;
        private StreamWriter sw;
        private bool realtimeStarted;

        protected override void OnStateChange()
        {
            if (State == State.SetDefaults)
            {
                BarsRequiredToPlot                             = 0;
                Description                                    = @"Enter the description for your new custom Indicator here.";
                Name                                        = "DailyVwap2";
                Calculate                                    = Calculate.OnEachTick;
                IsOverlay                                    = true;
                DisplayInDataBox                            = true;
                DrawOnPricePanel                            = true;
                DrawHorizontalGridLines                        = true;
                DrawVerticalGridLines                        = true;
                PaintPriceMarkers                            = false;
                ScaleJustification                            = NinjaTrader.Gui.Chart.ScaleJustification.Right;
                realtimeStarted                             = false;
                //Disable this property if your indicator requires custom values that cumulate with each new market data event. 
                //See Help Guide for additional information.
                IsSuspendedWhileInactive                    = false;
                AddPlot(Brushes.DarkTurquoise, "Vwap");

            }

            else if (State == State.DataLoaded)
            {
                sessionIterator = new SessionIterator(Bars);
            }

            else if (State == State.Configure)
            {
            }
        }

        protected override void OnBarUpdate()
        {    
//            if(Bars.IsFirstBarOfSession && IsFirstTickOfBar)
//            {
//                Print("BU new session");
//            }
        }

        protected override void OnMarketData(MarketDataEventArgs marketDataUpdate)
        {


            // Resset for every session
            if (sessionIterator.IsNewSession(marketDataUpdate.Time, false))
            {
                Print("NewSession: " + marketDataUpdate.Time.ToString());
                totalShares = 0;
                totalDollars = 0;
                sessionIterator.GetNextSession(marketDataUpdate.Time, false);
            }

            // Reset for playback history load bug
//            if (State == State.Realtime && !realtimeStarted) 
//            {
//                Print("Realtime started: " +  marketDataUpdate.Time.ToString());
//                totalShares = 0;
//                totalDollars = 0;
//                realtimeStarted = true;
//            }

            // Check session time
            if (!sessionIterator.IsInSession(marketDataUpdate.Time, false, true))
            {
                return;
            }


            Print(marketDataUpdate.Time.ToString() + " Session total shares: " +  totalShares.ToString() + " " + State);

            // TickReplay events only occur on the "Last" market data type
              if (marketDataUpdate.MarketDataType == MarketDataType.Last)
            {
                totalShares += marketDataUpdate.Volume;
                totalDollars += marketDataUpdate.Volume * marketDataUpdate.Price;
                Vwap[0] = totalDollars / totalShares;
            }

        }

        #region Properties

        [Browsable(false)]
        [XmlIgnore]
        public Series<double> Vwap
        {
            get { return Values[0]; }
        }
        #endregion

    }
}

#region NinjaScript generated code. Neither change nor remove.

namespace NinjaTrader.NinjaScript.Indicators
{
    public partial class Indicator : NinjaTrader.Gui.NinjaScript.IndicatorRenderBase
    {
        private DailyVwap2[] cacheDailyVwap2;
        public DailyVwap2 DailyVwap2()
        {
            return DailyVwap2(Input);
        }

        public DailyVwap2 DailyVwap2(ISeries<double> input)
        {
            if (cacheDailyVwap2 != null)
                for (int idx = 0; idx < cacheDailyVwap2.Length; idx++)
                    if (cacheDailyVwap2[idx] != null &&  cacheDailyVwap2[idx].EqualsInput(input))
                        return cacheDailyVwap2[idx];
            return CacheIndicator<DailyVwap2>(new DailyVwap2(), input, ref cacheDailyVwap2);
        }
    }
}

namespace NinjaTrader.NinjaScript.MarketAnalyzerColumns
{
    public partial class MarketAnalyzerColumn : MarketAnalyzerColumnBase
    {
        public Indicators.DailyVwap2 DailyVwap2()
        {
            return indicator.DailyVwap2(Input);
        }

        public Indicators.DailyVwap2 DailyVwap2(ISeries<double> input )
        {
            return indicator.DailyVwap2(input);
        }
    }
}

namespace NinjaTrader.NinjaScript.Strategies
{
    public partial class Strategy : NinjaTrader.Gui.NinjaScript.StrategyRenderBase
    {
        public Indicators.DailyVwap2 DailyVwap2()
        {
            return indicator.DailyVwap2(Input);
        }

        public Indicators.DailyVwap2 DailyVwap2(ISeries<double> input )
        {
            return indicator.DailyVwap2(input);
        }
    }
}

#endregion

[michelm]

So to recap, market replay sends out duplicate MarketDataEvents on the replay day,first one version of an event with a state of historical, and the second version of the same event with a state of realtime.

The duplicate events were breaking my vwap indicator since trades and volumes were being counted twice on the replay day.

I tried to work around the issue in several different ways, but i could not figure out a reliable way. Lots of dead ends.

I finally tracked it down to:

IsSuspendedWhileInactive                    = false;

Set it to true(it's default value) and no more duplicated trades during replay, the duplicate historic version of the trades are filtered out leaving just the realtime version.

Anyway, I'm still not sure why replay sends out duplicate MarketDataEvents on the replay day, but there probability is a reason.

[michelm]

Setting IsSuspendedWhileInactive did help, but I can still get my indicator to receive duplicate trades for the same day.

I switched the candlestick time resolution around from 5 minute to 30 seconds and back to 1 minute. I paused and restarted the replay a few time also. Not sure when in all that it started happening again.

For example, right now I have replay paused at 10:06:37 AM, and I can see that the vwap indicator on the chart is incorrect. It is reading way too high.

The logging in OnMarketData clearly shows that I'm getting trades with a state of historical for the entire replay day session even though the replay is paused at 10:06:37 AM.
Then the indicator receives a trade at 10:06:36 with a realtime state. At this point the vwap is off because the indicator has built up all the trades for the entire day even though it is on 10:06:36 AM.

Below show paused market replay and logging output.



Shows vwap way too high

[michelm]

Easy to reproduce, even when closing-restarting NT, the issue starts up right away as soon as I start up replay and start setting the replay dates. Reloading and removing-adding the indicator back does not fix the issue.

The only other weirdness I notice in general is that when i right click on one of my charts, the "Reload all historical Data" menu item is disabled.

[michelm]

So why is Market Replay sending out duplicate MarketDataEvents on the replay day, first one version of an event with a state of historical, and the second version of the same event with a state of realtime ?

[NinjaTrader_Jim]

Hello michelm,

I have been able to reproduce the issue after further testing. I would not expect to see duplicate Realtime and Historical ticks, and I see where it is affecting the indicator's output. I will follow up with more information as this develops.

Thanks for your patience.

[michelm]

Thank you for hanging in there on this. I know it sounds kind of crazy and doesn't seem like anybody else has noticed this. I've been working this for days now, searched around quite a bit, and I don't see any other reports of this.

I'm really going out on a limb here, but I think I *might* have been seeing this issue for a while now. I've never really tried to track it down like we are doing now, but it seems like over the years I've seen my tick indicators looking odd with replay and just kind of gave up thinking I was doing something incorrect.

I would think the issue is somewhere in the interaction between market replay and the backfill logic. But perhaps that is expected behavior and somewhere down the indicator stack itself is supposed to filter out the extra backfill ticks ? Wish I could work it myself...

Something that would get me going is being able to somehow get the current replay time and filtering out historic ticks past the current replay time. Any way to do that ?

[NinjaTrader_Jim]

Hello michelm,

I do not think this is expected behavior and I do not think efforts should be spent to work around what we are seeing. As I get more information from QA and Development, I will let you know.

[michelm]

Yea, I get that. Was just looking for a little temp work around to put in my OnMarketData() while your team works it so I can keep moving forward with my tasks.

BTW, I think I might hit the issue with the NT provided Volume Profile Indicator. Volume Profile is similar to mine in that it accumulates volume in OnMarketData(). When I am in the duplicate message state, Volume Profile stops drawing and reports the following in the error log:

2019-10-17 12:47:39:208|3|4|Indicator 'Volume profile': Error on calling 'OnMarketData' method on bar 1232: Object reference not set to an instance of an object.

[michelm]

Got a filter to get me going for now:

protected override void OnMarketData(MarketDataEventArgs marketDataUpdate)
        {

            if ( (Connection.PlaybackConnection != null) && (State == State.Historical) && (marketDataUpdate.Time >= Connection.PlaybackConnection.Now))
            {
                return;
            }
            ........

[NinjaTrader_Jim]

Hello michelm,

I'm glad you have found a workaround for the time being.

In our tests we are not able to encounter this issue any longer. Could you please test the following as well as a test of your own using more recent Market Replay data after updating NinjaTrader 8 to version 19.1?

If we have new context to hit this issue that we can test, please let me know.

Update NinjaTrader 8 - https://ninjatrader.com/PlatformDirect

Steps to Reproduce:

1. Import workspace, data and test script Download
2. Start NinjaTrader and connect to Playback (Start: 10/05/2019, End: 10/11/2019)
3. Fast forward into 10/7/2019 session. I dragged slider to 8:58:17 AM (MST)
4. Observe output prints from OnMarketData with timestamps after time used above.

I look forward to investigating this further.

[michelm]

Hi,

I upgraded from NT 8 19.0 to 19.1 and I was not able to reproduce. Thank You for sticking with this and following up.

Today I'm working on tick level indicators and testing with replay. It looks like the issue is resolved but I'll keep a lookout for it.

Thanks Again