Last/First tick of volume bar changed in V7

[HNWtrader]

I have an indicator that works in V6.5 for time, range, or tick based charts and also for volume based charts. For a volume based chart, a single tick at the end of a fixed-volume bar often has volume in excess of that needed to complete a bar. In V6.5 this tick is identified as the first tick of a new bar. To handle this situation, I have code to compare the tick volume of the first tick of a bar with Volume[0]. When the tick volume is greater than Volume[0], I use the difference to in effect create a "final tick" for the previous bar, and reduce the volume of the first tick of the new bar. I don't remember if I was careful enough to handle the rare cases when that last/first tick extends the range of the previous bar, but at least for all other cases the analysis I do within each bar is applied to a total volume that is exactly correct.

Something has changed in V7 related to the system processing of volume bars. By looking at the changes to some similar NT supplied indicators (Volume Profile), I was able to get my indicator running in V7. However, my scheme for splitting the last/first tick between the new bar and the previous bar no longer works.

I am wondering if V7 has incorporated some processing that takes care of the splitting of the last/first tick of volume based charts that I can utilize. If not, I could use some hints about how the system processing has changed that prevents my V6.5 scheme from working in V7.

[NinjaTrader_Brett]

Hello,

Thanks for your patience.

I am checking into this.

[HNWtrader]

I added a print statement to my code to look at the volume of the tick, and Volume[0]. In many cases for V7 I am seeing the volume of the first tick of the bar being less than Volume[0]. I do not see how this can happen unless the processing for computing Volume[0] is outrunning my indicator processing of the first tick of the bar, and is processing subsequent ticks before the processor gets to my first tick calculation.

[NinjaTrader_Brett]

Hello,

What code are you using to print the volume of a tick?

I look forward to assisting you further.

[HNWtrader]

The following code is from the NTV7 provided indicator @VolumeProfile

protected override void OnMarketData(MarketDataEventArgs e)
{
	if (e.MarketDataType == MarketDataType.Ask)
	{
		askPrice = e.Price;
		return;
	}
	if (e.MarketDataType == MarketDataType.Bid)
	{
		bidPrice = e.Price;
		return;
	}
	if (e.MarketDataType != MarketDataType.Last || ChartControl == null || askPrice == 0 || bidPrice == 0)
		return;

	if (Bars != null && !Bars.Session.InSession(DateTime.Now, Bars.Period, true, Bars.BarsType))
		return;

	double	price	= e.Price;
	long	volume	= e.Volume;

	if (!volumeInfo.ContainsKey(price))
		volumeInfo.Add(price, new VolumeInfoItem());

	VolumeInfoItem volumeInfoItem = volumeInfo[price];

	if (price >= askPrice) 
		volumeInfoItem.up += volume;
	else 
		if (price <= bidPrice)
			volumeInfoItem.down += volume;
		else
			volumeInfoItem.neutral += volume;
}

I reused this code to get the volume of the tick and modified some things for my indicator. I eliminated the 4th if statement because I thought it might be giving me some problem that I do not remember (I think it was because the indicator would not run in replay mode. I later found that eliminating the "#region Miscellaneous" and #endregion lines that came from VolumeProfile took care of this problem. My memory is fuzzy on this; I should go back and confirm this at some point). For no particular reason I have "else if" instead of "if" for the second and third ifs. Right after this I print and then compare volume with Volume[0], and split the tick volume between the previous bar and the new bar. My code looks like this

		protected override void OnMarketData(MarketDataEventArgs e)
		{
			if (e.MarketDataType == MarketDataType.Ask)
			{
				askPrice = e.Price;
				return;
			}
			else if (e.MarketDataType == MarketDataType.Bid)
			{
				bidPrice = e.Price;
				return;
			}
			else if (e.MarketDataType != MarketDataType.Last || ChartControl == null || askPrice == 0 
					|| bidPrice == 0)
				return;

			double	price	= e.Price;
			long	volume	= e.Volume;
			[COLOR="Blue"]barVolume = (long) Volume[0];[/COLOR]

			if(FirstTickOfBar)
			{
				
				/// This "if" checks to see if part of first tick that creates a new bar should be applied to the
				/// previous bar.  This will often happen for volume bars; it never happens for time or range bars
				/// When it happens, the tick is devided across the old bar and the new bar so as to keep all bars
				/// at the "data series" volume specified.
Print (volume + "   " + barVolume);
				if(volume > barVolume)
				{
					remainVolume = volume - barVolume;
					volume = barVolume;

					///Finish the old bar
					if (!volumeInfo1.ContainsKey(price))
					{
						volumeInfo1.Add(price, new VolumeInfoItemClass());
					}
	
					VolumeInfoItemClass volumeInfoItemLast = volumeInfo1[price];
	
					if (uptick == true)
					{
						if (price >= oldPrice) volumeInfoItemLast.up += remainVolume;
						else if (price < oldPrice) volumeInfoItemLast.down += remainVolume;
					}
					else if (uptick == false)
					{
						if (price > oldPrice) volumeInfoItemLast.up += remainVolume;
						else if (price <= oldPrice) volumeInfoItemLast.down += remainVolume;
					}
				}

uptick is a boolean used to keep track of upticks and downticks (instead of ticks at bid or ask as in VolumeProfile). Its value was established while processing the previous tick, and is updated when processing the part of this tick's volume that belongs to the new bar in code that follows. The blue line where barVolume is defined was actually the first line in OnBarUpdate. I moved it here so you could see it, and I am now running the indicator with this change. It was declared in #region Variables and is used elsewhere in my indicator.

At the time this code is run, volumeInfo1 is the dictionary for the bar that is being finished. I have a dictionary for each of the last few bars. Immediately following this code I initialize a new dictionary for the new bar and process the first tick of that bar with the modified value of volume.

Note that the volume I am printing is the unmodified value obtained from e.Volume. I don't see how this can ever be less than Volume[0] if the tick being processed is indeed the tick that straddles bar [1] and bar [0], but the print statement shows that it often is.

[HNWtrader]

I spent some time watching a 5-minute ES 12-10 chart while the market was very slow in the wee hours of the morning. I have V6.5 running on one machine, and saw perfect agreement between Volume[0] and the accumulated volume of ticks within the bar. On another machine I have V7 running, and what it looks like to me is that the first tick of every bar in V6.5 is being handled as the last tick of the previous bar in V7, except that the volume of this tick is being included in the calculation of Volume[0]. I can check in the data boxes to confirm that both versions show the same total volume for every 5-minute bar, and they do. However, in V7 the difference between the volume of what V7 is calling the first tick of the bar and Volume[0] is exactly the volume of what V6.5 is calling the first tick of the bar. Furthermore, the volume of the tick that V7 is saying is the first tick of the bar is the volume of the second tick of that bar, according to the time stamps in the time and sales record. In other words, after the second tick arrives according to the time and sales record, Volume[0] is correctly shown as the sum of the volumes of the first two ticks, but the second tick is the one that V7 is identifying as the first tick. All the processing that is done for the real first tick is being assigned to the old bar by the indicator processing.

When there is enough time between the first and second tick, you can even see that the V6.5 chart creates a new bar on the chart on the first tick, while the V7 chart creates a new bar on the second tick.

[NinjaTrader_Bertrand]

HNWtrader, thanks for providing the code snippets used and your observations - we will look into and attempt reproducing here on our end.

[NinjaTrader_Brett]

Hello,

Thanks for the information.

I believe what you are running into here is expected with 7 since 7 is multi threaded now.

http://www.ninjatrader.com/support/helpGuides/nt7/onmarketdata.htm

Please see tip number 4: With NinjaTrader being multi-threaded, you should not rely on any particular sequence of events like OnMarketData() always being called before OnBarUpdate() or vice versa.

Therefor, you could see OnBarUpdate() fire Before OnMarketDate() or vice versa. This could be what your seeing here.

There is no other solution at this time other then to code your strategies/indicators with this in mind.

Let me know if I can be of further assistance.

[RJay]

I have an indicator that works in V6.5 for time, range, or tick based charts and also for volume based charts. For a volume based chart, a single tick at the end of a fixed-volume bar often has volume in excess of that needed to complete a bar. In V6.5 this tick is identified as the first tick of a new bar. To handle this situation, I have code to compare the tick volume of the first tick of a bar with Volume[0]. When the tick volume is greater than Volume[0], I use the difference to in effect create a "final tick" for the previous bar, and reduce the volume of the first tick of the new bar. I don't remember if I was careful enough to handle the rare cases when that last/first tick extends the range of the previous bar, but at least for all other cases the analysis I do within each bar is applied to a total volume that is exactly correct.

Something has changed in V7 related to the system processing of volume bars. By looking at the changes to some similar NT supplied indicators (Volume Profile), I was able to get my indicator running in V7. However, my scheme for splitting the last/first tick between the new bar and the previous bar no longer works.

I am wondering if V7 has incorporated some processing that takes care of the splitting of the last/first tick of volume based charts that I can utilize. If not, I could use some hints about how the system processing has changed that prevents my V6.5 scheme from working in V7.

HNWtrader,

If you read this thread, I think it will explain the difference between 6.5 and 7.

http://www.bigmiketrading.com/ninjatrader-programming/5913-using-ninjatrader-volume-indicators-better-read.html

RJay

[HNWtrader]

I believe what you are running into here is expected with 7 since 7 is multi threaded now.

Based on what I observed today, I do not believe multi-threading is causing the problem. Isn't NT6.5 also multi-threaded? I never had this problem in NT6.5. Moving my definition of barVolume from OnBarUpdate to OnMarketData as I did in preparing my earlier response may have resulted in stabilizing the outcome of my indicator processing if there ever was a threading issue, but having done that and focusing on a time based chart instead of the volume based chart has, I believe, revealed a consistent error in NT7. I now have very strong evidence that NT7 is tagging the second tick of every bar as the first tick of the bar.

I should add some things to make the evidence more transparent to others if I have to prove the point, but today I observed this consistent error on every 5-minute bar on ES 12-10 starting with the 12:40 bar through the last bar of regular trading hours. I have a couple of screen shots showing the T&S, Output Window with the print of volume and barVolume (i.e. Volume[0]) that illustrate the point. Within the next couple of days I can modify my indicator to automate the generation of similar screen shots and prove that every new bar suffers from the same error. I am not going to post the screen shots I already have here because they require more explanation than I want to get into right now.

As I was writing this, RJay posted his contribution to this issue. I have not read all of his link, but it seems I am not the first person to encounter this problem, or to do something to prove there is a problem with NT7 mis-identifying the first tick of a bar.

[NinjaTrader_Brett]

Hello,

Thanks for the follow up.

If you check RJays post as we did a lot of research on this with development the outcome I explained to you is what we ended up finding as the issue.

If you have information otherwise please post it so that I can take a look at it with development and look into this further.

Please post this information with a simple script that we can test on our side that shows the issue if you see it this way I can get this script over to development and get back to you.

Thank You.

[RJay]

Based on what I observed today, I do not believe multi-threading is causing the problem. Isn't NT6.5 also multi-threaded? I never had this problem in NT6.5. Moving my definition of barVolume from OnBarUpdate to OnMarketData as I did in preparing my earlier response may have resulted in stabilizing the outcome of my indicator processing if there ever was a threading issue, but having done that and focusing on a time based chart instead of the volume based chart has, I believe, revealed a consistent error in NT7. I now have very strong evidence that NT7 is tagging the second tick of every bar as the first tick of the bar.

I should add some things to make the evidence more transparent to others if I have to prove the point, but today I observed this consistent error on every 5-minute bar on ES 12-10 starting with the 12:40 bar through the last bar of regular trading hours. I have a couple of screen shots showing the T&S, Output Window with the print of volume and barVolume (i.e. Volume[0]) that illustrate the point. Within the next couple of days I can modify my indicator to automate the generation of similar screen shots and prove that every new bar suffers from the same error. I am not going to post the screen shots I already have here because they require more explanation than I want to get into right now.

As I was writing this, RJay posted his contribution to this issue. I have not read all of his link, but it seems I am not the first person to encounter this problem, or to do something to prove there is a problem with NT7 mis-identifying the first tick of a bar.

HNVtrader,

I have overcome this problem by temporarily saving every tick in OnMarketData until the next tick is received.

When FirstTickOfbar is received, add temporarily stored tick to current tick and send to the chart.

Check sample code in thread from my prior post for examples.

RJay

[eDanny]

RJ's solution works well for the math end but it still does not solve problems with plotting, especially when there are 1 tick bars on the chart. The plot will show the previous bars value until the next tick arrives and if the next tick is the beginning of the next (new) bar the plotting is off.

Dan

[HNWtrader]

Hello,

Thanks for the follow up.

If you check RJays post as we did a lot of research on this with development the outcome I explained to you is what we ended up finding as the issue.

If you have information otherwise please post it so that I can take a look at it with development and look into this further.

Please post this information with a simple script that we can test on our side that shows the issue if you see it this way I can get this script over to development and get back to you.

Thank You.

Edit- Attachments Removed

I certainly cannot say that multi-treading will not cause problems. I will look at some other types of charts with the scripts that I am posting to see if I can detect such problems. However, for now I believe that I have clearly demonstrated that the bar that is being processed as FirstTickOfBar is in fact the second tick of the bar for time based charts. I suspect this is true of other type charts as well. If this error is corrected, I believe my indicator will work the same way in NT7 as it did in NT6.5.

I have scrubbed my indicator down to the essential code needed to support my case. I hope I have not left any irrelevant code, but if I did I don't think it is doing any harm. I am attaching the source code for the scrubbed indicator SecondTickProof, and the source code (derived from other peoples ideas and code) for a strategy called SaveChart2 that I developed to repeatedly capture chart images. I am also posting one of the images created using these scripts. As it stands, the strategy captures an image of the chart after the third tick of a new bar. The number of bars between captures is an adjustable parameter called SaveCount2. (I usually do not save images in the evening and early morning, but I edited the code to save around the clock- see lines 181-183.) If you run both the indicator and the strategy, you can walk away and come back later to examine as many images as you care to accumulate. You will have to edit the strategy to set the path to the folder where you want the images to be saved.

Setting up the chart properties to match mine is worthwhile, at least to get started. Two important parameters are the right side margin that I have set to 310, and the bounds.Width of the chart that I have set to 1035; the latter number is displayed toward the lower right side of the chart to help achieve this setting as you adjust the chart window. If you then adjust the separation of chart bars to view only 6 bars as I have, you will see charts like mine.

Every one of these charts confirms that the displayed Bar Volume at the top right is the sum of the first three tick volumes that follow the minute rollover that starts a new bar, and yet the graphic to the left of the listed ticks shows that only the second and third of these has been processed as belonging to the new bar. The first of the three, the real first tick of the bar was processed as the last tick of the previous bar, even though its volume has been added to the volume of the new bar.

There is no randomness to this as you would expect from a race condition. RJay's scheme for correcting his charts by saving the information for the prior tick depends on this consistency. His scheme can only work if the tick that is being treated by NT7 as the first tick of the bar is in fact the second tick of the bar, every time. I have not looked carefully enough at his code to see if he is somehow keeping the real first tick from being processed as the last tick of the previous bar, but for me that would be a nightmare. And eDanny has noted the additional problem this causes for 1-tick bars. How do you delay the processing of every tick until after the next tick arrives? It is probably possible, but why not fix the source of the problem instead of writing logically ugly code to compensate for it?

I hope I can convince you that there is a consistent error in NT7 in that it is consistently tagging the second tick of every bar as the first tick. Fixing this might not solve all the problems that may be encountered due to multi-threading, but I am convinced that my (not to mention everyone else's) time based charts for sure and very likely also my volume based charts will be just fine if this error is fixed.

[NinjaTrader_JoshP]

HNWTrader,

This SecondTickProof is an invalid test. You cannot use FirstTickOfBar outside of the bars context of OnBarUpdate() as outlined here: http://www.ninjatrader.com/support/h...ttickofbar.htm

FirstTickOfBar is updated on the OnBarUpdate() event of the tick that is the first tick. There is no guarantee OnBarUpdate() will come before or after the OnMarketData() event.

Consider this sequence of events:
1. The theoretical first tick of bar is received in OnMarketData() first --> FirstTickOfBar evaluates to false because it has not gone through OnBarUpdate() yet
2. OnBarUpdate() processes that tick and now FirstTickOfBar = true
3. Second tick of bar is received in OnMarketData() --> you check FirstTickOfBar which was just set to true, but never set to false yet
4. OnbarUpdate() processes that second tick and now FirstTickOfBar = false.

Because of this you assume it was the second tick that triggered FirstTickOfBar, but it is simply not the case. This is why you cannot use FirstTickOfBar outside of OnBarUpdate().