What I see in the 1 second bars is not reflected in the data. What is going on?

[carnitron]

See this image:



Here is the corresponding event data:

# __Open __High ___Low _Close
1 $73.19 $73.19 $73.19 $73.19
2 $73.18 $73.18 $73.18 $73.18
3 $73.19 $73.19 $73.19 $73.19
4 $73.20 $73.20 $73.20 $73.20

I would expect that the OHLC values would vary on time points 2 and 3. Which is correct, the chart or the data? If the chart is correct, how do I get the data to match? I don't see any flags or options that seem relevant. I'm developing an AddOn and fetching using the BarsRequest object to get 1s data back, which works fine, other than that I've noticed this.

Cheers,
Peter

[carnitron]

P.S. I don't think this is a bug on my end. I use the same data ingest code on higher time steps and do see variance between OHLC values there.

[QuantKey_Bruce]

You can go into the historical data with Tools -> Historical Data yourself and take a look at the tick data that is used to compose these 1-second bars and see for yourself which is "correct" based on the underlying base 1-tick series data. You do not say how you made this chart - are these supposed to be regular 1-second bars with a chartstyle of OHLC bar? Or, is this a custom bar type or custom chart style?

Are the bars on the chart cached? You can right-click and Reload Historical Data to make sure you are not seeing a stale cached composition of the data yet the 1-tick data may have been updated.

[carnitron]

This is 1-second simulation account data with OHLC bars. There are no custom bar types or other customizations that might affect the presentation.

This data was freshly generated at the time I took the screenshot and is not a one-off. It happens continuously as far as I can tell.

I have not spent much time with historical data and looking at tick data yet, but will look at that Soon(tm). When I do, I will remember your point about tick data being the source of truth here.

Does historical/tick data behave differently from simulation data, in terms of accuracy? I.e. simulation data is generated in "good enough" form and isn't necessarily fully accurate and/or self-consistent down at this level?

After reading your post, I thought perhaps turning on Tick Replay for the simulated data series might fix the issue, but it does appear to have any effect.

[NinjaTrader_ChrisL]

Hi carnitron, The 1 second data is built using 1 tick data, so looking at the Tools>Historical data window will show you the historical tick data that NinjaTrader is using the build the chart. When you use the simulated data feed, NinjaTrader will backfill the chart with existing historical data, then it will start to generate a real time tick stream from which bars are built.

[QuantKey_Bruce]

I would be very cautious about using Simulation data for this - you are going to confuse yourself - it's easy to do - you should probably switch to a non Simulation connection.

[carnitron]

Chris
1. I'm looking at the tick data in the historical data window here, that's pretty cool. Thx for pointing this out.
2. What I see in the historical data window for tick data on the ticker I'm looking at (AMD) is for 2022 (probably b/c that was the last time I futzed w/downloading it), while we are in 2023. There are no entries for 2023. Is there a way to view tick data for a given simulation data run? Or is this window for historical data only? (Given that is labeled Historical Data, hah.)
3. According to your comments, once we are in the middle of a run, the simulation feed is generating data on its own. And what shows up on the chart repeatedly does not match what I see during data ingest, in the manner described above. I'd like to get the bottom of why. On some level, it doesn't matter what the tick data is, b/c if a bar's top on time point 10s 73.20 and the bottom is 73.19, then that should show up in the OHLC data somehow. I suspect there is a bug here.


Bruce
1. I've worked on many software engineering project vastly more complex than this and am not worried about temporary confusion. I am trying gain understanding as to why the data and the chart don't match. I'm definitely interested, if you have specific, factual reasons or ideas as to why.

[NinjaTrader_ChrisL]

Hi carnitron, I am missing the question you are asking here. Once the chart hits real-time and you are connected to the simulated data feed, this is fake data generated by your PC, its not real and is only a computer-generated stream of ticks incoming into the platform. I am not seeing what you are saying is not "Matching". If you could exemplify I would be happy to help.

[QuantKey_Bruce]

Hi carnitron, I am missing the question you are asking here. Once the chart hits real-time and you are connected to the simulated data feed, this is fake data generated by your PC, its not real and is only a computer-generated stream of ticks incoming into the platform. I am not seeing what you are saying is not "Matching". If you could exemplify I would be happy to help.

carnitron This, precisely, is why I said "I would be very cautious about using Simulation data for this - you are going to confuse yourself - it's easy to do - you should probably switch to a non Simulation connection."

[NinjaTrader_ChrisL]

carnitron, I tested the simulated data feed with this and compared to the chart and could not reproduce this.

protected override void OnBarUpdate()
        {
            Print(CurrentBar + "");
            Print(Open[0]);
            Print(High[0]);
            Print(Low[0]);
            Print(Close[0]);
            Print("");
        }​

You can match the Print(CurrentBar + ""); coming from the output windows to the bars on the chart if you right click>Show Data Box, then right click the Data Box>enable "Show Bar Indexes".

[carnitron]

Chris - Thank you for testing this, taking concrete steps like this is how we get to the end. The code path in your example is different than mine, and perhaps why you are not seeing what I am seeing. I am:

1. Creating a BarsRequest object.
2. Assign an event handler to the "public event EventHandler<BarsUpdateEventArgs> Update" member.
3. I then get data from BarsUpdateEventArgs.BarsSeries, via routines like GetOpen(), GetClose(), etc.

The reason it doesn't "match" is this. In the image at the very beginning of the post, if you look at the bar I have a 2 written under, you see that is spans from 73.18 to 73.19. I would expect OL to be 73.18 and HC to be 73.19, no? Yet if you look the data OHLC are always identical to one another on each time point.​

[NinjaTrader_ChrisL]

Hi carnitron, thanks for clarifying. I made an indicator that performs a bars request, can you please try reproducing the issue being reported on my indicator:

https://forum.ninjatrader.com/forum/ninjatrader-8/indicator-development/1036851-barsrequest#post1036928

[carnitron]

Hi Chris, thank you for this code and apologies for the delay in responding.

So the good news is that the code in your indicator prints the values that I would expect.

Unfortunately, very similar code in my infrastructure does not, so I am trying to narrow down what is different between the two code paths. I am not getting data from within an indicator, of course, but from an AddOn. But I have a ways to go before I can know that this is the telling difference, and I'm certainly not sure why that should matter. I will keep you posted.

I would love it if it was some dumb bug on my part, hah.

Cheers,
Peter

[carnitron]

Well this on my end. I did not realize I can get multiple updates on the same second. Most of the time we only get one, but sometimes we get two, and it appears that only on this 2nd update can the OHLC values be different from one another.

In an attempt to jump on prices as quickly as possible, I act as soon as the first set come in, ignoring the 2nd set in some places.

That's on me though, not NT8, ofc.

It does raise two questions though:

1. Can there be any number of price updates for a given 01s interval? Like it's normally one update, sometimes it is two, but could it be three or more? I.e. we cannot make any assumptions about how many price updates we will get that belong to a given second?

2. Is the only way to be sure we've gotten the last set of updates for second N is when we get the first update for second N+1?

[QuantKey_Bruce]

You cannot know for certain that you have received the last intrabar update of a 1-second bar until the first intrabar update that falls into the next second. This is because you cannot predict the network or processing latency with certainty, nor can you know how many trades will take place in that second in advance. If you're trying to "declare the second closed" without a tick happening in the next second, you could collect some statistics on how long your latency is usually and then when it's been a good bit longer than that into the next second (perhaps three standard deviations?), you could just presume that no more is coming, but you would occasionally be incorrect. You might also find that three standard deviations is more than an entire second into the future. These latencies will not be normally distributed, but you have to start somewhere.