Why do live data bars differ from market replay bars?

[NinjaTrader_BrandonH]

Hello QuantKey_Bruce,

Thanks for pointing that out.

Yes, this was a typo and I have corrected it in my previous post.

I had mean to say:

The Simuation data feed connection will provide delayed market data and the data will be delayed by 10 minutes.

The Simulated data feed connection will automatically use the last price from the last connection as the starting price for the instrument and the data provided will be randomized.

[bltdavid]

Yeah, I see that.

But when typos are so damn easy ... there's a problem here.
(See the EDIT of my previous post.)

I think the depth and breath of the stupidity of recent NT
decisions has just been widened ... 8.1 is reusing the word
'Simulation' in their sign-on process . that is, this word has
gained another meaning/usage in the 8.1 world ... and now
that term is too overloaded and causing confusing.

A realistic looking 'typo' like this, esp by NT support, can be
extremely difficult if not impossible to disentangle, esp for new
people, but also for experienced users -- it's a pity that users
have to know these very similar terms cold to correct NT
support. Good catch, Bruce.

[QuantKey_Bruce]

A realistic looking 'typo' like this, esp by NT support, can be
extremely difficult if not impossible to disentangle, esp for new
people, but also for experienced users -- it's a pity that users
have to know these very similar terms cold to correct NT
support. Good catch, Bruce.

bltdavid I think NinjaTrader's changes were well intentioned, and it's all quite new, so we're all trying to clarify all this and make constructive suggestions for how to fix it, such as my suggestion thread referenced above. I made a mistake in repeating verbatim, with a citation to the source, a statement from NinjaTrader Support, that resulted in some kerfuffle here in this thread. I was only trying to help and made sure to cite exactly where the information came from (though that turned out to be a typo). When that was determined to be in error, we got it clarified and corrected, and have carefully documented the correct information now so I don't think it will happen again (or at least, I won't make this mistake again). But, I completely agree this wording is potentially quite confusing - as you pointed out, it even confuses NinjaTrader support sometimes and it's easy to make a typo in haste because the words are so very similar. That's why I suggested that the wording be changed all the way back in March because I already suspected then that this was going to be problematic.

[bltdavid]

The Simulation data feed connection will provide delayed market data and the data will be delayed by 10 minutes.

The Simulated data feed connection will automatically use the last price from the last connection as the starting price for the instrument and the data provided will be randomized.

And this is my point!

How in the heck is anyone (new or experienced)
going to realize these are two separate things?

Typo, my butt.
There is a larger problem at play here:

These two terms are too similar.
There is too much room for confusion.

(Obviously, Bruce, I agree with you on this)

The new 8.1 'Simulation data feed' concept is too similar to
the already existing 'Simulated Data Feed' feature.

In 8.1, the pre-built connection 'Simulated Data Feed' should
have been renamed to something like 'Fantasy Data Feed'
or 'Fake Data Feed' or 'Test Data Feed' or 'Synthetic Data
Feed' or 'Artificial Data Feed' ...

So, why didn't the existing one get renamed? NT Mgmt showed
arrogance on changing other things, so perhaps unawareness
of the confusion they're about to create, so it never occurred
to anyone to rename the old 'Simulated Data Feed' to make
way for the new 8.1 concept?

This would have avoided a lot of confusion by a lot of people,
even NT support people.

[QuantKey_Bruce]

bltdavid Yes, we've been hashing out this same thing since March at https://forum.ninjatrader.com/forum/...n-vs-simulated and a few other threads. They're too similar. They need to be changed. I agree that changing Simulated Data Feed to Synthetic Data Feed is a decent way to resolve it.

[NinjaTrader_BrandonH]

Hello bltdavid,

Thanks for your note.

As stated by Ryan on the forum post QuantKey_Bruce linked to and seen linked below, this has been reported to our product team and we are tracking customer feedback on changing the name of either the Simulation or Simulated data feed connections.

https://forum.ninjatrader.com/forum/...95#post1242895

I have added your notes on the internal ticket that we are tracking feedback on this topic for so the product team is aware of it and will take it into consideration. The ticket number is NT-3733.

[bltdavid]

bltdavid Yes, we've been hashing out this same thing since March at https://forum.ninjatrader.com/forum/...n-vs-simulated and a few other threads. They're too similar. They need to be changed. I agree that changing Simulated Data Feed to Synthetic Data Feed is a decent way to resolve it.

Yeah, I'm a bit late to the party on this.

As time has passed, it is clear NT mgmt doesn't care about
the user's ability to bypass the mandatory password requirement.

Frankly, the thought of upgrading really depresses me.

Thus, I'm not keeping up to date as I normally would.

Personally, I'm going to stay with pre '8.1' releases forever.
Sigh ... and it depresses me that 'forever' may not be possible.

I've added my feedback to your linked thread.

Thanks

[imonlysleeping]

I have tested comparing a 500-Tick ES 06-23 chart when connected to a realtime data feed connection to a 500-Tick ES 06-23 chart using Market Replay data on the Playback connection and the OHLCV bar values when comparing the charts are matching. See the attached screenshots.

First, I connected to a realtime data feed connection and opened a 500-Tick ES 06-23 chart. Then, I opened a Data Box window and noted the OHLCV values in the Data Box window when hovering the mouse over bars on the chart. Next, I downloaded Market Replay data for the ES 06-23 instrument for the dates 04/27/2023 and 04/28/2023.

After downloading Market Replay data, I connected to the Playback connection and set Playback to use Market Replay. I compared the same exact bar OHLCV values on the chart that I noted when connected to a realtime data feed connection and see that they do in fact match up exactly. The attached screenshots demonstrate that the bar values do match between a realtime data feed connection and Playback with Market Replay.

Are you able to reproduce the behavior you are reporting when following the same steps listed above?

After following your test exactly as described above, i can conclude that:

The issue of OHLCV inconsistency between live data and market replay data is real and appears to be found all throughout any date/time ranges i choose.

My settings once again:

Connection: "Live"
Chart: ES Jun 23, 500 tick
Market Data: subscribed with NT
Date/Time: see screenshots below
NT version: 8.1.1.3

Here is just one screenshot comparison chosen randomly out of the MANY instances i'm seeing.

PLEASE NOTE:
Usually the disparity is just one tick off.

Volume and time are also both off.

The bar immediately following the labeled bar happens to also be a problem bar.

[NinjaTrader_BrandonH]

Hello imonlysleeping,

Thanks for your notes.

The data pump can mean that ticks could be off by up to one second. This could be seen in the video shared about comparing real-time performance with historical or market replay data on the forum thread linked below.

https://forum.ninjatrader.com/forum/...nce#post100192

You can record market replay data from real-time data by enabling the 'Enable market recording for playback' option in the Tools > Options > Market data tab to make the market replay results more similar to the realtime data.

Record market replay data from real-time data: https://ninjatrader.com/support/help...ReplayRecorder
How to record live market data: https://ninjatrader.com/support/help...LiveMarketData

That said, I have attached a demonstration video here showing the market replay data matches or is very close to matching when comparing market replay data to realtime data: https://brandonh-ninjatrader.tinytak...M18yMTM5NzM0MQ

In the video linked above, we can see that the realtime bar at 09:39:44 has a Close price of 4075.50 and a volume of 647. When comparing this bar using Market Replay data on Playback, we can see that the 09:39:44 bar Close price and Volume match the bar when we were connected to realtime data.

We can also see the bar at 09:41:22 when connected to realtime data has a Close price of 4076.25 and a Volume of 796. When comparing this bar with Market Replay data on Playback, we see that the 09:41:22 bar Close price and Volume match. We then compare a third bar at 09:43:17 and see the Close price and Volume match again when comparing realtime data to Market Replay data.

[imonlysleeping]

Hello imonlysleeping,

Thanks for your notes.

The data pump can mean that ticks could be off by up to one second. This could be seen in the video shared about comparing real-time performance with historical or market replay data on the forum thread linked below.

https://forum.ninjatrader.com/forum/...nce#post100192

"The data pump" . PLEASE EXPLAIN THIS.

that thread you linked to represents all that i hate about ninjatrader. yuck. the longest craziest explanations for such extreme technical mayhem that it just makes you want to find another platform. its SUCH A MESS and so difficult to sift through all of that madness. i could spend all day asking questions about the unclarified, ambiguous statements in that thread and i'd probably end up more confused than when i started, as has been the usual way it goes in this forum.

do you think that if long elaborate explanations are necessary to answer simple questions, maybeeee the platform needs work?

OBJECTIVELY, its way way way too much for an average user or programmer to have to sift through in order to get results that are meaningful and valid. things should just work. and if they don't work simply, then they shouldn't be found anywhere on the platform. this is common sense.

did NT ever think to just make things good? remove all the options and the trash, and just make it good and accurate so i don't have to worry about or question every single little thing. right? does anyone see this madness and agree with me? i've never spent this much time on a forum and had to go around and around in circles like this and only ended up worse off than when i started, especially in terms of general frustration/confusion.

NT has a lot of potential, but there is a major oversight happening where common sense has been replaced with technical overload, when it just needn't be that way.

high user case functionality and design needn't be a complicated mess. the best things in the technical realm are simple and elegant and don't need explanation because they have a high level of intuitiveness built in for the user. NT lacks this in more ways than one. convince me otherwise.

[NinjaTrader_BrandonH]

Hello imonlysleeping,

Thanks for your notes.

When using market replay, the NinjaTrader core market data updates occur at the granularity provided by the market data provider. However, the NinjaTrader user interface only visually updates in 1-second intervals for performance optimizations. Even though the NinjaTrader UI's are only visually updating at 1-second intervals, orders, indicators, and strategies will calculate just as they were running in real-time.

This is noted in the "Understanding how the Playback Works" section of the help guide linked below.
https://ninjatrader.com/support/help...ePlaybackWorks

The concerns you mentioned have been noted and will be taken into consideration. I submitted a feature request to the Development team for you regarding adding the ability to make the backtest engine automatically implement 1-tick intra-bar granularity (at the cost of performance). This request is being tracked under the number SFT-5912.

As with all feature requests, interest is tracked before implementation is considered, so we cannot offer an ETA or promise of fulfillment. If implemented, it will be noted in the Release Notes page of the Help Guide.

Release Notes — https://ninjatrader.com/support/help...ease_notes.htm

Note that if you are using Calculate.OnBarClose in a strategy, High Order Fill Resolution could be used when testing scripts on historical data. Selecting order fill resolution of "High" will allow you to set a secondary bar series, such as a 1-Tick bar series, to be used as the price data to fill your orders, this allows you to bring in more granular data then you are currently running the strategy on.

High order fill resolution: https://ninjatrader.com/support/help...FillResolution

Currently, Tick Replay and High Order Fill Resolution do not work together. If you would like me to I could add your vote to an existing feature request for using Tick Replay and High Order Fill Resolution together.

[QuantKey_Bruce]

NinjaTrader_BrandonH We do already have the ability to enforce 1-tick intra-bar granularity by submitting the orders to a 1-tick series, do we not? And if we did so, the orders would always be filled basically as if "High" and "1-Tick" were used regardless of those settings, because we're putting the orders directly on a 1-tick series. This does not, though, address the original poster's concern that the bars are different in some cases.

[NinjaTrader_BrandonH]

Hello QuantKey_Bruce,

Thanks for your note.

Yes, there is the ability to add intra-bar granularity by submitting orders to a 1-Tick series.

To address the original poster's inquiry, i have clarified what is referred to by "The data pump".

"When using market replay, the NinjaTrader core market data updates occur at the granularity provided by the market data provider. However, the NinjaTrader user interface only visually updates in 1-second intervals for performance optimizations. Even though the NinjaTrader UI's are only visually updating at 1-second intervals, orders, indicators, and strategies will calculate just as they were running in real-time.

This is noted in the "Understanding how the Playback Works" section of the help guide linked below.
https://ninjatrader.com/support/help...ePlaybackWorks"

I have attached a demonstration video here showing the market replay data matches or is very close to matching when comparing market replay data to realtime data in post # 90. Also in post # 90 I stated you can record market replay data from real-time data by enabling the 'Enable market recording for playback' option in the Tools > Options > Market data tab to make the market replay results similar to the realtime data.

[QuantKey_Bruce]

NinjaTrader_BrandonH I think the key question here is when you say the replay results will be "similar to realtime data" - why would they differ at all? Especially if, as you suggest, the checkbox 'Enable market recording for playback' is used to play back the exact same data as was received in realtime? How, then, could they possibly not be identical?

[NinjaTrader_BrandonH]

Hello QuantKey_Bruce,

Thanks for your note.

When comparing Tick data using Market Replay/Playback the Market Replay/Playback data pump can mean ticks are off by up to one second.

This means results will be exactly the same or very close to the same.

We are tracking interest in an internal feature request for having the Playback connection update tick per tick instead of a 1-second interval. Please let me know if you would like me to add your vote to this feature request.