Why do live data bars differ from market replay bars?

[QuantKey_Bruce]

Have you done the test I suggested - delete the historical data, restart, connect to the live data feed (with subscribed market data), download the historical data, compare the tick charts with your earlier screenshots - do those match the "realtime" chart from Friday or do they match the market replay? I would expect they would match the market replay and there's something wrong with your live data you received on the fly.

[imonlysleeping]

Have you done the test I suggested - delete the historical data, restart, connect to the live data feed (with subscribed market data), download the historical data, compare the tick charts with your earlier screenshots - do those match the "realtime" chart from Friday or do they match the market replay? I would expect they would match the market replay and there's something wrong with your live data you received on the fly.

And what exactly would be "wrong" with the live data provided by NinjaTrader? Lol (this oughta be good).

If they are providing "wrong" data to a user on the fly on a live account, they will get sued. So not sure what you are saying here bruce. Or wait, are we finally in agreement that NT is a scam? hahaha this is so hilarious. you are hilarious. and nt is hilarious.

[QuantKey_Bruce]

It's hard to say what's wrong with the data you recorded, and you can't say either, if you haven't even looked. That's why I suggested you compare it with newly downloaded historical data so you would have a reference point.

[imonlysleeping]

It's hard to say what's wrong with the data you recorded, and you can't say either, if you haven't even looked. That's why I suggested you compare it with newly downloaded historical data so you would have a reference point.

Bruce, the data i recorded or the live data? You just said theres an issue with the "live on the fly data", now you're saying data i recorded, which i did not do. I did not record any data. So what are you saying? The madness is so real. Holy ****. hahahah

[QuantKey_Bruce]

When data arrives in realtime it is recorded to your db -> tick, db -> minute, or db -> day folders. When I wrote recorded, to be clear, I meant your computer received it live and saved it to disk rather than downloading it ex post facto which is historical data download rather than recorded in realtime. That could be different than historical data downloaded later, depending on a few factors (the most obvious is if the realtime data was simulated, but also if it's a different provider who has different tick filtering or handling of correction ticks, or bundling, etc. or if maybe your computer missed some because the connection disconnected and reconnected or something else of this sort). For instance, if you recorded realtime data coming in live from Rithmic and later reload the historical data from NinjaTrader's servers (which record CQG data) there could be differences there and sometimes there are because there could be a server with an issue or a different handling of something. Re-downloading the historical you should be able to get a solid match from historical to market replay. For data in realtime there are additional questions which is why I suggested the experiment as a way of clarifying the matter instead of just talking about it.

[imonlysleeping]

When data arrives in realtime it is recorded to your db -> tick, db -> minute, or db -> day folders. When I wrote recorded, to be clear, I meant your computer received it live and saved it to disk rather than downloading it ex post facto which is historical data download rather than recorded in realtime. That could be different than historical data downloaded later, depending on a few factors (the most obvious is if the realtime data was simulated, but also if it's a different provider who has different tick filtering or handling of correction ticks, or bundling, etc. or if maybe your computer missed some because the connection disconnected and reconnected or something else of this sort). For instance, if you recorded realtime data coming in live from Rithmic and later reload the historical data from NinjaTrader's servers (which record CQG data) there could be differences there and sometimes there are because there could be a server with an issue or a different handling of something. Re-downloading the historical you should be able to get a solid match from historical to market replay. For data in realtime there are additional questions which is why I suggested the experiment as a way of clarifying the matter instead of just talking about it.

Bruce, i never said anything about having an issue between historical and market replay data, though i'm sure that's a whole other contradictory NT mess, as i've come to expect from the best trading platform on the internet. Your whole post therefore is, once again, a waste of both our times and adds yet an even deeper layer of confusion and madness as usual. You'd think at some point that would become impossible, to further complicate complete madness, but lord willing, we are not there yet.

I get the feeling you like wasting time, a lot. So lets just stick to my original post. When you're ready to give some kind of answer to that discrepancy, laid out there in plain english (with pictures) you go ahead. Though I would really prefer to hear from someone else. Or is bruce the guy they send in to numb the minds of the ones who are seeing what's really going on with NT? lol SKETCHY

PLEASE DO NOT WRITE ANOTHER POST HERE UNLESS YOU ARE SPECIFICALLY ANSWERING THE QUESTIONS LAID OUT IN THE FIRST POST OF THIS THREAD.

[QuantKey_Bruce]

I've been trying to help you, but I'm growing pretty weary at this point. I'll wait for NinjaTrader Support or other users to step in here and see if they can assist you further.

[imonlysleeping]

I've been trying to help you, but I'm growing pretty weary at this point. I'll wait for NinjaTrader Support or other users to step in here and see if they can assist you further.

Based on my personal experience, you're not good at it Bruce. Do you see how you've wasted my entire afternoon and i am now further from the truth? I do commend whatever it is you've been trying to do. You need to practice focusing your answers on the issue at hand. Don't post them until they are refined and concise, accurate and actually helpful, and most important - PERTAIN EXACTLY to what the person is asking. My two cents. Let's wait for someone else to try. I will keep calling out all the bull****. Thx

[bltdavid]

1. why is the data from the two images completely different if it all came from the same data provider and data feed?

I've seen this happen with Tick based charts when the charts didn't
start at the same time on the same day. That is, if one chart has
'Days to load' set to 3 and the other chart is not 3 days. Obviously,
the bars are built from the ticks, but the starting point for the tick
data is not in sync between the two charts, so this is reflected in
the bars, and it builds up over time.

NT7 had this issue a lot. It may be that the 'Break at EOD' option
when opening a new chart will re-sync the tick data at EOD, such
that new bars in the new session will be synced.

Do you have 'Break at EOD' set in both charts?
Are both chart using the same 'Trading hours' template?

I don't think I've ever noticed it w/Minute charts, which makes sense.
A 5min chart with 10 days loaded should still sync pretty well with,
say, a 5min chart with 3 days loaded replaying the previous day with
time warped 5min bars from Market Replay.

2. none of this was "recorded" to my local machine using the "record" option, BUT if the technology exists within NT8 for me to be able to "record" the live data i have coming in and then use that as historical data, shouldn't that be the same technology NT is using to convert the live real-time data into the Market Replay data?

If desired, you can record your live data for use with market replay,

Tools > Options > Market data -> Enable market recording for playback

You could also record your live data, saving it as historical data in the db
folders Tick/Minute/Day.

Tools > Options > Market data -> Record live data as historical

Market replay is a just a time-warped disk-based data feed being sent
to the chart window. In essence, NinjaTrader is lying to the chart window.

3. in the reference manual, the "Playback" data is equated with "Real-Time" data multiple times (see this link in particular), but we can see from my images here that that is actual bull**** on a grand scale.

See if my answer to your first question helps your situation.

Hang in there!

[marcus2300]

I compared in excel for the period from 03/12/2023-04/11/2023 in total almost 30,000 lines with open, high, low, and close.
I first took the data thru Ninjatrader Continuum, and then thru PLAYBACK.
So I checked for differences in 118,668 cells in total. Difference was Zero. So for the whole period all open, high, lows and close are identical between the two data sets.

[QuantKey_Bruce]

I compared in excel for the period from 03/12/2023-04/11/2023 in total almost 30,000 lines with open, high, low, and close.
I first took the data thru Ninjatrader Continuum, and then thru PLAYBACK.
So I checked for differences in 118,668 cells in total. Difference was Zero. So for the whole period all open, high, lows and close are identical between the two data sets.

Yes, I have done the same (just not with Excel because it is limited in the total number of data rows, and I compared individual ticks which were identical after the startup synchronization). Market Replay and Continuum are both from CQG and match perfectly unless there is a temporary server error (which is rare and almost certainly not the case here). I suggested an experiment above to demonstrate this by having them delete and re-download their historical data and re-compare the charts. Having tested this, I am confident in my statements above that there is nothing wrong with market replay in this regard and that it is lock-step in sync with CQG/Continuum data. They are likely doing something wrong or misunderstanding how NinjaTrader works. Perhaps as bltdavid suggests, they have a simple mistake, such as not checking Break at EOD and yet using tick charts that cross the session boundary with different starting dates, or they have compared an imperfect real-time data recording with the replay data rather than redownloading the data to be sure it's from the same source and complete as I suggested so many posts ago.

[QuantKey_Bruce]

I've seen this happen with Tick based charts when the charts didn't
start at the same time on the same day. That is, if one chart has
'Days to load' set to 3 and the other chart is not 3 days. Obviously,
the bars are built from the ticks, but the starting point for the tick
data is not in sync between the two charts, so this is reflected in
the bars, and it builds up over time.

bltdavid This is a really good idea. Especially since they are doing a market replay - if they have unchecked Break at EOD the 3 days starts at the starting date of the replay, while if they later refresh the chart or reopen the platform without a replay going, the 3 days starts 3 days ago. That alone would mean that tick bars would differ because despite that they are both 3 day charts, the replayed chart starts 3 days before the replay start date, while the non-replay chart does not. I didn't even think of this because I always keep Break at EOD checked for this very reason, so it's not really on my mind anymore. This might or might not be their specific mistake but it's definitely a speed bump to watch out for and good catch thinking of this.

[imonlysleeping]

“mistake” lol. you’re kidding bruce. the mistake would be on the developers of NT, not the user of course. it would be a “mistake” to leave this kind of trashy potential within their application, rather than take care of it so that the user can’t see an issue like this. it’s pretty simple, in general, just don’t make your application so crappy.

the fact that i would even have to jump through the hoop described in detail by bltdavid proves my point about the NT madness, no? have any of you guys ever checked out any other professional trading platforms? i’d recommend go on out into the world and have yourself a look around. get more of a birds eye view. NT is a messy trash heap.

break at EOD has always been checked on my end, but i will still consider your detailed words bltdavid. thanks for the effort.

[imonlysleeping]

I compared in excel for the period from 03/12/2023-04/11/2023 in total almost 30,000 lines with open, high, low, and close.
I first took the data thru Ninjatrader Continuum, and then thru PLAYBACK.
So I checked for differences in 118,668 cells in total. Difference was Zero. So for the whole period all open, high, lows and close are identical between the two data sets.

i am not familiar with continuum. but look at my original post and you can see something that differs from your little excel test results. bar data that was live became fake once it was on the market replay. i’m not necessarily saying this is being done on purpose as a conspiracy by NT to **** with its users on purpose, but simply that it’s just trash and all the data differs and it creates confusion, frustration, wasted time and resources, and it’s madness.

[QuantKey_Bruce]

the fact that i would even have to jump through the hoop described in detail by bltdavid proves my point about the NT madness, no?

In fact, "No." NinjaTrader has many capabilities and features that can be turned on or off and settings that might affect the way something like this compares. That's why it isn't so simple that there's "one right" way for everything to be configured. The "right" way depends on your use case and you seem to be assuming that everyone would want it to work the way you want it to work. That is not the case, which is why over the years the product has evolved to have many different capabilities and its users have all sorts of different configurations because they best suit their needs.