Real-time vs. Historical Tick data not match - NinjaTrader FX connection data feed

[ToNovy]

Hallo,
can someone explain that, please? It's seems to me, that there is some kind of tick data aggregation or compression. In the past, historical and real time tick data matched 1:1. Why not now?Picture of the same situation. Look at delta, absorption, PoC etc..

[ToNovy]

oh no, HDS (Historical Data Server) is malfunctioning again - not providing any data today

it would be nice if there was a page showing the service availability status.. like https://discordstatus.com for example

[NinjaTrader_Jason]

Hello ToNovy,

Do you experience the difference if you test a 1 Minute or 150 Tick chart without any add-ons or indicators applied?

In addition, please follow the steps below to manually attach your log and trace files in an email and send it to "platformsupport[at]ninjatrader[dot]com". Put 'Att Jason, 03588032' in the subject field of the email.

• Open the Windows File Explorer (hotkey = Windows key + E) Then navigate to, "Documents" > 'NinjaTrader 8' folder
• Right-click on the 'log' and 'trace' folders and select Send To> Compressed (zipped) Folder.
• Send the 2 compressed folders as attachments to this email (and don't attach each log or trace file individually)
• Once complete, you can delete these compressed folders.

Unfortunately there was an issue with the historical data servers. They did not record historical data from the Sunday open. They are currently recording historical data again. We are working on restoring the missing historical data.

Thank you for the suggestion.

I have submitted this as a feature request to the Development Team. I will follow up with an internal tracking number for your reference as soon as it is created.

Thanks in advance for your patience.

[ToNovy]

Hi Jason,

unfortunately historical data still doesn't work on FX.. both minute and tick data. So I can't test now.
I'll like to test later, I just have no idea how to tell the difference on a clean graph, but what can I say now, sometimes the OHLC changes if the source is real-time vs. historical when candles are created from tick data

I also tried different addons for OF, it shows the same - difference between data.

I will do some tests and email later.

Thank you

[EB Worx]

I'm in a discord group with 200+ order flow footprint traders trading futures, mostly ES
Many of us experience historical tick data issues several times a week
They use all kinds of footprint indicators (mzPack, Orderflows, TDU, Gomi, Analytica) and these issues appear with all of them.

• historical tick data not matching the bar open/high/low/close
• or just no historical tick data at all

Sometimes resetting the database or deleting the entire documents\ninjatrader\db folder helps..
But often times nothing helps and you have to wait for it to resolve itself which can take hours

Another strange thing I noticed is that simply restarting NinjaTrader can cause it to lose all historical tick data
All is good, I close NinjaTrader and start it again, and suddenly all historical tick data is gone.


I already mentioned this several times before in this forum, but I did not get any solution yet, other than the default 'reset your DB or delete the DB folder'
As said this is not helping.. It's time NinjaTrader starts doing something about this

[NinjaTrader_Jason]

Hello ToNovy,

The internal tracking number for your feature request (webpage that shows the status of the historical data servers) is SFT-5713. Please reference this internal tracking number when contacting Platform Support if you ever have questions regarding this feature request.

When a feature request is implemented, you'll find a description of the new feature in the release notes:

• https://ninjatrader.com/support/help...ease_notes.htm

[ToNovy]

Still no progress on why are tick data so different? Or NT don't care?
Pictures before and after reload historical data....

[NinjaTrader_Jason]

You can experience difference in data when the data received in realtime is reloaded as historical data. Please see the link below that explains why.

https://ninjatrader.com/support/helpGuides/nt8/index.html?how_bars_are_built.htm#UnderstandingWhyAChartCanLookDifferentAfterReloadingHistoricalDataFromTheServer

[ToNovy]

funny.. under the link it is written that the problem is rather on my side, but that is not entirely true
real time data still looks the same, only historical data has changed
from my experience it used to be fine 1/2 year ago and the whole year before that the data was almost 99% correct but it seems you don't want to hear it
ok, is it at least possible to record real data and use it instead of historical data from HDS? because the quality of historical data from NT is very bad, it is currently unusable for me

[NinjaTrader_Jason]

You can record all realtime data that is received by enabling the 'Record live data as historical' setting at Control Center-->Tools-->Options-->Market data. However if you connect to broker/data feed that provides historical data, unfortunately it won't work. It will overwrite the data you recorded once you connect and historical data is loaded.

I suggest to go to Control Center-->Tools-->Options-->Market data and enable 'Show Tick Replay'. Click OK. Create a new chart and enable 'Tick Replay' within the Data Series menu. Check if realtime data and historical data will better match in the chart.

[ToNovy]

ok, probably the last question because I discovered unpleasant differences for me even on historical minute data
I have a small PC that runs 24/7, is it possible to turn on real time data recording as historical (both tick and minute)? and then copy this data and use it as historical data on the main PC that does not run 24/7?
by the way... I compared the difference in real time tick data with my colleague approx. 350km away and they are basically identical, so there is no problem with latency, connection, PC etc...
the only problem is with how NT stores this data

[NinjaTrader_Jason]

Yes, this is possible. You need to avoid right clicking in the chart and select 'Reload All Historical Data' and avoid disconnecting/reconnecting your broker/data feed connection to avoid the realtime data from being overwritten.