Forex.com Current Daily Bar

[kevinenergy]

Hi there,

There is an issue with Forex.com whereby historical data does not deliver today's in-progress bar. Today's bar starts printing when the first tick arrives after the chart loads.
Clearly this is not expected behaviour.

Steps to Reproduce:
1. Disable Ninjatrader Historical data on a Forex.com/CityIndex account (NB: NT data is missing for many instruments, so it needs to be disabled when trading with this provider)
1. Connect to aforementioned Forex.com/CityIndex account
2. Open EURUSD on the daily timeframe


Expected Result:
- Chart loads with historical data including today's partially-completed daily bar which continues to print as ticks arrive.

Actual Result:
- Chart loads with all bars until yesterday, but Today's bar is missing. Todays bar starts printing afresh when the first tick arrives after chart loads.

FYI I have confirmed by making REST requests in POSTMAN to the API for historical daily data, that confirms that the today's bar is being delivered. The Close price updates as expected between subsequent requests. I have also confirmed the same with the StoneX C# API.

As a side note.... Please don't log this is a "feature request" - which seems to be common practice these days when bugs and issues are reported. This is an issue and should be treated as such.

Many thanks,

Kevin

[NinjaTrader_Zachary]

Hello Kevin,

Thank you for your post.

I can confirm that I experience the same with the steps provided using Forex.com's own historical data servers. I have a few questions about this for you;

Does this issue affect all instruments for you or just EURUSD?

Does the issue persist for you if you right-click on the chart and select 'Reload All Historical Data'?

Has this always been an issue when using Forex.com's historical data servers or did this issue recently begin to present itself? If so, when?

Do you experience the same if you use a 1440 Minute chart or does this only affect Daily data?

I look forward to your response.

[kevinenergy]

Hi Zachary thanks for the response.

Does this issue affect all instruments for you or just EURUSD?

It affects all instruments from Forex.com / CityIndex, including all Forex Pairs and CFDs

Does the issue persist for you if you right-click on the chart and select 'Reload All Historical Data'?

Re-loading historical data does not resolve the issue. The bar is always missing from the historical data. In fact if I leave the chart for a few hours and allow the daily bar to start painting then reload historical data, the today's bar is cleared and starts repainting again from scratch when the next tick arrives.

Has this always been an issue when using Forex.com's historical data servers or did this issue recently begin to present itself? If so, when?

Its been happening for some time, definitely not a recent thing, But as the markets have become a lot more volatile over the last month I've found myself wanting to see the daily chart more often and it has become a particular annoyance.

I'm still on 8.0.24.3. I did test Ninjatrader 8.0.25.0 using Hyper-V to see if it had been fixed but it hadn't. I prefer not to upgrade to X.X.X.0 releases unless there is a feature or fix on the release notes that I need/want.

Do you experience the same if you use a 1440 Minute chart or does this only affect Daily data?

The issue of Todays bar is resolved by using a 1440 minute chart. But in doing so I only get 1 month of visibility as Forex.com only provide 1 month of historical minute data, so this is the most you can ever see:

[NinjaTrader_Zachary]

Hello Kevin,

Thank you for your reply and for providing this information.

Since this issue is not present when using the NinjaTrader Historical Data Servers, then this must be a limitation with their servers specifically. We don't really have much control when it comes to third party historical data servers since we do not maintain them. Since that's the case, the only thing we could do is try to work around it, either by using our historical data server or a 1440 Minute chart.

However, I will say that I'm seeing slightly different behavior than you are, from what you've described. I am able to load the full Daily candle for today when I right-click and select 'Reload All Historical Data' on the chart after I initially create the chart. After I select 'Reload All Historical Data' on the chart, the candle populates correctly from then forward. I've included a video to demonstrate;

https://www.screencast.com/t/HSuobssxd6X5

This worked for me unfailingly, I was able to load today's candle in this way for any instrument that I tried.

Please let us know if we may assist you any further.

[kevinenergy]

Hi Zachary,

I confirm that I can get the daily bar by doing "Reload all historical data". I think when I first responded I was actually choosing the "Reload Ninjascript" option by mistake.

I disagree with your hypothesis that "because the issue is not present when using Ninjatrader historical data servers that this must be a limitation on their (Forex.com) servers", for the following reasons:

- Their historical data servers work fine in my own tests with their REST API and their C# API, their response delivers the OHLC for the partially complete latest bar in the data, and if I perform subsequent requests I can see the close price changing, this indicates that their API is returning today's bar along with the prior historical data
- Historical data is presented correctly without any need to reload on other platforms that use the Forex.com API, including TradingView, MetaTrader, WebTrader, ATPro, their Mobile Apps.

This problem is unique to Ninjatrader.


Furthermore I have performed the following tests:

TEST 1 - Purge historical data and load a blank chart (FAILS)
--------------------------------------------------------------------------------
1. Create a new blank workspace 'historical data test'
2. Choose Tools -> Historical data
3. Delete all historical data for EURUSD (or any instrument)
4. (optional) restart Ninjatrader to create a new Trace file to make it easy to find
5. Start Ninjatrader and Load chart EURUSD (or same instrument as above)
RESULT: Incomplete daily bar

Trace file for data request:

2022-02-14 09:31:30:280 Cbi.Instrument.RequestBars (to Provider): instrument='@UK100' from='11/11/2017 00:00:00' to=[COLOR=#c0392b][U][B]'13/02/2022 00:00:00[/B][/U][/COLOR]' period='Daily'
2022-02-14 09:31:31:310 (XXXXXXX) Gain.Adapter.GetG2MarketId instrument='@UK100' marketId='99500'
2022-02-14 09:31:31:620 (XXXXXXX) Gain.Adapter.GetG2MarketId instrument='@UK100' marketId='99500'
2022-02-14 09:31:31:943 Cbi.Instrument.RequestBars (to Provider): instrument='@UK100' from='14/02/2022 00:00:00' to='15/02/2022 00:00:00' period='Daily'

TEST 2 - Purge historical data then Load historical Data using the Historical Data tool (SUCCEEDS)
--------------------------------------------------------------------------------
1. Repeat steps 2-4 from previous test, up to delete prior historical data
2. Open Tools -> Historical Data tool
3. Click "Load" tab, choose EURUSD
4. Select Daily bars, Deselect Minute bars, Request one year of data: 14/2/21 - 14/2/21
5 Click Download
6. Open new chart, choose EURUSD Daily chart, request 360 bars (just less than we requested before)
RESULT: Todays bar is loaded and complete

Trace File:

[B][I]Data Load:[/I][/B]
2022-02-14 09:46:43:203 Cbi.Instrument.RequestBars (to Provider): instrument='EURUSD' from='14/02/2021 00:00:00' to='15/02/2022 00:00:00' period='Daily'
2022-02-14 09:46:43:607 (XXXXX) Gain.Adapter.GetG2MarketId instrument='EURUSD' marketId='400494226'

[B][I]Chart Open:[/I][/B]
2022-02-14 09:49:55:530 Cbi.Instrument.RequestBars (to Provider): instrument='EURUSD' from='03/08/2020 00:00:00' to='[COLOR=#c0392b][B][U]13/02/2022 00:00:00[/U][/B][/COLOR]' period='Daily'
2022-02-14 09:49:56:732 Cbi.Instrument.RequestBars (to Provider): instrument='EURUSD' from='14/02/2022 00:00:00' to='15/02/2022 00:00:00' period='Daily'

TEST 3: (Purge historical data, and load a blank chart (FAILS) then "Reload historical Data" (SUCCEEDS)
--------------------------------------------------------------------------------
1. Repeat steps 2-5 on Test 1, up to loading chart with incomplete daily bar
2. Choose Reload historical Data
RESULT: Reload historical Data completes the daily bar

Trace file

Chart Load - with Incomplete Daily Bar
2022-02-14 10:02:00:108 Cbi.Instrument.RequestBars (to Provider): instrument='EURUSD' from='11/11/2017 00:00:00' to='[B][U][COLOR=#c0392b]13/02/2022 00:00:00[/COLOR][/U][/B]' period='Daily'
2022-02-14 10:02:00:408 (XXXXXX) Gain.Adapter.GetG2MarketId instrument='EURUSD' marketId='400494226'
2022-02-14 10:02:02:621 (XXXXXX) Gain.Adapter.GetG2MarketId instrument='EURUSD' marketId='400494226'
2022-02-14 10:02:02:709 Cbi.Instrument.RequestBars (to Provider): instrument='EURUSD' from='14/02/2022 00:00:00' to='15/02/2022 00:00:00' period='Daily'

Reload Historical Data:
2022-02-14 10:04:06:810 Cbi.Instrument.RequestBars (to Provider): instrument='EURUSD' from='12/04/2018 00:00:00' to='15/02/2022 00:00:00' period='Daily'
2022-02-14 10:04:07:124 Cbi.Instrument.RequestBars (to Provider): instrument='EURUSD' from='14/02/2022 00:00:00' to='15/02/2022 00:00:00' period='Daily'

Observations from these tests:

- When Ninjatrader opens a chart, it makes two BarsRequests:
-----Bars request 1: from N-Days ago until "Yesterday", which in the above tests is 13/02/22
-----Bars request 2: from Today until Tomorrow. (which in case of these tests is 14/02/2022 - 15/02/2022

- When loading data via the historical data tool, it does things differently:
----Bars request 1: from N-Days ago until "Tomorrow" which in the case of these tests tomorrow is 15/02/2022

- When loading data via "Reload historical Data" it does the same as the historical data tool, but does an additional request for "Today - Tomorrow"
----Bars request 1: from N-Days ago until "Tomorrow" which in the case of these tests tomorrow is 15/02/2022
----Bars request 2: (only in the case of "Reload historical data) From Today until tomorrow, which in this case is 14/02/2022 - 15/02/2022

Based on these tests, and my observations, there is a clear difference to how Ninjatrader is handling historical data loading for Chart Opening. Why does the initial request stop at "Yesterday" (This is not how futures instruments on Ninjatrader Brokerage Loads their data, which Loads from N-days back until 'tomorrrow')

I would hypothesise that Forex.com's historical data works just fine, and there is a bug in Ninjatrader's code whereby on chart opening, the second request ("yesterday to today") is being made but the response is not being handled correctly.

Clearly the data is coming from Forex.com, However, Ninjatrader is failing to present it on the chart in a Chart-load scenario.

Just FYI: The reason I don't upgrade to Ninjatraders latest version any more is I got fed up with having to waste my own personal time on diagnoses like these to persuade you guys that there is a bug in Ninjatrader. Doing these tests and writing-up this analysis has taken 2 hours out of my day this morning.

I Just hope I've now sufficiently persuaded you that this is a Ninjatrader issue and not a Forex.com issue, and you will raise the issue with the dev team and get it fixed.

Many thanks,

Kevin

[NinjaTrader_Zachary]

Hello Kevin,

Thank you for your detailed response.

I've opened a request with our development team regarding this, and am awaiting further information about the nature of the problem.

[kevinenergy]

Zachary,
No problem, I am more than happy to spend my time helping to investigate things like this if it ultimately leads to improvement of the platform. Sometimes things that are clearly bugs are dismissed as 'expected feature' and its quite frustrating.

So I appreciate you taking the time to read my findings and raise it with development.

Many thanks,

Kevin.