Historical data requests timing out prematurely.

[pjsmith]

Hi,

Given Ninjatraders historical data servers for FX and CFD have been unreliable of late and are missing data, I have been attempting to use the broker side historical data. This mostly works, except it is very slow as NT8 seems to request/load far more data than it really needs, especially on cold start.

The buiggest issue is that the request timeout seems too short as I frequently get this error



Is there anyway to increase it beyond 30 seconds? This is never enough time for a day or 2 of tick data to load and you end up with gaps all over your charts.

fwiw - I cannot reduce the data requested on my chart and I am never requesting more than a few days of tick data (as the rest is already in my historical data cache)

Thanks.

[NinjaTrader_LuisH]

Hello pjsmith ,

Thank you for writing in.

Just to confirm - who are you requesting data for? The verbiage for it seems to imply that NinjaTrader asked for the data and their server did not reply or did not reply in time.


I look forward to your response.

[pjsmith]

In this example, it is FXCM. I agree with what you are saying - That NT did not get the response in time, but I think that's because the connector is simply so inefficient and slow vs the native connectors like ninjatrader historical data connection, or native apps direct to FXCM requesting the same data. If it waited longer, I think it would work as the other platforms and native platforms get the data fine. In fact, NT will as well as long as you request it chart by chart. If NT waited a bit longer, I think it would likely be complete. 30 seconds is not that long for a slow connector I think. It's on startup where all charts are refreshing this occurs, and it basically stops you from getting the platform up and running at all.

[NinjaTrader_LuisH]

pjsmith,


Thank you for the clarification.

What data set are you requesting from FXCM? I will attempt to reproduce this issue on my end in hopes of shedding some light.


Thank you.

[pjsmith]

Well, I honestly have no idea. I would assume on startup, Ninjatrader would be requesting the data from its last save (given that is all stored, recorded, and cached historical data from it's prior use), to now, which is often only a few minutes. Maybe a few hours if overnight. I do have some minute charts that go back 400 days, but again, all that data is already on my side in the historical data files. And even minute data for that long is not that much. For tick data, I go back MAX 5 days, which again, is not much really and certainly, 95% of that is already cached on my side.

So, the data I am expecting NT to be requesting is quite small. Certainly, if I flick to NT8 historical servers, it works and only takes a minute or so to start up. The problem is that the data on those servers is broken and even completely missing overnight data etc., of late. There is missing data on them now, so the charts have gaps.

I am simply trying to find a working, tolerable solution. NAS100 is the symbol I am mostly trying to get. It seems if you have multiple charts in your workspace for this symbol, the startup of NT8 is the biggest issue.

And I thank you for attempting replication of the issue.

[NinjaTrader_LuisH]

pjsmith ,

Does the same behavior occur if you use a lower amount of 'days to load' on a brand-new workspace with nothing other than the NAS100 chart?


I attempted to replicate this on my platform and it loaded as expected but only using the chart. I believe this happens to you when you are loading multiple instruments at once with various days to load that goes beyond the adaptor's capabilities.


Please advise.

[pjsmith]

As I have used the platform and all data was up to date, say, as of 5 minutes ago, if I close the platform and re-open it, it should only be requesting a very small amount of data, no? Or does this historical data cache only apply to data directly from Ninjatraders own historical data servers? I was sure I had read this was not the case. Yet the issue still occurs. I only have 6 NAS100 charts open. 1 SPFX 500 chart, 1 US30 Chart, and 1 GER30 chart. That's it. None of them should be requesting more than today's data given everything else is already loaded and stored in the cache, right? So, really, what would it matter what the chart duration is set to? As mentioned, though - 5 days tick data for nas100 is the most days back I have. Everything else is minute based. I do have a chart that goes back 400 days for the 4 hours (which comes from minute data). I will try reducing this, but to me, that seems to be (or should be), irrelevant.

At the moment, if for any reason the platform closes, or shuts down, it's quite difficult to get it back up an running during RTH. Not so much out of hours (where I guess data is a bit quicker in forthcoming).

So, I guess either Ninja is requesting a ton of excessive data it does not really need (given it should already be cached), or the connection is really, really slow and timing out due to the default 30 second timeout.

[NinjaTrader_LuisH]

pjsmith ,

I've confirmed with our Product Team that it is in fact just the adapter offering slow bits of data.

This is especially prevalent when it comes to tick data.


Please let us know if there's anything else we can assist with!

[pjsmith]

Hi - So I was looking into this a little further.

I note - The Ninjatrader application appears to be using a BETA (!) version of the FXCM FORREXCONNECT API, per the below. It is quite old. The FORREXCONNECT API was updated about a year ago. In the release notes/comments, it is stated about significant improvements, as well as quote 'Improved stability on all platforms', as well as a margin bug fix.

Given this is likely a very simple upgrade (switch out the library for the new version), perhaps this would be wise to pass on to your dev team. Thanks.

I tried my self to switch out the library, but it seems with the NT8 encryption and other hooks it is not that simple. Looking forward to knowing if this can be done (as indeed it certainly should be). Thanks.