Downloading large amount of tick data take infinit time

[bourasrafik]

Hi,
I'm trying to test my strategy using tick data from dxfeed, want to test for long time (6 months and 12 months), downloading data from historical page or through strategy analyzer take very very long time (infinite ?)
in the download page it saus (MES download) --> Loading Data ... MES03-22(2021-12-28 - 2022-03-08)
How can I track wha't's going on behind is small message? is it really downloading ? is it the speed provided by Dxfeed which is slow ? How to prove it ?
Thanks

[NinjaTrader_Manfred]

Hello bourasrafik,

Thank you very much for your post.

Downloading historical data, especially tick data, can take upwards of a few minutes if several months or years are being downloaded.

However, you can check what data has been downloaded within the Replay window directly. If you see that historical tick data is available for these dates you should experience no issues using it for the replay.

Unfortunately, only the log files themselves would show the data that is currently being downloaded while this is still in progress.

If you are using historical data, an easy way to ensure this data has been downloaded would be to create a tick-based chart. E.g. by creating a 10,000-Tick chart loading 2 years of data, the platform would then request the historical tick data and download it, if available.

Please don't hesitate to let me know, should you have any further questions or if you encounter any issues!

[bourasrafik]

Hi manfred,
1- I tested the tick downloads and something does not match, as you see in the attached image, I displayed 10K ticks chart 2021 range (full year), the candles are displayed, we can expect that the data on contracts (like MES 09-21) should be available in historical Data panel, as you can see, no tick data is added to the contracts even if data is displayed, seems not normal, could you explain this behaviour ?

2- Could you show me how display log to see in real time which data is downloaded to ensure that all data is done and have an idea of download speed?

3- On behaviour that seems not normal is the RAM usage when data is downloaded, I understand that displaying a chart requires that all data is loaded in RAM, in Historical download , it downloading data than store them into files in the hard drive (and should free memory), freeing memory is not done, downloading 1 year of tick data consumes all my 32GBytes of RAM that I have, get 95% usage and seems stop downloading, need to restart to get a stable system again. this behaviour seems not correct as no indications is given to follow up the download process to continue where it lefts.

Thanks for your support

[NinjaTrader_ChristopherJ]

1. When viewing data for the date 9/21/2021 you would want to look at the available data for the expiry trading at that time for example in this case ES 12-21 as opposed to ES 09-21.

2. All downloaded data will be displayed within the Historical Data window which can be accessed by going to Tools->Historical Data then within the Historical section under Loaded. Unfortunately there is not a window that displays data as it is downloaded real time.

3. To reduce the load on your PC when downloading Tick data I suggest downloading data in sections such as 3 months at a time as opposed to all at once for 1 year.

[bourasrafik]

Hi christopher,
I understand that it'll took time but I think Ninjatrader does not handle well the download data, at the end, it's stored as an ncd file of 88K~400K, I think Ninjatrader try to download all requested data before writing them to disk which is not very efficient, fo r example ES 03-22 contract, I cannot download even one month of tick data, I have GBytes RAM and even with on month I get 95% memory error, happens through downloading the files or through chart , it does not make sens that with 32GB RAM I cannot display 1 month of tick data data
Thanks for help

[QuantKey_Bruce]

You should not be running out of RAM downloading tick data. There is something else going on.