running backtests with strategy analyzer while importing historical data

[Jason Fornkohl]

I have 3.5 TB of data to import into the historical database. It's a lot of data as it is tick data for many years of over 900 instruments. I've estimated it's going to take around two months to complete the import based on it's current import speed.

I only have one computer, and I'd like to continuing working on developing strategies with Ninjascript code while I'm waiting for the import to complete. For these strategies, I'm wondering if running backtests via the Strategy Analyzer is okay, while at the same time, data is being imported into the historical database. I tried a backtest while importing historical data and everything seemed fine, but I did not run the backtest on any instruments that were currently being imported.

If ever, under what circumstances is it safe to run backtests on historical data while, at the same time, importing historical data into the database? I do not want to have to worry that my imported data could get corrupted while importing.

[NinjaTrader_BrandonH]

Hello Jason Fornkohl,

Thanks for your post.

You would need to first import the historical data in question and then backtest the strategies in the Strategy Analyzer window.

You could consider importing historical data for a single instrument or a few instruments at a time. After importing data for an instrument, you could run a backtest on that instrument while importing historical data for a different instrument(s).

It would be expected that importing that large amount of historical data would take a while to complete.

There would be no means to increase the speed at which historical data is imported onto NinjaTrader.

[Jason Fornkohl]

Brandon,

That is what I was trying to ask, if it is of no concern to run a backtest on instruments that are not currently being imported. Thanks for the guidance on that.

You mentioned there would be no means to increase the speed of the import. Do you think upgrading hardware could help in any significant way? I've noticed my C: drive is growing and shrinking in size as the import process runs (as if it is being used as a cache for the import process?), even though I have the database on a different drive entirely. I've mapped my "C:\Documents\Ninjatrader 8" folder to a different faster NVME SSD drive, but my C: drive is an old platter drive which is not as fast as an SSD. I was wondering if I replaced my C: drive with a fast NVME SSD if that could speed up the process. Do you think there could be a chance that could maybe speed up the process a little? Or maybe a faster processor, etc? I'm assuming even with hardware upgrades it's going to take a considerable amount of time. Any idea, in general, what would be the main bottleneck for the import process? Maybe my question here is impossible to address without monitoring a particular PC's performance.

[NinjaTrader_BrandonH]

Hello Jason Fornkohl,

Thanks for your notes.

Yes, upgrading to an SSD or NVME M.2 drive could likely increase the speed that the historical data is imported into NinjaTrader. This is because the read speeds of the drive are faster than a standard drive.

When the historical data is imported it is duplicating the data into NinjaTrader format so if you are importing 4TB of data then you should have more than 8TB of avaliable space to store and import.

Also, this means you need to put that data on the same drive where NinjaTrader is installed. If your C drive is a regular drive and you upgrade it, you need to move the txt files to the C drive before importing. You will gain no additional performance during import if you used a different drive for the data and import from that drive.

Further, you might also not get much of an improvement because importing that amount of data causes a heavy CPU process. If your CPU is the limiting factor you might not even be hitting your regular drives read speed.

You could use the Windows Task Manager application to see if 100% of your CPU is being used or paging. If so, then new drives would not help with performance.

[Jason Fornkohl]

Interesting conversation.

My setup is as follows:

C drive (old slow platter drive) -> Ninjatrader installed in this drive but "C:\Documents\Ninjatrader 8" folder (which holds the historical database location) mapped to the F drive, maybe the import process uses this drive as a workspace for the import data conversion to the Ninjatrader format

F drive (NVME SSD) -> holds the data for: "C:\Documents\Ninjatrader 8", import process writes imported data to this drive

H drive (SATA SSD) -> tick data stored here waiting to be imported into the database which lives on the F drive, import process reads from this drive

When initially investigating, I was surprised to see the C drive growing and shrinking during the import process, but the F drive not changing in size. The F drive does not change in size until the import process for an instrument finishes, and then it updates the F drive size by showing an increase in drive space used.

At this point, I think I'm just going to wait the import process out. I'll make good backups of the database once the import process is completed, so I hopefully won't have the need to ever do this again. It sure is a long process to complete, but for this much data, it makes sense as to why it takes so long