Trying to import a ~6GB file of ES tick data using NT 7 beta 3 with generate minute bars and generate day bars selected, and not having much luck.
By the time I killed it, NT 64-bit was using 4GB RAM (VM size 4.2GB) but had only read 1.5GB and written < 1MB (i.e. almost nothing) according to Task Manager. I killed it as it had slowed to reading only 1MB ever ~3 seconds on a machine with 6GB RAM.
NT 32-bit had the same problems except it used somewhat less memory (2.5GB RAM, 2.7GB VM) to read 1.4GB. Killed it as it also slowed to a crawl.
Currently re-running it without generate minute bars & generate day bars selected and the RAM usage looks better (~1.5GB RAM to read 3.5GB), however it still seems to be trying to read everything into RAM before writing anything to disk, so I expect it will die eventually.
Note the tick file is named "ES 09-09.txt" (i.e. the NT 6.5 name; the NT 7 docs suggest it should be called 'ES 09-09.last.txt').
Can NT process the data in chunks rather than trying to read it all upfront? The current method doesn't scale well.
-Swig-
By the time I killed it, NT 64-bit was using 4GB RAM (VM size 4.2GB) but had only read 1.5GB and written < 1MB (i.e. almost nothing) according to Task Manager. I killed it as it had slowed to reading only 1MB ever ~3 seconds on a machine with 6GB RAM.
NT 32-bit had the same problems except it used somewhat less memory (2.5GB RAM, 2.7GB VM) to read 1.4GB. Killed it as it also slowed to a crawl.
Currently re-running it without generate minute bars & generate day bars selected and the RAM usage looks better (~1.5GB RAM to read 3.5GB), however it still seems to be trying to read everything into RAM before writing anything to disk, so I expect it will die eventually.
Note the tick file is named "ES 09-09.txt" (i.e. the NT 6.5 name; the NT 7 docs suggest it should be called 'ES 09-09.last.txt').
Can NT process the data in chunks rather than trying to read it all upfront? The current method doesn't scale well.
-Swig-
Comment