Error importing tick data

[gregid]

I went full vanilla route for these tests.
There are few issues around importing of the tick data:
1. For forex data instrument name has changed - importing old NT7 name, eg. $EURUSD causes: "$EURUSD.txt. Instrument is not supported by repository." error. Removing the $ fixes the issue of not supported instrument.
2. Importing tick data is unsuccessfull - see the screenshots:

Attached is also sample file used for import

[NinjaTrader_ChelseaB]

Hello gregid,

Thank you for reporting this issue.

I am able to confirm the issue on my end and I have forwarded this to development.

[NinjaTrader_Brandon]

Hello gregid,

I have been able to properly import the data by changing the file name to $EURUSD.Last.txt

[gregid]

Hello gregid,

I have been able to properly import the data by changing the file name to $EURUSD.Last.txt

Thanks,
The filename that worked for me was:
EURUSD.Last.txt

($ sign removed)

[gregid]

out of memory and db crash on import

Import started working with the right file name, unfortunately it seems the import process takes too much memory (more than NT7).

The file I was importing was 2.2GB and I used to successfully import the same file in NT7.
Importing in NT8 I got out of memory exception, then NinjaTrader crashed. Now I can't open NinjaTrader as it crashes during initialization of database - so the interruption obviously corrupted the db.
I am fine with installing NT8 again since this was vanilla setup but I recommend some profiling on the process and exception handling to avoid corrupting the db.
The machine I was testing with has 16GB Ram so it shouldn't be an issue.

[NinjaTrader_Brandon]

Hello gregid,

Thanks for your reply.

Please send the file you are attempting to import to platformsupport[at]ninjatrader[dot]com
Include a link to this forum topic and send it using the following website.

https://www.wetransfer.com/

[gregid]

Hello gregid,

Thanks for your reply.

Please send the file you are attempting to import to platformsuppport[at]ninjatrader[dot]com
Include a link to this forum topic and send it using the following website.
https://www.wetransfer.com/

wetransfer.com has 2GB limit, so I will test on slightly smaller size first before sending.
Also I didn't have to reinstall NinjaTrader to start it - deleting NinjaTrader.sdf was sufficient for successful start

[gregid]

Update

I did another test - ensuring nothing else is running on my machine and controlling memory/process usage. This time tested another file USDCAD.Last.txt slightly smaller then the previous one.

The import process was taking on average 37% of my CPU but memory only increased finally when reaching 98% (about 14GB) it crashed my system - my Windows 8.1 display stopped, have lost input control (mouse/keyboard) and I will have to physically restart the machine if it won't recover - even if it recovers I expect out of memory info and db crashed.

I am in the process of transferring the file with wetransfer (uploading) - so you should receive it shortly. I forgot how good text file compression is, so the transferred zip file is only about 250MB

[NinjaTrader_Dierk]

Curious: how many data points are we talking here?

Please note that tick data import with NT8 takes more RAM than with NT7 (NT8 has more feature like e.g. subsecond support) -> a large file like yours which you could import no problem with NT7, might fail for RAM reasons with NT8. As a rule of thumb I would 'guess' (educated) 20-30% more RAM is reasonable to expect. I would be surprised though if it e.g. would be double the RAM...

-> I recommend splitting the file up.

[gregid]

I'll give file statistics shortly - in the meantime - the email address provided seems to be incorrect: platformsuppport[at]ninjatrader[dot]com - I thought maybe it is on purpose but will try sending again to:
platformsupport[at]ninjatrader[dot]com

Can you confirm the email address is right?

[NinjaTrader_Brandon]

Hello gregid,

Thanks for your reply.

That is correct and I have fixed the typo. The correct address is platformsupport[at]ninjatrader[dot]com

[gregid]

I used to import the same file on 4GB machines and even smaller VPSs - no problem at all. The way I see is it looks more like a memory leak

[NinjaTrader_Dierk]

Understand. We'll look into then. Thanks for reporting.

[gregid]

OK, you should got the file already. Since I have no doubt you were testing the import on larger files here is a clue of what was different in my case - after I finally recovered my system
I've noticed huge amount of errors in log file with error like the following:

2015-05-06 13:45:20:748 ERROR: USDCAD: Illegal ask value in line 8379062 at 25/03/200820:19:04. Ask value needs to be greater/equal close value: 20080325 201904;1.01785;18600000

So the memory issue may be simply due to errors and error logging.

[NinjaTrader_Dierk]

Thanks for making us aware.