Importing data from a very large csv excel file

[pstrusi]

Hi Ninjas,

I've found a reliable source of historical data which I'd like to import in my database. This file comes as a CSV Excel file, and its data comes in a non-format NinjaTrader, therefore I have to edit it before converting it to a .txt file and later importing it into NT's database.

So, this is the situation: I need to edit each line of that file cause it comes in the following format: EUR/USD,20130401 00:00:00.258,1.29885,1.2989

In order to edit this data into NT format data as: 20130408 040000;1.29885;1000000

I'd need a software capable to open a massive csv file ( 467 MG ), and editing automatically for millions of lines. I've seen software capable to open massive csv files but to edit some lines "on demand" and not the whole file.

Is the original data format, some format known by you ( by NT ) which could easily import ? or must I look for to do all this job of editing, converting...etc ?

I'd like to hear any suggestions from your experience and knowledge to achieve this

Thanks in advance

[NinjaTrader_JC]

Hello pstrusi,

I am not sure what you mean by: "Is the original data format, some format known by you ( by NT ) which could easily import ?" but the only import formats that NinjaTrader accepts natively is the ones on our Help Guide at the following link that it looks like you are aware of.

http://www.ninjatrader.com/support/helpGuides/nt7/importing.htm

The only other way you can import it without converting the file would be to write an Import Type that will read a file like that so when you get data in that format you just have to select that Import Type. A few members have written some Import Type files that you may view at the following link.

http://www.ninjatrader.com/support/forum/local_links.php?catid=6

Let me know if this makes sense or if you can clarify about the "known format".

[pstrusi]

Thanks for your response JC.

I'm sorry not to have properly explained what I meant. So, let me explain myself:

When I export data from NT, the coming file is a .txt file with the following format:
20130408 040000;1.29885;1000000 That's what I called the "original NT format". Now, in the market there are several types of format, many of them come in CSV files, with a format like this:
EUR/USD,20130401 00:00:00.258,1.29885,1.2989

I've read the NT help guide and basically it accepts the typical .txt file with the NT classic format, or others like Metastock or TradeStation; so my conclusion is that I need ac .txt file with the classic NT format, so I had to edit this massive file that I have as CSV excel file, but it's so large that I cannot do it through excel...so I ask you for your help:

What software can I use ?

[NinjaTrader_JC]

Hello pstrusi,

Thanks for the clarification.

I have seen people use "Notepad++" to be able to do this but I am not sure what the exact size limit is which you may try to use since it is free at the following link:

http://notepad-plus-plus.org/

The one that is the most common that I use as well for large files is going to be "Visual Studios Express 2012" which is free to use.

http://www.microsoft.com/visualstudio/eng/downloads#d-2012-express

Happy to be of further assistance.

[pstrusi]

Thank you JC for your help, hopefully these suggestions could help me