Getting up and running with a massive forex tick data history

[adamus]

Hi All,

I'd really appreciate it if someone could give me a couple of pointers so that I don't make a mess of my trading set-up.

I just downloaded NT.6.5 over the weekend and simultaneously bought 12 years of tick data for 10 forex pairs to allow me to do the backtesting I am planning.

It takes a long time to import the tick data, I've been working for about 8 hours today downloading, decompressing, renaming and importing tick data files just for EUR:USD.

So you can appreciate that I don't want to get it wrong and have to do it again.

Anyway, when I get the data for the other 9 pairs loaded, the plan is to configure NinjaTrader to download the tick data real-time and keep the database up-to-date for backtesting and trading.

How should I set this up? Do I have to have a chart window open for each forex pair all the time? Is there a special document on this on the NT website?

Secondly, back-ups: what do I need to backup to keep all this stuff safe?

Thanks in advance!!!

[NinjaTrader_RyanO]

Hello adamus,

Thank you for your post, and welcome to the forum.

Once you unpack all the data you will want to make sure that you have a data provider that offers real-time data. Please see the following link.

http://www.ninjatrader-support.com/HelpGuideV6/helpguide.html?HistoricalData

You can get free real-time forex data from GAIN, and since you have the historical data you will want to make sure you specify that you will get access from another data source. To create the connection please refer to the following link.

http://www.ninjatrader.com/webnew/support_GAIN.htm

You can open a chart for each pair, but likely you will want to add each instrument to record real-time data for using the Market Analyzer.

http://www.ninjatrader-support.com/HelpGuideV6/helpguide.html?Overview38

There is a backup feature that you can use which is available by going to File-->Utilities-->Backup. When backing up you will want to specify that you backup the database, as the historical data is stored there.

http://www.ninjatrader-support.com/HelpGuideV6/helpguide.html?CreatingABackupArchive

[adamus]

Thanks for the reply.

I have an account with Interactive Brokers which I have successfully hooked up.

I am a total NinjaTrader newby, so I haven't actually worked out everything that MarketAnalyzer does, but from initial impressions it doesn't look like the part of NT that I need.

Does it handle backtesting NinjaScript-coded strategies? That's my aim but I didn't see how.

[NinjaTrader_RyanO]

Adamus,

The part that backtests custom strategies would be Strategy Analyzer. I suggested the Market Analyzer as a method to capture real-time data that would be stored in the database, in addition to your historical data.

The Strategy Analyzer would take advantage of that historical data for backtesting.

Please see the following link.

http://www.ninjatrader-support.com/HelpGuideV6/helpguide.html?Overview37

[adamus]

OK, I'm with you now.

Do I have to keep the Market Analyzer window open all the time then?

Should I set the "# number of bars to look back" or does that have no impact on data collection? I won't be doing any analsys in this window.

Another question: if something happens and my computer goes down for X hours, will this leave a gap in my history? Or will NinjaTrader automatically detect the last timestamp and fill up the history from there?

Thanks!

[NinjaTrader_RyanO]

Adamus,

If you wish to pull data for a particular group of instruments then you will want to the Market Analyzer open.

The bars look back can be set if you are only interested in real-time data by setting this to 5 or so, and if you want to pull data from times that you are not connected then having the Market Analyzer set for the times when you are not connected.

[adamus]

Just to get this straight, I want to make sure I collect an uninterrupted history for these 10 forex pairs that I'm going to be trading.

If this involves doing stuff that I'm likely to forget like making sure a particular window is open all the time, then it means it's only as reliable as I am! (No smart comments please)

What exactly do you mean when you wrote:

having the Market Analyzer set for the times when you are not connected.

Thanks again

[NinjaTrader_RyanO]

Adamus,

When you first connect Market Analyzer will pull historical data from the times when it was not connected.

You can create a workspace so that whenever you open NinjaTrader the Market Analyzer with the 10 forex pairs are already present, and then all you need to do is connect. With that it will save your lookback settings, as well.

Please refer to the link on workspaces.

http://www.ninjatrader-support.com/HelpGuideV6/helpguide.html?ManagingWorkspaces

[adamus]

I launched the backup of the database this morning and it is still running now 10 hours later.

I've only imported one symbol into the database - 12 years of tick data for the Euro USD forex.

Admittedly the computer is not new - it's an AMD Athlon XP 3000+ 2.17 GHz with 1.50 Gig of RAM but surely I don't have to consign it to the scrap heap now?

[NinjaTrader_RyanO]

Hi Adamus,

If the backup procedure is frozen you can go to My Documents-->NinjaTrader 6.5-->db and create a copy of the NinjaTrader.mdb file.

That file is the database, and if you make a copy of it at various points you can then use it to replace later modified files.

[adamus]

I took a bit of time out from worrying about the data collection issue to put my strategies together, but now I've moved my attention back to the data issue and I've got a problem.

I've got 10 forex pairs in my default list and I'm still loading the tick data I bought for the last 10 years into the Historical Data Manager.

I need to keep this database up to date so I can use it for backtesting more strategies in the future, so to this purpose I've put the instruments in the default list into a Market Analyzer window.

However in the Historical Data Manager I see for instance in the Edit tab that there are only entries for instruments that I imported data for myself, and there's no sign of the data that the Market Anaylzer is meant to be collecting.

Shouldn't I be able to see the data for these other symbols already?

Thanks

[NinjaTrader_RyanO]

adamus,

When you add those instruments to the Market Analyzer it populates the bid/ask and last data.

Also, are these instruments in the Market Analyzer different than the ones you have imported data for.

[adamus]

Market Analyzer puts that data in the historical database for all instruments its displays?

It doesn't appear to be doing that.

I'm using only 10 forex pairs / instruments. I have imported data for 7 of them. These are the ones that appear in the HDM. The others don't.

[NinjaTrader_RyanO]

adamus,

Please go to Tools-->Options-->Data and please make sure that Store Real-time bar data is enabled.

[adamus]

Ryan,

here's my screenshot. In the "Real-time data" panel, 'record for market replay" was not checked, but I have checked it now. The other two were already checked.

I still don't have any entry for the instruments I'm talking about in the HDM.