Getting up and running with a massive forex tick data history

[NinjaTrader_RyanO]

Hi,

With those settings enabled you should have stored the real-time data from the Market Analyzer.

However, can you please open a chart for the instruments that you do not see and then see if that data is loaded into the historical database.

[adamus]

OK this is slightly confusing.

There are 2 tabs in the HDM with my instruments listed - 'Edit' and 'Export'. At one point this morning, the instrument appeared in one tab but not in the other, but then things began to go wrong!

I opened a chart for $EURJPY and nothing happened. I then shut down and reopened NinjaTrader and the instrument was there - but minute data only. When I tried to open a tick chart, NinjaTrader refused to load any past data, and then crashed!

So lots of things going wrong!

I can start the investigation again if you want, by deleting the data for $EURJPY and just running the Market Analyzer. Should I do that or can you reproduce the original problem I'm having - Market Analyzer not pumping data into the HDM?

[adamus]

Forgot to mention, this is using NT7 beta15.

[NinjaTrader_Kyle]

Hi adamus,

So that we may investigate the underlying cause of the crash, please send a note to Support [AT] NinjaTrader [DOT] com with "ATTN: Kyle" in the subject line.

In your message, please include the following:

1.) A link to this forum thread.

2.) Your trace file from the date the crash occurred.
You will find the trace file in the My Documents > NinjaTrader 7 > Trace folder.
The file will be named 'trace.YYYYMMDD.txt'.

[adamus]

I have sent you the requested info.

I have already had a couple of crashes and I think it's probably being caused by a totally seperate issue.

What I'd like to know is whether you can reproduce the problem I'm talking about, or whether you need more help from me on that. I would love to see the issue resolved.

[NinjaTrader_RyanO]

Adamus,

Unfortunately, I can not find the particular trace and log files. If you provide them and reference this thread we can review the crash you are seeing now.

If you do not think they are related then we can determine the precise cause. As for the issue with the historical data that is something we will want to test further. In which case the files will aid in that diagnosis.

I will look for those files.

[Trader.Jon]

Adamus,

What was the source of your back data and how LARGE is your database file?

I think I had the exact same issue over a year ago, and it was confirmed via NT Support, that they were able to reproduce that the historical tick database cannot handle over 800 million lines of data, and will not accept new data once it is full. Reference is RE: Support request {168253}. I was also using ForEx data. NT6.5, but I never did get an answer that it was fixed in NT7 ... guess maybe not.

Overall it took 6 weeks of emails to reproduce the issue. It had nothing to do with the data itself. YMMV A LOT of wasted time and three re-installs of NT and different mixes of datafiles showed consistently the same outcome at almost the exact same amount of data filled the database.

I havent taken the time yet to test NT 7 ... I wasnt invited to the beta program and the support staff were given sufficient data by me for them to develop an answer. The files and data might still be available if they want to review the support ticket referenced here.

I am offering input in this thread to allow a (potentially) quicker efficient resolution and support can have more time to resolve other questions. I appreciate their work that they did with my problem.

Jon

[adamus]

Ryan, Jon,

the data is from disktrading.com (some people deride them because they are so cheap but i've only found a couple of errors in about 25 gigs of data).

The database in NT7 isn't a single file, but the directory where all the files are stored is big. Really big. Not as big as the 25 gigs of text file that I received from disktrading.com, but around 15 Gig.

I had problems importing the disktrading.com data but it was something to do with the data itself - I removed a single line from row 12343553425 (whatever) from the 33 million line file and then it imported. That's another thread and another bug though.

What I'm concerned about here, that the Market Analyzer isn't saving data to the HDM, is something that I'm going to investigate quite intensively today because I have a new computer which I bought just to run NT7.

I would be surprised if the NT6 problem you had remains in NT7. I think it would be interesting to hear the 'official' word from the NT support people though.

re: the email to support and the trace file - I sent it at more or less the same time as I posted the message above, i.e. 2010-05-20 00:45 GMT+0

[NinjaTrader_RyanO]

Adamus,

Likely the two issues are related, and thanks for the input Jon. If you could follow up in the email you sent earlier we can work to reproduce and resolve the issue.

320043 is the issue number we have on record.

[Trader.Jon]

Adamus,

I got my data from the same source. I did find some gaps in 1 day of tick data: I was a little disappointed that it was an ongoing gap unresolved as it was a year back in time from my date of purchase. I never tested the minute based data.
I have NT7b15 installed now, running an optimization ... looks like a 2 day run so I cant test the importing of that data but I will when I can spare the time. I am still intersted in seeing the data available for backtests.

Ryan,
Your welcome. Looking forward to seeing the resolution on this thread.

Jon

[adamus]

Just done a quick test in NT7 beta 16.

I did a fresh install on a new computer (I won't be able to do that everytime!)

I connected to InteractiveBrokers.

I opened a new Market Analyzer and added my forex pairs.

I let it run for a few ticks.

I opened the Historical Data Manager and there in the 'Export' and 'Edit' tabs, there was nothing listed - I had collected no data despite running the Market Analyzer.

I then opened a chart for GBPJPY and when I opened the HDM, the instrument appeared - so charting is collecting data, but Market Analyzer isn't.

[NinjaTrader_JoshP]

For your Market Analyzer to collect data you will need to add an Indicator column on it. Then it will collect it.

[adamus]

I tried to set up MarketAnalyzer to collect tick data from the InteractiveBrokers feed by putting a couple of indicator columns against my instruments.

However there's still no tick data in my HDM. It ran since forex session start last night, and right now, I have no tick data for any instrument.

For indicators, I am using the 1 min ATR(21) and the 60mins ATR(21). Does it matter what indicators I set up?

[adamus]

Just thought it might need to be a tick-based indicator which triggers the tick data collection, so I added another indicator to the MarketAnalyzer - the 21 bar ATR based on 150 tick bars.

So far, no tick data in the HDM is showing up.

[NinjaTrader_ChristopherJ]

Please go to Tools-->Options-->Data tab-->Enable "Save chart data as historical"-->Click Ok.

Keep your indicator set to 150 Tick within the MA window. If the issue persists please open a new chart and let me know if data is then available.