Basic NT questions from a prospective user

[bluelou]

Ray,
If I use IQFeed where is the historical tick data stored and how is the continuous contract adjustment performed? IQ Feed doesn't offer continuous contracts, just the feed. So, NT must be creating the continuous contract with the IQ Feed data.

Are you saying that the IQFeed data would be stored on my PC and that NT is somehow using the data stored on my PC to create a continuous contract with several weeks history? Is this correct?

But, then if this is correct I don't understand how I was able to get 5 months of tick data with NT + TTFIX since I didn't have any historical data on my PC at the time.

Regarding memory, I have 3 GB RAM available for NT. I run 3 separate computers. I won't be doing any charting or strategy development on the auto-strategy/execution PC.

[NinjaTrader_Ray]

Correct, NT stores data from IQFeed and you can merge historical data from the expiring month into the front month as per the link I provided several posts ago. Thus, you chart or use the front month but it includes merged back adjusted data from the expiring month.

There is some windows limitation where 2 GB or 3GB willl not make a difference. The historical data you saw comes with part of the TT service as a bonus. Its good to have but its not cleaned for bad ticks or gaps etc...

[bluelou]

Ray, that's exactly what I was hoping to hear. It sounds like I'll probably have 2 datafeeds, IQFeed + TTFIX (as a backup). I still need to check out the NT basic features such as optimization routines available. I can do this on my own. But you've answered all of my critical questions and I'll likely make the switch to NT.

As for the tick data limitations in Windows, there's not much I can do. That's the way the algorithms work. Someday, I'll have to figure out how to put the strategy on a much more powerful server - but I'm not there yet

Thank you,
Lou

[mason]

bluelou,

Will you tell us the difference between OQ and NT that make your probable decision to switch ?

I know both of them can connect to multiple datafeed and brokers, one is light, the other is heavy, one is into automated solely, the other is embracing all sides, one is lack of charting features, the other is supporting discretionary.

[bluelou]

mason,
I had used TradeStation for a few years and have been working on recoding my strategies for OpenQuant for about 2 months. Actually, paying 2 coders to do most of the work.

Here's what my experience has been w/OQ so far:
1) Very weak customer support. Sometimes they answer your question in a few days and sometimes not at all. Also, since they are in Russia they answer questions during their business hours, not US market hours.

2) The DataManager - used to collect real-time data - can cause the platform to crash when used.

3) The platform does not have continuous futures contract features that are available in NT and other platforms. ESignal is the only OQ data provider offering continuous futures contracts (I've called them all). ESignal data costs an extra $100 w/OQ (for a total of $250/mo b/f exchange fees) b/c you must use the ESignal API.

4) OQ is an IDE only, they don't collect the data of other providers as NT does. For example, you can call up several months of data in NT depending on your data provider. This doesn't exist in OQ. With OQ you either buy the data elsewhere, your data provider offers it, or you collect it using the DataManager (which can cause OQ to crash).

5) The Broker Info tab appears to be there just for show. In other words, unless you code this yourself the platform doesn't retrieve any account info from your broker.

6) There is no real-time charting. I think they claim to have this feature but it doesn't work for several users on their forum.

You can verify my claims by checking their user forum. If you read a post and there's no resolution of the problem at the end of the post that means the user just gave up. It happens all the time.

Having written all of this, users swear by OQ's strategy automation. If your data management needs are low and you have little use for technical support it's probably a good choice.

If I could resolve issues 1)-6) I'd stay with OQ but I'll probably move my C# code over to NT. I like what I've seen of RightEdge so far but only NT manages tick data in a way that I'm comfortable with and at a retail price point.

P.S. 7) Not much documentation. There isn't much, if any, description or examples for many of the methods and properties in OpenQuant. I started using the NinjaTrader documentation as a reference which is how I ended up switching to Ninja.

[mason]

bluelou,

Thanks for the information, it's great to have that from someone who has already done their due diligence.

I knew them from elitetrader forum, most of the posts seems positive, but I am not ready for automation yet.

[bluelou]

Tick data and RAM limits - Q for Ray

Ray,
Following up from your comments on loading 8 wks of tick data and the effect on RAM...

1) I didn't realize that NT doesn't have a Bar Factory like OpenQuant. In OQ, Bar Factory allows you to take tick data or other data and compress it into any tick/time interval you want and it drops the extraneous data.

I'll probably end up exporting NT tick data, compressing it in OQ, and then re-importing it into NT for backtesting. Then, the RAM problems you mentioned should be solved as far as backtesting is concerned. [Anyone can do this since OQ has a 30-day trial.] Maybe NT should consider adding this feature?

2) However, what about the impact of loading 8 weeks of tick data in a live strategy in OQ. Are you saying that this will be problematic in NinjaTrader? What problems, if any, should I anticipate w/NT regarding this?

I'd like to run this strategy on 20-30 markets. The most I've attempted was 6 markets (i.e., 6 live automated strategies each running on 6-8 weeks of tick data) and this wasn't a problem in TradeStation.

-BlueLou

[NinjaTrader_Ray]

1) NT does this internally. As you request 100 tick bar for example, we load the base unit for this bar which is "tick" data, compress it into 100 tick bars, hold on to the bars object and release the majority of the "tick" data required to build the compressed series.

2) I can't answer this. What I have stated in the past is that loading 8 weeks of tick data is memory intensive just on a single market. We wil be improving the memory footprint with NT7 due out in 09, but for now, you will have to see where you hit the wall. I can tell you that if you try to load 8 weeks of tick data for 30 markets you surely will hit the wall on memory. If this is your real requirement, I suggest now that NinjaTrader will not be sufficient.

[bluelou]

Fair enough. I just figured out a way to get the data requirements down to about 15 days of tick data. Hopefully that will help. If I can get 10 instruments loaded for each NT license I'll be happy with NT.

[mikeh]

Possible workaround

Here's an possible idea for working with your large tick dataset. The technical issue is the 32 bit Windows operating systems only allocate 2GB memory to a particular application. So at some point you will run out if you try to keep a large amount in memory.

Do you really need to have all the tick data accessable, or do you just need a calculation from it periodically? If you hired coders, you could have them collect the tick data from some source and write it directly to a database along with your calculated value. Then have the ninjascript indicator or strategy query that database for the calculated value.

[bluelou]

mikeh,
Yes, I really use the tick data in real-time on every bar. I need 1000 tick bars and the size of the tick bar varies with the insturment. I think I can get by with about 15 days of tick data.

I'll look into your idea about having the tick data go directly to a database. It sounds like it could be a long-term solution. I need to get my strategy moved over to NT and see how many instances of it that I can run b/f venturing into the database idea.

Also, I'd probably want the best data feed I could find b/f tying up my algs w/a single database. The TT and IQFeed data feeds are what I'm starting with and are dependable enough that I can switch b/t the two if there's a problem. But, if I'm responsible for the database I'm going to need something like the higher-end CQG/Reuters/Bloomberg, etc. feeds.

Thank you for your input.

-Lou