NT Data Handling - Huge Disappointment

[scrilla_gorilla]

NT seems to have a fatal flaw for backtesting strategies, and that is how it handles historical data.

In short, it doesn't seem to handle historical data. I am running a relatively simple strategy using 3 instruments with 5 min bars. (Really, 1-day bars would be sufficient, except there's no way to use 1-day bars and actually process an order at the close of that day, so 5 min bars it is.)

I have found it absolutely impossible to run a backtest before the date of 1/1/2009 (8 months). As soon as I do this, NT attempts to re-download ALL historical data for all three instruments, even though I've already downloaded this same data dozens of times.

To make matters worse, I'm using IB, which simply won't let me download all that data all at once. But that's besides the point, because it's ridiculous that NT is trying to re-download that data (for the 100th time) in the first place.

I understand NT maintains a "cache" but if NT is to be a legit strategy development system then it doesn't need a "cache", it needs a database that stores downloaded historical data indefinitely.

Once a set of historical data has been downloaded once, there is no reason why it should EVER be necessary to download it again.

If there is any way to work around this flaw, please please please let me know. NT is much easier to work with than OpenQuant, and is just better software and more robust than TradeStation, but this data issue is making it impossible to backtest my strategy.

Thanks.

[scrilla_gorilla]

Even using 30 minute data, and even when setting the end date to a week prior, (8/24/09), and even setting the start date to the beginning of the year (1/1/09), and even with charts open containing data for all 3 instruments being used...

About 50% of the time NinjaTrader insists on re-downloading all the data for the 200th time, which of course is not possible.

Are there plans to fix this problem and make NinjaTrader a legitimate development platform? Or will NinjaTrader continue to use a "cache" rather than a proper database to store historical data?

[Baruch]

Hi,
I see that you are new to NT. I don't think that you browsed the forum. In any way NT saves the data in database and you can run back testing disconnected from a data provider.
NT has many flaws, but please be certain that you know more about the platform before you place your "findings and conclusions".

Baruch

[scrilla_gorilla]

Actually I've spent hours reading the NT forum, as I initially assumed that I was doing something wrong.

To download new historical data, you obviously have to be connected. But when you're connected NT forces you to download all the previous data. So for instance, if I have data from 12/01/08 to current, but I want to get data for 10/1/08 to 12/01/08, even if you set the strategy time from 10/1/08 to 12/01/08, NT starts the download from the current and works backward. Why it does this, I don't know.

So again, you are severely limited in the amount of historical data you can access, because in order to download new data you must be connected, in which case NT attempts to redownload all the data you've already downloaded.

[Baruch]

After you got your historical data you can disconnect and run the back tests. NT doesn't reload all historical data again and again, but if you are connected it checks if any new data have arrived.

Baruch

p.s.
The best way is to run tests disconnected!!!

[piersh]

NT seems to have a fatal flaw for backtesting strategies, and that is how it handles historical data.

yes, NT's support for historical data is quite pathetic. hopefully they will address this in NT7. don't hold your breath...

it's a shame, like you say, because NT could be quite a powerful tool if it wasn't such a pain in the ass to use.

the disconnected 'solution' is really insuficcient - this should have been fixed ages ago.

[scrilla_gorilla]

yes, NT's support for historical data is quite pathetic. hopefully they will address this in NT7. don't hold your breath...

it's a shame, like you say, because NT could be quite a powerful tool if it wasn't such a pain in the ass to use.

the disconnected 'solution' is really insuficcient - this should have been fixed ages ago.

The backtesting is better when disconnected, but that, of course, means you can't actually use NT and develop at the same time. If you're running Strategy A, you can't work on Strategy B, because you can only do the first connected and the second disconnected.

It would also be extremely helpful to be able to see what data you had for each underlying, and to be able to download the data "directly" rather than having to open a series of charts. Like I said, a proper database.

It is really a shame because, the data notwithstanding, strategy development is SO much easier with NT than with more robust solutions like OpenQuant, while nearly as powerful. But while managing OpenQuant data takes a little more work, you can actually manage it, which you cannot do at all in NT.

[Baruch]

You can run live and develop in the same time. Only on diferent computers. NT has its disadvanteges, but it costs ZERO if you are not trading live.
You can have 1 chart open to gather all the data you need, just change the instrument after you loaded the data, or you can have a workspace with all your charts and load it only when you want to collect all the data.

Baruch

[NinjaTrader_Dierk]

Guys,
1) On NT6.5 you could work around that issue if you ensured that at the date the backtest started there would be data in the database. NT just would keep trying to pull data from your provider if for the starting date (+- a few days tolerance) there is no data in the DB. This implies that NT would keep trying to pull data from your provider if around the starting date of your backtest your provider is just unable to provide any data. NT does not have a logic like "I tried it once, got nothing and won't try again".
2) On NT7, there will be a feature where you could disabling pulling data from your provider. Thus, you could set NT to state where it would not try to pull any data.

Hope this helps

[scrilla_gorilla]

So if you are using 300 different instruments then you need to open 300 different charts? Or go through all 300 instruments on one chart window every few days? That does not sound practical.

There needs to be a data management dialog where you have all the different instruments in your database and the granularity of data you're collecting.

[Baruch]

You are trading only 300 instruments? So few? I thought at least 10K.

Baruch

[NinjaTrader_Dierk]

So if you are using 300 different instruments then you need to open 300 different charts? Or go through all 300 instruments on one chart window every few days? That does not sound practical.

There needs to be a data management dialog where you have all the different instruments in your database and the granularity of data you're collecting.

Thanks for your suggestion. We'll add it to the list of future considerations.

[piersh]

Guys,
2) On NT7, there will be a feature where you could disabling pulling data from your provider. Thus, you could set NT to state where it would not try to pull any data.

how is that different from disconnecting from the provider?

it sounds to me like NT7 isn't going to be any better than NT6.5 in this regard.

given that pulling trade data from providers is the CORE of your application, and you still haven't gotten it right after SEVEN versions, I'm not holding much hope for the future.

[NinjaTrader_Dierk]

>> how is that different from disconnecting from the provider?
You still would get realtime data.

[piersh]

>> how is that different from disconnecting from the provider?
You still would get realtime data.

why can't it just remember that a particular date range has no data.

for example, why does it miss the cache when i accidentally set the start or end date to lie on a weekend?

it should be doing everything possible to avoid going to the provider. and when it does have to go to the provider, it should only be fetching those ranges that it doesn't already have. and when any part of any of those ranges fail, it should remember that fact and never go to the provider for those parts of those ranges. and there should be some UI for this so the user can for retries for given ranges.

and don't "future consideration" me on this. this stuff needs to be fixed yesterday.