Importing Data from TradeStation problem

[tooearly]

When importing data from TS I get an error message in the log that reads:
"Default ... : Time stamp line 102 is smaller/equal than timestamp of previous line."

At other times it will read: 7/22/2009 4:27:57 PM Default MSFT: Time stamp line 65 is smaller/equal than timestamp of previous line."

I have tried this on three instruments, all of which are in the Instrument Manager and for which I have requested several months of back data, with the same result.

The only file that opens properly is that from which only two days of back data have been requested.

Open in C# editor the .txt data files all have data complete back to the date requested.

The computer clock is set to market hours EST, but strangely all the files begin with a line such as

20081107 101100

which I read as 10:11 am Nov 7th, 2008 (please see screenshot attached below) but on the following day, and all days thereafter, the first time indicated is

20081108 093100.

Why would the data not begin at 9:30 am on the opening day?

Thanks in advance.

Tooearly.

[NinjaTrader_Bertrand]

tooearly, which timeframe do you use in TS to export this? Please ensure this is a 1 min frame for intraday data as outlined here - http://www.ninjatrader-support.com/H...ownloader.html

[tooearly]

Thanks Bertrand,

Yes, the time frame set is 1 min. I have followed the instructions in the video.

Here are two screenshots: 1) shows the settings for importing the data.

2) shows the error messages in the Log file after the data import fails, which may hold some clues

I followed the same procedures for QID going back to Nov 3, 2008 and that file was imported successfully.

Seems odd.

[NinjaTrader_Bertrand]

Thanks, please try exporting the data in TradeStation again and take care to set

a) Last data to today's date (22nd)
b) using local timezone (not exchange)

[tooearly]

I changed the settings as you advised (taking care to set
a) Last data to today's date (22nd)
b) using local timezone (not exchange)

and was able to import 1 year of MSFT and an equal amount of data from one of the EFTs.

The with CSCO and several other EFTs I continue to have the same error message.

Default    CSCO: Time stamp line 102 is smaller/equal than timestamp of previous line

It is identical in all cases.

"Local time" on the computer is set to same time as NYSE.

Puzzling...

[NinjaTrader_Bertrand]

Do you notice gaps in your TradeStation data? Have you tried reloading the data in TradeStation and then exporting again?

[tooearly]

Yes, I reloaded all the data in TradeStation before sending it on again to NT.

What I noticed is some files were double the size of others even though the downloads of data for both covered the same period. That suggests gaps.

But TS schedules data downloads and the Scheduler notes whether the gaps have been filled when the d/l is "successful".

In what file would I find the data that has been successfully imported into NT?

[NinjaTrader_Bertrand]

It would be stored in the local database file, you can work with Tools > Historical Data > Export to check for consistency.

For NinjaTrader 7 this will become easier, as we'll include a Historical Data Manager then - http://www.ninjatrader.com/webnew/NT7/NinjaTrader7.html

[tooearly]

Tools/Historical Data/Export gives me a drop down box, with one month of data selected by default for all instruments.

If I go to C:\Documents and Settings\admin\My Documents\NinjaTrader 6.5\db\export I find the folder is empty.

C:\Documents and Settings\admin\My Documents\NinjaTrader 6.5\db\data\20090722 however has two NTM files of 1 kb each for FAS.ntm and FAZ.ntm respectively.

There doesn't seem to be any consistency with the reports of successful data import of MFST and one or two other EFT -- neither FAZ nor FAS -- which I can not remember.

[NinjaTrader_Bertrand]

tooearly, let's rewind a bit here, the data you reference in the different folder is not comparable. If you import TradeStation historical data from the txt data files, you should export throught the Tools > Historical Data > Export command to check. Of course you would need to select 'Stocks' as Type to work with MSFT and ETF data.

[tooearly]

If I go to Tools > Historical Data > Export and select MSFT under Type Default and chose the dates Oct 22, 2008 to July 22, 2009 and then select Bar one minute, then I can export the data file which is in the NinjaTrader 6.5\db\Import folder to any other folder.

I can run through the list which I tried to import from TS with the same result.

Does this mean that the data has been imported successfully?

Does it mean that the .txt files in the DB/Import folder have data going back to Oct 22, 2008? (I think some may and some may not, since FAS & FAZ for instance only started trading in Nov 2008 as far as I know.)

If this resolves the problem then why is there an error message in the Log?

How do I call on this data for backtesing purposes?

[NinjaTrader_Bertrand]

The TradeStation historical data downloader strategy we supply stores the data you manually have to import in the NinjaTrader 6.5\db\Import folder. The data being placed there does not mean it's imported successfully to the NinjaTrader database. This is what you do by going to Tools > Historical Data > Import and then pointing to the file.

Once it's imported you can chart it and backtest on it.

[tooearly]

That's what I thought to be the case.

Again, I tried to import MSFT from NinjaTrader 6.5\db\Import and got a message saying "Historical Data Imported Successfully."

FAZ & QID also report "Historical Data Imported Successfully."

But FAS, CSCO, UYG, URE, SRS, SSO, SDS, SKF and DTO all report the same error message, Default Time stamp line 102 is smaller/equal than timestamp of previous line, identical in all cases.

Does this error message have any significance?

I have used the same selections for importing all the data but only 3 of 12 succeed.

Guidance much appreciated.

[NinjaTrader_Bertrand]

tooearly, I think the error you get is related to gaps in your TradeStation data - can you please check the TS downloader and see if all download requests have been fullfilled for the history and then run the data downloader strategy again on the full TS data set, then reimport into NinjaTrader.

[tooearly]

Sounds like a plan....

I'll give you a rest on this one and go about redoing the downloads during off-peak hours, which might give a better chance of filling gaps, and then report back tomorrow.

Thanks for persevering at least up to this point!

Cheers,