Importing tick data

[NinjaTrader_Brett]

my pleasure,

Have a great rest of the trading week!

Let me know if I can be of further assistance.

[sgordet]

I've started importing the historical futures tick data from a txt file supplied by the vendor. While importing the 6E data (6E ##-##.txt), I received the following two error messages about five minutes after the import started. Is there any way that I can fix these errors:

6E ##-##: Illegal volume in line 1904806. Volume needs to be greater than 0.20031107 102000;1.1452;-2

6E ##-##: Illegal volume in line 1892184. Volume needs to be greater than 0.20031103 102000;1.1498;-2

Thanks,
Stephen

[Crassius]

Sometimes there are errors in the data. You'll need to open the tick data file in a fast editor like UltraEdit -- avail for free trial.... you can try notepad, but if it is a big file it can get bogged down. Also Ultra Edit has line numbers and search features that will help you find the offending lines.

Then you'll need to manually edit the volume field to fit NinjaTraders format. It looks like what ever is in the volume field is not compliant with the format at the line numbers the error message is referring too.

Since it is only one tick you might average the volume of the one before and the one after...

[sgordet]

I've now imported tick data going back several years for quite a few continuous futures contracts which I will use for backtesting. I need to create a strategy for keeping all of these contracts up to date. My thoughts on how to accomplish this are in the next paragraph. I would be appreciative if you would let me know if you have a better idea and/or if you think that there is a problem with the way that I'm planning to do it.
My data vendor, IQFeed, provides several months of current tick data per contract. I was thinking of creating a separate workspace and placing, for each of the continuous contracts, 800 tick charts set to load 60 days of data from current date. I would then load the workspace once a month to update the historical data manager. If there were a way that I could just create an instrument list of all the continuous contracts and then load them into a market analyzer window to do the monthly update, rather than having all of the charts, that would certainly be better. I would think that I just need to know which indicator will pull 60 days ot tick data- into the Analyzer--if there is one.
Assuming there's a method that works, a corollary question: when I imported the initial tick data, the historical data manager gave me the option of automatically generating minute and day bars, an option which I selected. How do I generate minute and day bars from the new incoming data or is that automatically done?

Thanks,
Stephen

[Crassius]

My thoughts in broad brush strokes

Sgordet,

First... you're welcome.

I think of my current setup as using NT in two ways, One, as an historical data collector and Database to add and store data to the prurchased historical data I have loaded to keep it up to date. I also have a subscription from that vendor, who sends quarterly updates. in case all else fails.

At any rate, I want to collect my own data from my broker, and build up a local database over time with that data for backtesting and system development

At the same time, I am trading everyday and using live data as an active trader would.

These are really two different functions, and I have found it best to seperate my historical DB for testing from my live data needs. To that end, I created a false symbol in Instrument Manager... something like EUHist.... under that symbol I store all the historical data for testing, and add to it every month.

I only trade one symbol at this time so its easy to collect new tick data with a chart open during market hours, set to tick data and left alone just for that purpose. From the tick data I can build whatever I want for testing.

Daily, at 4:58PM EST I disconnect and backup the collected data by doing an export from the 1st day of the month to the current day. On the last day of the month my export will then be one month's worth of ticks. That gets zipped and stored off site on a cloud drive.

Quarterly or sooner if I want, I add my collected ticks to the EUHist symbol.

I have found it best to seperate the historical database from the data I am actively using for trading because I need to protect it from getting erased during day to day trading activities.

One of the irritating things about NinjaTrader is if you have data for a particular time period already stored in your local database, then make a request... by opening a new chart for example... for a date or range of dates that exist in the local database, but that are not on the data vendor's or broker's server, NinjaTrader will erase the locally stored data, and you lose it.

The real world being what it is, it is surprisingly easy to get data erased this way.... from time to time the data vendor or broker will have technical issues on their end, that you don't know about until it is too late, and you make a request for data from their server that will end up erasing your locally stored data. Or you are deep in thought about some new trading idea, and you open a chart requesting more data than exists on the data vendor's server... your range was bigger than they have.... and again... you lose data in the local DB. Or you make a typo opening a chart.... and BAM!! you just blew a hole in your local DB.

Long story short, I have found it best to seperate my historical DB from the data I am using for live trading to prevent these accidental erasures by seperating it into a fake symbol. That way I at least have the data stored under a different fake symbol.... my EUHist.... so I can restore my live data DB from that.

[sgordet]

Brett, I'm hoping you can take a a crack at answering my new question.
As previously, Crassisus's comments were most helpful. He's taught me quite a bit about the Historical Data Manager; however, I don't really want to put as much effort into this as he does. Can you tell me if my "once a month" method will keep my separate continuous contract file up to date or if I need to do something more involved.

[NinjaTrader_Brett]

Hello,

Yes this should work just fine to keep your database up to date and is what I would do.

Let me know if I can be of further assistance.

[sgordet]

Brrett, is there an indicator that I can place in Market Analyzer that I can configure to load 60 days of tick data so that I can place all of the instruments in it and not have to load a ton of charts when I do the database update once a month?

[NinjaTrader_Kyle]

Hi sgordet,

That should work for you with any indicator so long as it is configured to load the entire 60 days.

[sgordet]

Is it the "# of bars to look back" that I need to set to 60?
Also, when I originally imported all of this tick data, the Historical Manager asked me if I wanted to also build minute and day bars, to which I responded in the affirmative. Do I need to separately build minute and day bars for this additional data and, if I do, do I do it by chaning the indicator to minute and then day after loading all of the tick data?

One last question:
I don't know if I need to post this separately in the Strategy Analyzer forum, but as long as I'm troubling you:
I imported all of the historical futures tick data as "Last." Was that wrong? If it was correct, my problem is that some of my strategies buy on the ask and sell at the bid. What happens in the Strategy Analyzer when the data is all "Last."

As always, thanks for your patience.

Stephen

[NinjaTrader_Brett]

Hello,

To keep the database running on all 3 data types this would be needed yes.

Also, when running on backtest you can only run on last/bid/ask data. If you need to run on all 3 such as your strategy this would require you using multiple data series and adding it in bid and ask data series so you have 3 data series in total on the backtested strategy.

[sgordet]

1. Just to make certain that I understand: I should use three identical indicators each "looking back 60 days," one based on tick, one on minute, and one on day. I was wondering, if I just collected tick data, would the minute and day data be built on the fly as necessary or would I not be able to use those time periods?
2. Since I don't have bid/ask data, I can't use multiple data streams. What happens when I try to back test strategies that enter based on bid/ask points?

[NinjaTrader_Brett]

Hello,

1) No the data would not be populated on the fly if tick data is only used.

2) It will run of last price data purely and not bid/ask data.

Let me know if I can be of further assistance.

[sgordet]

In trying to set this up to capture the last 60 days of tick data through Market Analyzer, I picked an inidicator for a column and used Tick 800 data series. When I went to indicate the # of bars to look back, I realized that I would be looking back at 800 tick bars, and I want to capture the last 60 days of tick data. Can I get the indicator to look 60 days back or do I need to build a tick indicator with 60 day look back expressly for this purporse?

[NinjaTrader_Brett]

Hello,

You would need to set this to a high number here to make sure you get enough data loaded. 60 days should not be needed if you keep the market analyzer runs all the time. You would simply need enough to make sure you cover any gaps from when you would have shut down NinjaTrader. For example, 5 days back of data if you start NinjaTrader at least once every 5 days and it will keep a running total of data moving forward.

You will want to make sure that you are aware of these data loading rules however.

http://www.ninjatrader.com/support/helpGuides/nt7/historical_data.htm

As you will want to make sure the data is being loaded in. To verify by hand its being loaded you could press the f5 key when the market analyzer is selected to force a manual load of data.

Let me know if I can be of further assistance.