Importing tick data

[sgordet]

I'm testing out a sample of historical tick data for the ES from a data provider that provides all of the historical futures contracts.
I'm running NT 7.
I seem to have gotten everything set up properly. I checked the boxes for building minute and day bars and they came up instantly in the charts; however, I couldn't see tick or range data on the charts.
Looking at the historical data manager I could see that days and minutes were there for this time period but not the ticks.
I tried running the import again without checking the minute and day boxes, thinking that maybe the tick data had to be imported separately to show up as ticks, but that didn't work either.
Can you give me any help on how to proceed?

[NinjaTrader_Chip]

Hello sgordet

Thank you for your post.

We will need to see a sample of the data to understand how it is formatted. If you can copy/paste a few lines this should be sufficient to determine the interval.

Additionally, as a reference, the following resource details the criteria for historical data import in NinjaTrader “ Understanding import file and data formats”:

http://www.ninjatrader.com/support/helpGuides/nt7/index.html?importing.htm

[sgordet]

20081026 180000;864.25;0
20081026 180000;864.25;0
20081026 180000;864.25;0
20081026 180000;864.25;0
20081026 180000;864.25;0
20081026 180000;864.25;0
20081026 180000;864.25;0
20081026 180000;864.25;0
20081026 180000;864.25;0
20081026 180000;864.25;0
20081026 180000;864.25;0
20081026 180000;864.25;0
20081026 180000;864.25;0
20081026 180000;864.25;0
20081026 180000;864.25;0
20081026 180000;864.25;0
20081026 180000;864.25;0
20081026 180000;864.25;0
20081026 180000;864.25;0
20081026 180000;864.25;0
20081026 180000;864.25;0
20081026 180000;864.25;0
20081026 180000;864.25;0
20081026 180000;864.25;0
20081026 180000;864.25;0
20081026 180000;864.25;0
20081026 180000;864.25;0
20081026 180000;864.25;0
20081026 180000;864.25;0
20081026 180000;864.25;0
20081026 180000;864.25;0

[NinjaTrader_Brett]

Hello,

Tick Format
Each tick must be on its own line and fields must be separated by semicolon (.

The format is:
yyyyMMdd HHmmss;price;volume

Sample data:
20061107 000431;1383.00;1
20061107 000456;1383.25;25
20061107 000456;1383.25;36
20061107 000537;1383.25;14


In your sample data, you have volume as 0 for each tick. This would cause issues with many tick based charts.

[sgordet]

I noticed that some of the data that I didn't send you did have the volume data as you showed it, rather than zeros. I wonder why it it's so inconistent.
In any case, I don't need volume data but I do need the tick data. If I replaced the zeros with 1's, would that work? If it doesn't, is there something else that would?
Thanks,
Stephen

[NinjaTrader_Brett]

Hello,

Yes this would be the first step since NinjaTrader does not support volume == 0.

Please attach the file as an attachment if still not working and I will look into it further.

I look forward to assisting you further.

[sgordet]

I did a search and replace for ;0 and changed it to ;1
The Historical manager still just shows day and minute bars. Tick and range bars still won't load on the charts. For the charts, I'm using ES ##-##. The actual contract for this period shows up in the historical data manager with day and minute period dates, but no tick data is reported until I get to the current contracts.
I'm sending you some of the new file. It zipped to 3 mb so I couldn't send it all.
Please let me know if there's anything else I can provide.

Stephen

[NinjaTrader_Brett]

Hello,

Ok thanks for posting.

Your issue is as follows. This file name is for the ES 06-11. And is also for back in 2008.

By default auto merge is enabled in NinjaTrader so it will pull in data for the previous contact after the merge point.

For example requesting data for the first of the year would roll back on the roll over date to the ES 03-11.

You would need to either turn off the auto merge function or rename the file to the ES ##-##.txt and import it again and instead use the ES ##-## contract instead of ES 06-11.

If you need to use ES 06-11, then turn off auto merge. Control Center->Tools->Options->Data Tab->Merge Policy->Do Not merge.

Let me know if I can be of further assistance.

[sgordet]

Just wanted to thank you.
The issue was the file name. The ticks imported fine once I changed it to ES ##-##.txt. And the chart rendered all right even with some of the vol data missing.

[NinjaTrader_Brett]

Excellent, glad you got it imported.

Let me know if I can be of further assistance and have a great rest of the day.

[sgordet]

I went ahead and purchased the tick data for the entire ES history . I imported it using your suggestion of using the ES##-##.txt symbol and the entire history imported, I think, perfectly, from what I checked. I now have several questions.
1. The data is provided as one continuous contract with no adjustments for contract changes; however, I do have the day of month when the contracts are rolled over. Is there any way that I can use the adjustment data that you provide to properly adjust this data.
2. I haven't tried it but from your remarks it sounds like I can import tick data for the entire history entering it as ES06-11 if I turn off the auto merge function before importing. If I do that, what do I wind up with? Is it just several years of data all under 06-11? and then what happens when I roll to 09-11?

[NinjaTrader_Brett]

Hello,

Glad to hear you got it imported.

1) On your first question, unfortunately this data would need to be changed in the file itself and then re-imported.

2) You would need to import in each contract individually for the auto merge function to work correctly.

For example, import in the ES 06-11, ES 03-11. So that the correct data is in the correct expiry, this way when auto merge is enabled it will get the data correctly.

Otherwise, what you could do is disable the auto merge function and then import this data into the current contract and you would simply have the current contract with a ton of historical data with no merging going on.

Let me know if I can be of further assistance.

[sgordet]

Thanks. I understand.
Now, the question is where do I go from here. I noticed that even though the data I imported in ES##.##.txt is now several weeks old it's being automerged with the data that's coming in under that symbol from IQFeed. IQFeed provides about 4 months of tick data. As long as I connect at least once every four months and have the data series input set to at least 4 months back will the NT7 database record all of the new tick data from IQFeed and keep it permanently in the database or is that new data transient?

[NinjaTrader_Brett]

Hello,

NinjaTrader will simply overnight the previous data in the database for what it could download from the provider. So yes you would want to pull up the instrument once every 4 months at a minimum to keep the data in the local database. You will also want to goto Control Center->Tools->Options->Data Tab and make sure the Save Chart Data as Historical is checked so that this works.

Let me know if I can be of further assistance.

[sgordet]

I just wanted to thank you for your attention to detail in these posts. If you hadn't mentioned the need to check "Save Chart Data to Historical," I wouldn't have looked there and found out that it wasn't checked and then wondered why I was missing data.
Much appreciated,
Stephen