Problem importing tick data into historical data for NT8

[billythekid72]

Hello,

I am seeing the following problem in the log: ES 12-04.Last.txt'. Instrument is not supported by repository.

My data format conforms to the import format (I have imported all years prior to 2004), example as follow:

20040908 153000;01119.00;5
20040908 153308;01119.00;2
20040908 153325;01118.75;35
20040908 153410;01119.00;10
20040908 153548;01118.75;1
20040908 153611;01119.00;1
20040908 153839;01119.00;1

[NinjaTrader_PatrickG]

The example you've given appears to follow the 'Tick Format (second Granularity)' example provided here:

• https://ninjatrader.com/support/help...AndDataFormats

What version of NinjaTrader 8 are you using? Please provide the entire version number. This can be found under Help -> About (Example: 8.0.?.?)

Please attach the data file you're using so I may test on my end. If the file is very large (>10mb), please create a zip file first then attach that zipped file.

[billythekid72]

Hello,

It is worth noting that I was able to import all the data files from year 2005 and later, only having problem starting 2004. Here are the version and data file.

Version: 8.0.22.1 64-bit

The forum does not allowed upload of file larger than 1MB, I am provide a link to Google drive here:

https://drive.google.com/file/d/1Q6XrgS5lLFXa7L5-leH69Pr-E-rKztRC/view?usp=sharing

[NinjaTrader_PatrickG]

It looks like, by default, NinjaTrader expiries for ES only go back to 03-05. This is the root issue - technically, ES 12-04 doesn't exist to NinjaTrader so it can't import data for that expiry. To work around this, you can manually add the 12-04 expiry by following these steps:

• Control Center > Tools > Instruments
• Find and select the ES futures instrument then choose 'edit' in the bottom right
• Scroll to the bottom of the Instrument window where you'll see 'Contract months'. Hover over 'xx rollovers' and click on 'Edit rollovers...'
• In the bottom right, choose 'add' > On the right, change the contract month to 12-04 and also change the 'Rollover date' to a Thursday in September (I chose the 9th) > OK > OK

Once NinjaTrader has the 12-04 expiry for ES, you should now be able to import the data you've linked to.

I have submitted a feature request to the Development Team to include more historical contract months. The internal tracking number for your feature request is SFT-4976. Please reference this internal tracking number when contacting Platform Support if you ever have questions regarding this feature request.

When a feature request is implemented, you'll find a description of the new feature in the release notes:

• http://ninjatrader.com/support/helpG...ease_notes.htm

[billythekid72]

Thank you, yes it is importing like you suggested after I made the change. Incidentally ES started in 1997, so your user should be able to import all the back to 97 historical data.

[billythekid72]

Hello,

After I uploaded the historical data for 2004, when I try to run the Strategy Analyzer against 2004, I got the "Error on requesting bars series: Unknown instrument ES 03-04" error.
I did notice that for ES 12-04, ES 09-04, and ES 06-04, the offset is N/A, but ES 03-04, the offset is 0 and I can't change it. I wonder if that has anything to do with the problem.

[NinjaTrader_PatrickG]

Creating additional expiries does not 'transfer' the data you've just imported to those expiries. The data you've just imported exists completely within ES 12-04. By default, NinjaTrader has a merge policy of 'Merge back adjusted' which means it uses the expiry roll dates to determine which contact to pull historical data from. Since you have no data in any expiry but ES 12-04, the test is failing (as expected). A likely solution would be to change your 'global merge policy' so that the Strategy Analyzer does not try to look for data for other contracts and only looks for data in the literal contract you have selected (in this case, ES 12-04). Follow these steps:

• Control Center > Tools > Options > Market Data
• Change 'Global merge policy' to 'Do not merge' > OK
• In the Strategy Analyzer, conduct your backtest

[billythekid72]

This does not seems to fix the problem. Now that when I run back test from 2003 to 2020, I only get 2020 test data. Whereas before, I was able to back test from 2005 to 2020 (before I added 2003 and 2004, I had historical data from 2005 to the present day) Please see attached.

[NinjaTrader_PatrickG]

That is actually expected due to the change in global merge policy. I incorrectly assumed you were running a backtest with ES 12-04 selected.

Please change the global merge policy back to 'Merge back adjusted' then delete the other expiries you created with have no data (anything prior to 12-04).

[billythekid72]

Hello,

I think I had done a terrible job in describing my problem, let me start over.

I was able to import historical and run back test for period 2005 until 2020 the present date, so far so good. I encountered problem importing historical data for 2004, which you shown me the way by adding new expiry dates to the instrument. I did just that, and was able to subsequently import historical data for 2004 and 2003. After which, when I try to run back test for period 2003 through 2020, I encountered the"Error on requesting bars series: Unknown instrument ES 03-04" error. This is where I stand. Looking forward to your advice. Thanks.

[NinjaTrader_PatrickG]

What is the start date for your backtest? If you incrementally push this date forward (lets say 30 days at a time), when does it stop occurring?

What rollover dates did you choose for the 2003 and 2004 expiries?

[billythekid72]

For the back test I choose start date on or after the rolling dates. For the rolling dates, I chose second Thursday of 3 months prior to expiry, e.g. ES 03-04, rolling date will be 11th Dec 2003.

[NinjaTrader_PatrickG]

If you incrementally push the start date of the backtest forward (lets say 30 days at a time), when does the error message stop occurring?

Please send me your database, log, and trace files so that I may look into what occurred.

• You can do this by going to the Control Center-> Help-> Email Support
• Expand 'Other files' and enable 'Database file'.
• Please reference the following ticket number in the body of the email: 2643468 ATTN PatrickG

[billythekid72]

Hello,

Looks like the problem went away. In the process of trying to find out where the error occurs, I deleted the expiry dates and added them back in one at a time and then re-import the quarter of historical data before running the back test again. That way, I was able to add historical data all the way to ES 03-03 and run back test without any problem.

[iplaymuzik]

billythekid72 where did you get your tick data?