Importing Historical Data - Associating the Data with a Specific Contract

[beejinator]

Hello,

I have purchased about 20 years worth of historical tick data for NQ, and imported it into Ninjatrader. However, I see that the data is all lumped into its own "Instrument", but this new instrumnent is not available in the strategy backtester. How can I either have Ninjatrader associate the imported data with existing instruments, or be able to select the new custom instrument in the strategy analyzer.




Thanks!

[QuantKey_Bruce]

Well, you have two choices here - you can rename the file and import it into a "real" instrument that already exists or you can add the details for the NQ_TICK_NT instrument to the instrument list. I would recommend you do the latter, so you don't have a hodge-podge instrument in which some data is from CQG and some from the file or something like that - that has the potential to waste a lot of your time trying to figure out why it "breaks" on a certain date. What you probably should do is to add that NQ_TICK_NT to the instrument list if it isn't already and fill in all of the details, copying them from the NQ future definition such as the tick size, point value, etc. and then you can pick the instrument NQ_TICK_NT in strategy analyzer.

[beejinator]

Thanks Bruce!

It looks like it already existed in the instrument list, as Type = Stock. So I manually added a new one with the same name, with the same details as NQ. However, when I select the one I manually added from the instrumnet list in strategy analyzer, it gives me the message "Please select an instrument to run a backtest on"

Not sure if there's another step I'm missing. Do I need to delete the instrument that was auto generated as type = Stock?

[QuantKey_Bruce]

Delete the stock one. That is what happened when you ran the import without having it in the instrument list.

[beejinator]

Ok, deleted that one. Now I'm getting a new error:



So far I only added a single contract month to this instrument, the current contract. I get this error if I try to run a backtest or just open a chart for this instrument.

[QuantKey_Bruce]

The one month you added - is it 06-23 and you set a rollover date that is in the FUTURE?

[beejinator]

No, I used the same rollover date as the 6-23 NQ - 3/13/23

[NinjaTrader_BrandonH]

Hello beejinator,

Thanks for your notes.

It seems you have created the instrument as an equities instrument in the Instruments window and not a future.

You would instead need to create a futures instrument in the Instruments window to add your instrument. Also, the historical data text file that you are importing has to have the contract month in the file name.

And, you would need to add the contract months to the instrument once it has been made as a futures instrument.

Working with Instruments: https://ninjatrader.com/support/help...e_database.htm

Please let me know if I may assist further.

[beejinator]

I did create it as a futures instrument - you can see in my earlier screenshot (the instrument underneath the green circled one) it says Futures in the Type column. Apologies, I'm not quite following your instructions.

I think it would be easiest if we maybe start over, and you can create a step by step list of all things that need to be done on my end to get this imported correctly? Please create the list of steps with the assumption that my data is all in a single file (20 years / 800 million rows) and it's not realistic to manually work with this file as it's too large to open in programs like excel.

Thanks so much for all your help

[QuantKey_Bruce]

In situations like this, I create an artificial futures instrument, then create a single rollover set to happen in the future (no past months), and then import all of the data into that one month.

[beejinator]

Ok thanks! I created a new instrument called NQ_Historical, which will also be the name for the text file I will re-import. The instrument has been duplicated from the system NQ instrument, and I've added a single future rollover month. Is there anything else that needs to be done for this to import properly and be a usable instrument for backtests?

New Instrument:


Single Rollover Month Dated in the Future:

[NinjaTrader_BrandonH]

Hello beejinator,

Thanks for your notes.

If you are importing historical data for a futures or forex instrument, the instrument MUST exist in the database. If it does not, you must add it first via the Instruments window before importing the historical data. This could be seen in the Importing Tips help guide below.

Importing Tips: https://ninjatrader.com/support/help...calData

When you import historical data for a futures instrument, the text file needs to include the expiry month. Please see the link below under 'Understanding import file and data formats'.
https://ninjatrader.com/support/help.../importing.htm

For example:
MSFT.Last.txt for Microsoft stock last price data
ES 12-09.Bid.txt for the S&P E-mini December contract bid price data
EURUSD.Ask.txt for the Euro/U.S. dollar currency pair ask price data

Please let me know if I may assist further.

[beejinator]

Hello NinjaTrader_BrandonH and QuantKey_Bruce - I have followed both of your advice, but unfortunately still having issues. I took the following steps:

1. Created a new futures instrument, using the same settings as the existing NQ instrument, named "NQ_HISTORICAL"
2. Within this new futures instrumnet, I created one single future dated rollover for December 2050, "12-50"
3. Renamed my import file (~20years of tick data for the NQ) to "NQ_HISTORICAL 12-50.Last.Txt"

When I run the import again, instead of importing the data as it did last time, I receive this error message:

[NinjaTrader_BrandonH]

Hello beejinator,

Thanks for your note.

It is likely that the data you are trying to import is not in a correct format. You would need to make sure that the data you are importing in the file is in one of the formats listed in the help guide page below.

Understanding import file and data formats: https://ninjatrader.com/support/help...AndDataFormats
Formatting data and formatting the file name: https://ninjatrader.com/support/help...romTickDataLLC

Further, I see you noted that you created a single rollover date for December 2050 which will not work. You should just add the current year that some of the data exists in. For example, if the data goes up to this month add a contract for this month.

The future instrument expects to have multiple contracts of data and to use the merge policy with offsets. If your data is already merged going back many years and isn't something to be traded on/fake instrument, it may be easier to just use a stock type instrument instead of a futures instrument.

Let me know if I may assist further.

[beejinator]

I was able to get it to work by removing ".last.txt" from the file name. In this case, it seems that the instructions sent were misleading. Might want to update the documentation to include correct file format for tick data, as well as tick-millisecond data. In my case, using tick data, adding the ".Last.txt" was breaking the import.

Working as expected now, all data imported and usable for backtests.

Thanks for the help!