Stop NT8 saving live data as historic data

[NinjaTrader_Manfred]

Hello Lantic,

Thank you very much for your post.

If NinjaTrader creates an entry in the historical data db with no values, this generally indicates that data was requested but was not available. For daily bars going back more than 10 years, this would not be unexpected depending on the provider.

Interactive Brokers generally suggests limiting daily bars to 1 year ( please take a look at their publicly available Historical Data Limitations page in this regard ), which might indicate they do not have data available going back to the 2011 contracts.

Furthermore, continuous contracts (e.g. ES ##-##) would not be supported through this provider. When using the regular expiration (e.g. ES 03-22), with historical data being merged (regardless of whether it is back adjusted or not), NinjaTrader would automatically download the historical data for prior contracts, e.g. the ES 12-21, based on how much historical data is being requested.

The difference between selecting "merge back adjusted" and "merge non-back adjusted" would be whether historical data for expired contracts is being adjusted, but would not directly affect the data itself, only the offset it is displayed with.

Last but not least, regarding the way NinjaTrader saves this data; while the platform would create the additional files when saving Futures data compared to Forex data, this is an internal process and would not affect the data or results. There is also no way to manually adjust how this data is being saved.

Should you have any further questions, please don't hesitate to let us know!

[Lanticalt]

Hi,

I guess what I am trying to say is, all futures contracts are not being merge because the NCD files, for each contract I download, are created independently in the db file instead of being put all together in a single CL file. The AUD/USD have all the NCD files inside all together, while the CL 01-02 files, etc. have been separatly created by contract date even if the MERGE non back adjusted is check in the option menu. My understanding is the system should creates a CL file and put all the NCD contract files downloaded into the db file call CL. Instead, It create daily file in which NCD file are put. My db file is clutter with those for all futures I have downloaded: CL, ES, GC, ZB, ZE, ZN, etc. But all forex pair downloaded does not do that.

[NinjaTrader_Manfred]

Hello Lanticalt,

This would indeed be the intended behavior. In short, the AUD/USD is a single instrument, whereas e.g. the ES 12-21, ES 03-22 and ES 06-22 are, theoretically speaking, three different instruments. NinjaTrader then merges the historical data of the two older instruments and adjusts this (if selected) to backfill historical data for the current front month.

While some providers perform this on the provider side, which results in the endless contract being offered (ES ##-##), the NinjaTrader platform would also be able to perform this locally, if selected. This happens in real-time as the data loads, and would not affect the downloaded data, which is why the individual Futures contracts would still be listed each as an individual instrument, rather than to have their historical data combined.

A good example of this can be seen when instruments are individually charted. The FGBL, for example, has a suggested rollover date of 3/8 (today). If we chart the FGBL 03-22 and 06-22 individually, you can see that both instruments have slightly different candles, a different real-time price, and volume:



If NinjaTrader now back-merges this data, it would use the historical data for the FGBL 03-22 from 3/7 going backward but would use the FGBL 06-22 from 3/8 onward, adjusted based on the value entered in the instrument itself.

With back-merge disabled, only the actual expiration would be charted.

I hope this explains a bit better how this data is stored and managed. Should you have any further questions, please let me know!

[Lanticalt]

OK,

There is no way I can make those files come all together ?

The contract are store as if they were individual but they are actually been merge non back adjusted. The definition says:

Merge non back adjusted will merge past contract months on the specified roll date, but will not adjust for any offset between the contract months.

To me if that is the case, the file should all be put in the same folder name under the future contract using the starting date and the last date available but I am not a computer guru so. I guess that is what CL ## ## us to do. Was that only for continuous adjusted or only for continuous non back adjusted? Why was this option removed?

Now, weither the files are all separated or not, my questions now are about the historical data update.

I want to make sure every historical data symbol downloaded yet from IB get updated with new data from the moment I close the software last week and reopen it today. For example. I downloaded historical data last week for many IB symbols. I have not reopen NT8 since. Base on the fact that the update historical box is check in the options window, I expect all my historical symbols to update the moment I log in into NT8 and connect to IB server. My understanding is, that it will only do that if the actual chart of that symbol is open and use. The other would update later when I open them the next time or any other day in the future?

Meaning also that if I don't click the reload data option, it means that only the new data since the software was close last week would be added to the symbol I open a chart for. Meaning historical data for that symbol will automaticly update the data whitout having to do anything else except connecting IB server on lauching NT8. Once I close NT8, my historical data have been updated up to that very last bar for each symbol I use that day. No previous data has been corrupt. Only the missing data since las open is added next to it?

Regards,

[NinjaTrader_Manfred]

Hello Lanticalt,

Continuous contracts (e.g. ES ##-##) have not been removed. If your provider supplies data for continuous contracts (e.g. Kinetick), you can chart the ES ##-##. However, neither Interactive Brokers nor NinjaTrader Continuum offers continuous contracts, this is why the NinjaTrader platform has the option to automatically merge and display this 'continuous' contract locally by using historical data of expired contracts.

NinjaTrader calculates this on the fly out of the various historical instruments, it does not modify the existing data that was saved or merge it locally. This ensures that the historical data itself has not been modified and can always be reused, e.g. if you only wish to chart an expired contract. This is intended and simply how the platform manages historical data.

The way historical data loads is as follows:

• If you open a chart for any instrument, NinjaTrader will first connect to real-time data.
• It now loads all historical data it has available on your hard drive for this instrument up to the requested "Days to load"
• Finally, it checks whether the already downloaded historical data suffices, or if the chart you opened requires more historical data than it has available, before requesting the missing data from your provider.
• If you already have all data available, no further data is being requested

If you click "Reload all historical data", it would always reload the whole chart, foregoing any data it had saved prior. This is beneficial e.g. if you switch from one provider to another, and want to make sure you use their historical data.

To give you a more visual example. If I disable the feature to merge historical data and request a daily chart for the last 10 years, NinjaTrader would only request data for one instrument for this timeframe:

2022-03-09 05:22:19:332 Cbi.Instrument.RequestBars (to Provider): instrument='ES 03-22' from='3/11/2012 12:00:00 AM' to='3/10/2022 12:00:00 AM' period='Daily'

When I chart this instrument, you can see that historical data prior to the rollover date is lacking and no data is available prior to 2021:


If I enable the option to merge historical data, NinjaTrader would first request historical data for all past contracts for their respective trading timeframes (in this case this would be 40 different expirations):

...
2022-03-09 05:24:05:082 Cbi.Instrument.RequestBars (to Provider): instrument='ES 06-18' from='3/4/2018 12:00:00 AM' to='6/10/2018 12:00:00 AM' period='Daily'
2022-03-09 05:24:05:082 Cbi.Instrument.RequestBars (to Provider): instrument='ES 03-18' from='12/3/2017 12:00:00 AM' to='3/11/2018 12:00:00 AM' period='Daily'
...

Before it merges these contracts on the chart directly:



While you are still viewing the same instrument, and its historical data has not changed, NinjaTrader uses past expirations to backfill any missing information, creating a continuous version of this contract locally within your platform.

If I close NinjaTrader and reopen a daily chart for this instrument in three days, NinjaTrader would only request the historical data for the missing three days, while using the locally stored data for the rest.

I hope this clarifies exactly why this data is not saved in a single, combined instrument for Futures contracts, and how this data is being requested/saved. Should you have any further questions, please let me know!

[Lanticalt]

Appreciate cthe larifications.

I will eventually download data from IQ feed. I will request merge non back adjusted data for all my futures.
How will NT8 deal with that new provider data. I know I need to pick the provider for each symbols I want IqFeed.
But I mean, if IQFeed provide merge non back adjusted data, how will the data be handle by NT8 in the db file.
I guess it all depend on how IqFeed build the data?

By the way. is merge non back adjusted the same as continuous non back adjusted?

For example, if I download ES back to 2007 on IqFeed (merge non back adjusted or whatever they call it as long as the result is the same)
how will the data be created in the db file.

Regards

[NinjaTrader_Manfred]

Hello Lanticalt,

If you download a continuous contract (e.g. ES ##-##), the "merge back-adjusted" part is taken care of by the provider. All the data your receive is then already pre-merged.

Whether you merge back-adjusted or non-back-adjusted, data would always be handled in the same way and would be saved per the individual instrument.

Only once you select to either merge back-adjusted or non-back-adjusted would it merge the data, and either adjust it by the offset or not.

In general, we expect that data for the current front month that is back-adjusted to reflect a regular continuous contract very closely. Or to say it this way: Whether you use Kinetick's or IQFeed's ES ##-##, or chart the ES 03-22/06-22 and back-adjust, the chart should be close to identical.

[Lanticalt]

Hi manfred,

Is there a way to add a symbol description in market analyser? Some of my symbols are frequently weird and not always easy to remember, Especially when you have 20 or 30 of them all mixup by high and lows, etc. It seems to me a really basic option to provide. the Description column will not find the symbol description automatically. So, I took an hour to write every symbols description manually into the note column, but everything was erased the minute I closed the market analyser. Even if I had saved it as a template. Really frustrating. In short, I need to see what the ticker do in order to be able to use MA properly. Is there a way to add a description of the symbol. It would be nice to be able to copy paste a description into the description column when the system cannot retrieve it automaticaly. Or, it would be nice if a descriiption column could be added in the import section under stock list. I am not a programmer so I don't know how compicated this is but it seems to me a description of the symbol is a basic option which should be provided...

[NinjaTrader_Manfred]

Hello Lanticalt,

The "Description" column in the market analyzer pulls the information directly out of the instrument manager.

E.g. if I add it on my Market Analyzer, this is the information it displays:



This applies to all default instruments already part of the platform. To add this description to new instruments you've added, please click on Tools > Instruments, then search for the instrument in question and double-click it.

Once you've entered a Description therein and saved the instrument, it would then appear in the "Description" column of your market analyzer.

Please let me know should you experience any issues setting this up!

[Lanticalt]

About export db historical data options in NT7 and NT8, There use to be an option to export by date in the export data manager window in NT7. I do have a data folder in NT7 DB folder but the data does not show up in the manager under export. For NT8, I can use export from load tab but only a certain number of lines are exported to .txt, Then I have to import into excel using the process... The edit side in NT8 only offer to export one day at a time which I found silly.

1. Why did you removed the export by date option in NT8?
2. Why am I not seeing my data in the export data manager window in NT7?
3. Is there any software, plugins, script, which would allowed me to export the entire content of a 1 minutes instrument which goes back to 2006?
I will not export this one day at a time, so there must be a way...

Regards,

Lantic

[NinjaTrader_Clayton]

Hello Lanticalt,

Thank you for the questions.

Within NinjaTrader 7 and NinjaTrader 8, it is still possible to export a large range of data by selecting the date. Please see the current interface for each below:

1. You are still able to export by date within NinjaTrader 8.
2. Please go to the location shown above and check what is showing there, if it is empty or missing Historical Data please send us a screenshot of what it looks like and what is missing
3. I am not currently aware of any 3rd party NinjaScript add-ons or strategies that would add this functionality. I suggest looking around the Forum or doing an internet search to see if there is anything out there that would add this functionality.

Please let us know if we may be of any further assistance.