Tick Based Indicators: Adding Persistance using Replay Capabilities

[aviat72]

NT does not natively support persistence of calculated values for indicators which use tick based data for calculated values. Compared to other packages, this is a serious limitation which can affect the usage in strategies which build up aggregated logic based on tick data.

When historical data is reloaded, it is reloaded on the basis of bar values and tick based indicators lose all the information. This can not only happen when NT is shutdown or restarted (say because it hangs), but also when the data-connection is re-established (e.g. after an internet outage). Of course something as simple as adding or modifying an existing indicator also triggers the loss of information.

gomifromparis has implemented a workaround where he stores the tick data and reloads it into the indicator. Though the solution is quite elegant, it requires a certain style of writing indicator code. Further since data is maintained in a somewhat ad-hoc manner, it also suffers from file-data integrity related issues.

It would be great if NT natively supports pumping tick data for indicators which require it, when historical data is reloaded. To ensure that this does not slow down data-reload, indicators which require tick based data will have to be explicitly classified as such by some flag.

In the replay mode, NT seems to have even DOM information and seems to drive the indicators in a tick-by-tick mode. So clearly NT has the data-storage, retrieval and driving capability to achieve this on indicators and strategies when historic data is reloaded.

Also the GOMI indicators reload quite quickly confirming that pumping tick data by itself is not going to have any significant effect on chart load speed.

------------------------------------------------------------
My hunch is that the bar-database and the replay database evolved separately and the data from the two is not shared. Hence the tick-data captured by replay is not pumped in when charts are reloaded. Perhaps integrating these two separate databases into one data management system will solve a lot of problems.

A quick solution to this problem would be an API which exposes Replay Data to NinjaScript in the read only mode. So the scripts can use NT's data-storage capabilities while using a GOMI like mechanism to recreate the indicator values on reload.

----------------------------------------------------------------

Also, does NT update the tick-data base when historical tick data is downloaded from the provider to create a chart?

e.g.:Suppose I shut down NT every night at 4:15PM and then restart at 9:00AM. I then request an overnight futures chart which requires tick data, which is downloaded from the data-provider. Will the new tick data be stored in the NT databases assuming that both real time bar-data and market replay recorder is enabled? Will this data be now available for back-testing using range charts and replay feature?

[NinjaTrader_Jason]

Hello aviat72,

I will forward your suggestion to our development team and ask them if they can add this to the list of future considerations for the software.

Correct, Market Replay data is saved separate from historical data in the database.

It depends on the data request if historical data is loaded from the data feed provider. Please see the following link for more information.

http://www.ninjatrader-support.com/HelpGuideV6/HowDoesNinjaTraderBuildChartBars.html

Once histoircal data is loaded, it will be stored in the database.

[aviat72]

Data Management Enhancement Suggestions

Jason:

Thanks for your response.

A followup to your response: If NT downloads historical tick data to build range bars for the period it was not connected to the provider, where it will be stored?

-> Will it be stored in the historical bars database or the replay database or both?

------------------------------------------------------------------------
As I review NT''s capabilities as an automated system development tool, one area which definitely needs investment is the data-handling system.

This is one of the most mundane tasks and can be easily automated. However, often this requires a lot of manual intervention which becomes a pain. Currently I will have to use QCollector to download the data, then export it to the NT format in a file, than import it into NT and so on.


Right now NT duplicates data stored the regular historical data to build bars and the replay data. These two exist as separate entities and require duplicate disk space. Using the GOMI interface forces another third database to be maintained on the disk apart from disk space issues.


-----------------------------------------------------------------------

What I would like is a Database utility for NT. The objective would be to have a unified database model for replay and historical data which would be maintained automatically and be available on demand.

This could be done in the following time-line:

PRIORITY LEVEL 0: Make Replay Tick Data Available to NinjaScript.
Provide an API for the NinjaScript to query replay database and pump tick by tick data. The scripts will then use that data to build up the indicator information as it is done in the GOMI model. However instead of creating a separate tick data-base which GOMI is doing, use the data already existing inside NT's Replay Database.

The GOMRecorderIndicator provides the abstraction layer which reads in the tick data from the GOMI database files and then pumps it to the indicator when needed. Similar APIs on top of the replay database will take care of most of the needs.

PRIORITY LEVEL 1: Use Available Tick Data.
(1) Pump in the tick by tick data to indicators which need them when reloading historical data as it is a live chart.

An Implementation Suggestion:
Inside NT create an abstraction layer which queries both the replay and the historic bar database and presents a unified facade to provide whatever information is needed.

Thus from the client point of view, the NT historical bars and the replay databases will exist as a single data-source with the additional layer deciding which database to use to get the data from.

PRIORITY LEVEL 2: DATA MANAGER: Maintain a complete tick database transparently with automated downloads and updates.

(1) Maintain a complete tick data base for the selected instruments and download historical data from the vendor automatically. Depending on the vendor historical data is available for anywhere from 3 to 30 days going back.

(2) Also maintain other information stored in the replay setup (e.g. DOM-Level 2) synchronized with the tick data.

PRIORITY LEVEL 3: Expose API to the unified database
(1) Expose an API which allows Ninja Scripts to query the database if needed for specific tasks. For example, I would like to see how the CL contract behaves between 9:00AM and 11:30AM on the Wednesday's when the EIA oil report is released going back 6 weeks to build some kind of a volume-price action based indicator or a statistical distribution based indicator which I will use in my strategy. CL can easily move 30-40 ticks in a second or two and without having the tick resolution you can miss out on some details.

(2)This API could also be a part of the ntdll so that other C#/.NET programs can query the database to build additional program logic outside the NinjaScript environment.

EVENTUAL GOAL
Use a single on-disk database for both historical and replay data to avoid duplication of disk storage.

[NinjaTrader_Ben]

Hello,

Thank you! I will send this to our development team for consideration.

[eDanny]

I like the ideas but there are always problems with what you are asking. First off, you would need to always run Market Replay Recorder to save the tick data in the newly accessible database. Any interruption would cause a loss of data. The biggest problem with reconciling live/saved tick data with historical data is they never match, even on a non-tick bar to bar basis. That is what causes your charts and indicators to look vastly different when reloading historical data during live sessions. There seems to be no way to get around this, believe me, I have been round and round with support on this issue. Basically what I am saying is if the data on the historical ZenFire (for example) server is considered correct, then your live charts are always inaccurate. Hopefully, someday all this will be fixed somehow and live data is real and live tick data can be reloaded on the fly.

[aviat72]

eDanny, I am sure most people who are looking for such features recognize that there are going to be bar to bar differences. The very fact that what I am looking for is tick data, means that my strategies are relatively less sensitive to the bar construction issue and more driven by tick data.

Also there is nothing which prevents NT to query and fill up the tick database after a loss of connection. Most paid data-feeds use time-stamps and you can ask NT to stick to those timestamps and ensure that at least the data is consistent with what the vendor provides.