Function of /db directories

[timmbbo]

Hi there, when is the data in the /day /minute /tick directories accessed? For what functions?

I frequently download replay data, for back testing purposes. I notice all replay data is placed in /replay, which makes sense. Given this, what functions would access the data in the /day /minute /tick directories?

Thanks

[bltdavid]

There are two major categories of historical data.

The first is 'Historical' data - downloaded as Tick, Minute, or
Day data.

The second is 'Market Replay' data, which is downloaded in a
form suitable for the Playback Connection.

Even though Historical data and Market Replay data are both a
type of historical data (note the 'h' vs 'H'), these data types are
not interchangeable. They use different file formats and their
filenames use different extensions.

-=o=-

Historical Tick, Minute, and Day data (downloaded or recorded) is
stored in folders of the same name. On my 8.0.26.1 system, the
folders are all lowercase, but case doesn't matter on Windows, so
the 'tick' folder is the same as the 'Tick' folder, as far as Windows is
concerned.

Market Replay data used by the Playback Connection (downloaded or
recorded) is stored in the Replay folder.

Which parts of the platform use this data?
Chart windows and Strategy Analyzer windows are the only two I know.

The Replay folder is only used by the Playback Connection. But the
Chart window 'Market Replay' experience actually involves both kinds
of historical data. Yes, both. Let me explain that, see below(*).

Whenever a Chart window requires historical data to populate prior bars,
(which is needed by every chart, every where on planet earth, by every
charting software package) regardless of current connection status and
type, (live, Playback, Simulated Data Feed, IQFeed/Kinetick, Kinetick End
of Day, CQG/Continuum/Rithmic, etc, etc, or even no connection at all)
NinjaTrader first checks the 'cache' folder, then it checks the Tick, Minute,
and Day folders, as dictated by the bar type, before finally trying to
download the needed Historical data from one of the available connections
(using your preferred connection first) -- assuming a connection with the
appropriate data has been made. Note that connecting to 'Kinetick End
of Day' Connection does not help with charts that require Tick or Minute
Historical data, but should completely satisfy the Historical data needs of
all Day/Week/Month/Year charts.

Also, the Strategy Analyzer uses Historical data for backtesting. That is,
all back-testing & optimization done with the Strategy Analyzer uses
Historical data, that means data from the Tick, Minute, or Day folders,
as dictated by the bar type.

The Strategy Analyzer never uses Market Replay stored in the Replay
folder -- it's only designed to use Historical data.

(*)If you're using the Playback Connection, the Historical data folders will
be accessed (and perhaps the 'cache' folder) to populate the charts prior
to your designated start time for the replay. After you press 'Play', the
forward data for the new bars comes from the Market Replay data itself,
which comes from the Replay folder.

[EDIT: The Playback Connection can also use Historical Tick data, but
I think this is less accurate. However, you can download 12mons of
Historical Tick data vs only 3mons of Market Replay data. Obviously,
you'd have to stay self-disciplined and download data on a regular
schedule to build up your supply of 'old' Market Replay and/or
Historical Tick data.]

-=o=-

Note that charts like Renko, Range, NinzaRenko, UniRenko, as well as
Tick and Volume charts -- these charts all get their historical data from the
'tick' folder, because that's how those bars are formed. Even Second-based
charts, (5-Second, 30-Second, etc) are formed using Tick data.

The other chart types use Minute or Day data. For ex, a weekly, or monthly,
or yearly chart uses Day data.

Make sense?

[bltdavid]

The NinjaTrader database, although seemingly spread out across all
the subfolders in the 'db' directory -- is really just the one file, named
either 'NinjaTrader.sdf' or 'NinjaTrader.sqlite'.

[For years, NinjaTrader used Microsoft SQL Compact Edition, I forget
the exact name, but starting with 8.0.27.0, they have moved to a different
database technology called SQLite, thus the newer '.sqlite' extension.]

And, as you have probably surmised, the NinjaTrader database file itself
does not store historical data -- that's what those folders are for.

[QuantKey_Bruce]

SQL CE went EOL in 2018 which is why it was appropriate to update to SQLite.

[bltdavid]

Some final thoughts ...

The data needed by the Tick Replay feature is stored along
with the Historical Tick data inside the files in the Tick folder.

To wit:
For NT8, the '.ncd' file format was invented to contain all the
Tick data needed for Tick based charts, which also includes
the extra data needed for Tick Replay. This means the '.ncd'
file format is able to include all bid/ask/last data points, tagged
in the exact sequence needed to completely rebuild the bar
from the exact same market data events.

The Historical Minute data stored in the Minute folder also
contains the extra data needed for Tick Replay.

The Tick Replay feature is not constrained to only Tick based
charts.

And finally, even though the names 'Market Replay' and 'Tick
Replay' are similar, they have absolutely nothing to do with
each -- they represent completely different concepts and
features.

-=o=-

In NT7, the Historical data was stored in the '.ntd' file format,
and these files were considerably smaller than the newer NT8
'.ncd' file format. Even so, in my experience, NT8 Historical
data files have always downloaded faster than NT7 Historical
data files.

[NinjaTrader_BrandonH]

Hello timmbbo,

Thanks bltdavid and QuantKey_Bruce are correct. There are two categories of historical data in NinjaTrader, Historical Data and Market Replay data.

Historical data is downloaded as downloaded as Tick, Minute, or Day data and is stored in the /tick, /minute, or /day folders.

The Strategy Analyzer window and Chart window will use the Historical data from the /tick, /minute, and /day folders.

Market Replay data is downloaded to be used specifically with the Playback connection and is stored in the /replay folder. Note that only the Playback connection accesses Market Replay data.

The Playback connection uses the Market Replay data downloaded in the /replay folder.

Note that the Strategy Analyzer only uses Historical data, not Market Replay data, to backtest/optimize NinjaScript strategies.

Please let me know if you have further questions.