Why is most replay data not being recognized?

[timmbbo]

Hi there, why is most of my replay data not be recognized?

I've been having problems running the Optimizer. Even though I seemingly had all of the replay data, the Optimizer reports zeros across the board. (I'm getting my replay data from the Market Replay Data by the Intentional Trader, so I'm pretty sure it's good.)

If you look at my screen shot, it shows an example. I'm using data from YM 03-22. Within the Historic Data Manager, you see that I only have data for a couple of days, Feb 22-27 2022. Yet, when I go into the /replay directory, all files for each day is there.

So, I do not understand why all data files in /replay are not being seen by Historic Data Manager (and can be used when I run the Optimizer.)

Thanks

[bltdavid]

You are confusing yourself.
You are using the Historical Data manager to look at Historical Tick data
for 'YM 03-22', but that data is not stored in the Replay folder.

Why? because it is 'Historical' data, not 'Market Replay' data.
See attached screenshot.

Your Windows Explorer window is showing the Replay folder, which is
storing Market Replay data, not Historical data.

-=o=-

Strategy Analyzer does not use Market Replay data.

The only component within the NinjaTrader Desktop platform that uses
Market Replay data is the Playback Connection.

In the screenshot you provided, the Historical Data manager is showing
the Historical data stored in the Tick folder for 'YM 03-22' for 4 days in
February.

If you use Windows Explorer and goto the folder 'db\tick\YM 03-22' you'll
see 4 files with the extension 'Last.ncd' that represent what the Historical
Data manager is showing in your screenshot.

-=o=-

It's confusing, so let's be very precise.

Market Replay data is not Historical data. (Note the big 'H')
Market Replay data, is, of course, historical data. (Note the little 'h')

It's very easy to get these concepts confused, esp in speech patterns, and
esp when someone's writing does not pay attention to capitalization to
maintain and highlight the difference.

Market Replay data is it's own thing, downloaded separately, stored
separately in the Replay folder, and used by a completely separate
part of the program, typically referred to as the Playback Connection.

Note that Historical data is the data stored in the Tick, Minute, and Day
folders.

Note that Historical data is the only kind of historical data used to populate
the prior historical bars in a Chart window.

Note that Historical data is the only kind of historical data used by Strategy
Analyzer.

[Note that I used the 'Note that' prefix to clearly show the 'h' vs 'H'
distinction of Historical data, which is a very specific thing, vs historical
data, which is more an adjective, like a category, so that we can talk
about historical data vs real-time data. The word 'historical' is a very
overloaded term. Note that there is also a State.Historical, which
represents the period where historical bars built from Historical data
is being processed by OnBarUpdate. Once real-time data is being
processed, State will change to State.Realtime -- this state change
also occurs with the Playback Connection. After you press 'Play',
the first bar painted into the Chart window is coming from Market
Replay data, but from the point of view of the chart, it is 'real-time',
such is the nature of Market Replay data and the time-warping
semantics of the Playback Connection -- the whole idea of Market
Replay is how to internally lie to a Chart window.]

Market Replay data and Historical data are not mix and match. These
data files are both historical data (ah, note my strict use of 'h' vs 'H'),
but there is no conversion between them. They have different file
extensions as well.

The only thing in common between Historical data and Market Replay
data is the fact that, yes, they absolutely are both different categories
of historical data, but more importantly (and perhaps confusingly) the
Historical Data manager let's you inspect and download both of them.

Remember, Market Repay data is a kind of historical data, but it is
not Historical data. Note the 'h' vs 'H', this is critical.

Again, see attached screenshot.

[NinjaTrader_BrandonH]

Hello timmbbo,

Thanks for your post.

bltdavid is correct. The Strategy Analyzer uses only Historical data to run backtests and optimizations on a NinjaScript strategy. The Strategy Analyzer does not use Market Replay data.

Market Replay data is only used for the Playback connection which allows you to replay data on a chart or SuperDOM as if it were running in real-time.

You would need to ensure you have enough Historical data downloaded for the instrument and timeframe that you are wanting to run optimizations on in the Strategy Analyzer.

To download Historical data, go to Tools > Historical Data > Download. You could also see the amount of Historical data you have downloaded on this window under 'Historical data'.

Downloading Historical Data: https://ninjatrader.com/support/help...8/download.htm

Optimizing a strategy in the Strategy Analyzer: https://ninjatrader.com/support/help...a_strategy.htm

Please let me know if I may assist further.

[jeronymite]

NinjaTrader_BrandonH says

Market Replay data is only used for the Playback connection which allows you to replay data on a chart or SuperDOM as if it were running in real-time.

Well, not exactly. The data is "replayed" in (at best) one second chunks. There is no intra-bar display of any data. If the speed of replay increases, further variations to the manner of display are necessarily made.

As for Replay vs Historical data, the Playback Connection actually allows the selection of either of those. It seems to not be limited to just Replay data. Perhaps NinjaTrader could elaborate on what exactly happens differently when choosing one or the other, aside from the actual data source? Obviously, there will be no level 2 data, but what else is different?

Thanks.

Update: NinjaTrader_BrandonH has provided some additional information here, but a fuller explanation would be welcome.