Market Replay in Playback Connection DateTime Precision

[jeronymite]

It appears that when using Market Replay data in the Playback Connection, the DateTime (Connection.Now) is, by default, limited to displaying full seconds only. For example, displaying .Now with a format of "hh:mm:ss.fff" always displays ".000" for ".fff" even when the times are clearly sequentially different within the same second. Displaying the real-time clock time in the same output clearly shows real-time milliseconds changing as the Playback milliseconds remain displayed as 000.

Is there a way to access/display Playback Connection times with millisecond precision?

Or is it the data (downloaded from NT)? Isn't it supposed to be tick data? It appears to be 1 second data. How does one obtain tick market replay data if this is not it?

Thanks.

[NinjaTrader_ChrisL]

Hi jeronymite, thanks for posting. I am getting millisecond values with this code using a 100Tick chart:

protected override void OnBarUpdate()
{
 Print("Time[0] " + Time[0].ToString("MM/dd/yyyy hh:mm:ss.fff tt"));
 Print("DageTime.Now " + DateTime.Now.ToString("MM/dd/yyyy hh:mm:ss.fff tt"));
 Print("");
}

The Time[] timestamps will not have a millisecond value if you are not using Tick based charts, the DateTime.Now value should display milliseconds given you have the correct formatting because this is just the sytem time stamp at the time of execution.

Kind regards,

-ChrisL

[jeronymite]

Thanks, NinjaTrader_ChrisL. I use the Playback Connection.Now value and it does not display milliseconds. Charts are irrelevant in this particular case, because I am displaying from an AddOn using BarsRequestors for the data, and simply using Connection.Now as the timestamp, and it always displays as ".000". The same, in fact, happens if I use Historical data in Playback, instead of Market Replay. In the Historical Data window, I can see that the historical data timestamps have millisecond precision, but displaying Connection.Now still shows only ".000". Furthermore, if I display the Time component of an update in the BarsRequestor, it also only displays ".000".

Your further advice appreciated.

Thanks.

[NinjaTrader_ChrisL]

Hi jeronymite, thanks for the follow up. Im not able to find Connection.Now documented anywhere. If you found this in the help guide please send me a link to it.

https://ninjatrader.com/support/helpGuides/nt8/NT%20HelpGuide%20English.html?connection_class.htm

If this is not documented, it might not be meant to be used and there for not supported.

Kind regards,
-ChrisL

[jeronymite]

NinjaTrader_ChrisL It's probably not in the Help because the section on the Playback Connection is "broken" (i.e. not written, just the template). Nevertheless, if you declare any NinjaTrader.Cbi.Connection, Intellisense offers .Now as one of its fields. It is common to every Connection, so real-time Connections also have it.

The question remains: In an AddOn, how does one obtain millisecond precision times for the Playback Connection, for both Market Replay and Historical data obtained using BarsRequestors?

Thanks.

[bltdavid]

Looking at the @BarTimer.cs, I see this code,

private DateTime Now
{
    get
    {
        now = (Cbi.Connection.PlaybackConnection != null ? Cbi.Connection.PlaybackConnection.Now : Core.Globals.Now);

        if (now.Millisecond > 0)
            now = Core.Globals.MinDate.AddSeconds((long)Math.Floor(n ow.Subtract(Core.Globals.MinDate).TotalSeconds));

        return now;
    }
}

jeronymite, did you try using Cbi.Connection.PlaybackConnection.Now?

[jeronymite]

Thanks, bltdavid. Yes, that's exactly what I'm using. It just does not seem to have fractional seconds resolution.

Even when I can see in the Historical Data window that the Historical data has multiple data within a second and timestamp millisecond resolution, Now does not resolve beyond one second.

Investigating further, running a Market Replay across a period covered by an NFP announcement is very revealing. Printing from the BarsRequestor OnBarUpdate method shows that data is received at approximately one second intervals ... only!

Example test output looks like this:

2022-12-13 15:10:49.763  2022-10-07 22:29:54.533  2022-10-07 22:29:55.000  144.983 144.962
2022-12-13 15:10:50.763  2022-10-07 22:29:55.390  2022-10-07 22:29:56.000  144.979 144.963
2022-12-13 15:10:50.764  2022-10-07 22:29:55.390  2022-10-07 22:29:56.000  144.979 144.963
2022-12-13 15:10:51.764  2022-10-07 22:29:56.415  2022-10-07 22:29:57.000  144.969 144.957
2022-12-13 15:10:51.764  2022-10-07 22:29:56.415  2022-10-07 22:29:57.000  144.969 144.957
2022-12-13 15:10:52.780  2022-10-07 22:29:57.708  2022-10-07 22:29:58.000  144.968 144.957
2022-12-13 15:10:52.780  2022-10-07 22:29:57.708  2022-10-07 22:29:58.000  144.968 144.957
2022-12-13 15:10:53.780  2022-10-07 22:29:58.901  2022-10-07 22:29:59.000  144.954 144.925
2022-12-13 15:10:53.781  2022-10-07 22:29:58.901  2022-10-07 22:29:59.000  144.954 144.925
2022-12-13 15:10:54.782  2022-10-07 22:29:59.944  2022-10-07 22:30:00.000  144.948 144.912
2022-12-13 15:10:54.782  2022-10-07 22:29:59.944  2022-10-07 22:30:00.000  144.948 144.912
2022-12-13 15:10:55.795  2022-10-07 22:30:00.528  2022-10-07 22:30:01.000  144.946 144.890
2022-12-13 15:10:56.798  2022-10-07 22:30:01.632  2022-10-07 22:30:02.000  145.068 145.028
2022-12-13 15:10:56.805  2022-10-07 22:30:01.632  2022-10-07 22:30:02.000  145.068 145.028
2022-12-13 15:10:57.797  2022-10-07 22:30:02.685  2022-10-07 22:30:03.000  145.126 145.044
2022-12-13 15:10:57.797  2022-10-07 22:30:02.685  2022-10-07 22:30:03.000  145.126 145.044
2022-12-13 15:10:58.817  2022-10-07 22:30:03.810  2022-10-07 22:30:04.000  145.147 145.042
2022-12-13 15:10:58.839  2022-10-07 22:30:03.810  2022-10-07 22:30:04.000  145.147 145.042
2022-12-13 15:10:59.836  2022-10-07 22:30:04.885  2022-10-07 22:30:05.000  145.140 145.071
2022-12-13 15:10:59.837  2022-10-07 22:30:04.885  2022-10-07 22:30:05.000  145.140 145.071​

where the lines contain: DateTime.Now, MarketData.Last.Time, PlaybackConnection.Now, and Ask & Bid of USDJPY.

By this test, it would seem that the Playback Connection is not playing back as it should, or the Market Replay data is not really market replay tick data! Interestingly, MarketData.Last.Time has milliseconds.

One more thought: the BarsPeriod of the BarsRequest is 1 Minute. Would that make a difference in all this? It certainly makes no (discernible) difference in all the other uses with real-time data, even with actions performed multiple times per second, so it would be surprising. Those data are resolved at milliseconds with real-time data changes intra-second. In any case, I have tested with a 100 Tick BarsPeriod for the BarsRequest and it does not change things.

Curiouser and curiouser!

Thanks.

[NinjaTrader_ChrisL]

Hi jeronymite, I've asked about this blank documentation page on Addon.Connection.PlaybackConnection and it should be resolved shortly. It is expected for the playback connection to pump data every one second. This is why Alerts will sometimes not work when you run the playback connection at full speed.

[jeronymite]

NinjaTrader_ChrisL So Market Replay in Playback is not streaming tick data?! Yet the documentation says it is the full market data stream. How can this supposedly full tick data stream only provide data once every second? That makes zero sense, particularly when the existing documentation strongly implies otherwise. The documentation needs to be explicitly clear that this is how it operates. And if that is how it operates, do not claim it to be a full data stream -- it is not. Even the Historical Data window shows historical data with lots of intra-second data, yet that data is not streamed in Playback? This bizarre limitation makes Playback much much less useful! If I run Playback at 1x speed, I expect to see every stored tick of data. At higher speeds, I can see it might need to "curtail" some data delivery ... but in "playback real-time" there is no reason to not deliver all tick data. Truly unfathomable.

Thanks.

[NinjaTrader_ChrisL]

Hi jeronymite, I was not quite correct, my apologies. The information is documented here:

https://ninjatrader.com/support/helpGuides/nt8/?playback.htm#UnderstandingHowThePlaybackWorks

Visually, the charts are updated every 1 second but indicators and strategies will calculate as they are running in real-time. I can't reproduce it using an indicator, the millisecond data is there, so please post a sample that can reproduce the issue.

[jeronymite]

NinjaTrader_ChrisL Thanks, Chris. There's lots to know, so I understand how something can be missed.

I am sending you a trace and log file by email (Attn: ChrisL) that show a SQLite error when connecting to the Playback Connection. That is an investigation in itself, independent of the current issues, even if it may be relevant at some point.

On the issues at hand though, I am working on a small demo AddOn to try to show the issue. The error above may be affecting this demo, but it does not affect the ability to show what I have detailed in previous posts when using my production software.

Thanks.

EDIT: I have sent a followup email with the demo AddOn and example output that clearly demonstrates the "once per second" data limitation issue.

[NinjaTrader_ChrisL]

Hi jeronymite, I got confirmation from our development team that PlaybackConnection.Now is a timestamp representation of the playback clock that we see in the UI, so the lack of millisecond values boils down to the fact I mentioned earlier that the UI updates are pumped out every 1 second. To keep things organized, please start a new forum thread about this SQLite error you are seeing and provide an example and steps on how to reproduce it.

[jeronymite]

Thanks for the update, NinjaTrader_ChrisL.

In followup:

• Since PlaybackConnection.Now is essentially an analogue for a one second clock, it is of relatively little use, to be honest; nevertheless, it is good to know this
• I note that the BarsRequestor MarketData.Last.Time can provide full millisecond resolution of the data in that update
• Nevertheless, as demonstrated in the code I sent you, data replayed in what was a heavy load period (NFP) was limited to a minimal one or two ticks per second, on the second -- this is not "market replay" in any sense, and should be addressed as part of the email with example code I sent you

As for the SQLite error, it is very simple to reproduce, as per the trace file sent in email to you: Start NT 8.0.27.0 and Connect to the Playback Connection. That is all I did to generate this error:

SQLite error (17): statement aborts at 50: [SELECT
Id,Account,BarIndex,Commission,Exchange,ExecutionId,Fee,Instrument,IsEntry,IsEntryStrategy,IsExit,IsExitStrategy,LotSize,MarketPosition,MaxPrice,MinPrice,Name,OrderId,Position,P

I look forward to these issues being understood and resolved.

Thanks.

[NinjaTrader_ChrisL]

Hi jeronymite, Im not getting this error upon connecting to the playback connection.

Kind regards,

-ChrisL

[jeronymite]

NinjaTrader_ChrisL With the two issues here (the error on connection, and the actual playback tick stream not being truly an accurate replay of the market), what is the state of investigation with the tick stream issue?

Thanks.