correct processing of historical and real-time L1 tick data

[pmbi00]

Hi,

I want to make sure I understand the correct processing of historical & real-time Level 1 tick data (Datetime, Price, Volume, Bid/Ask Prices at time of trade).

The main use cases I'm interested in:

a. Historical back-fill of charts, indicators and SuperDOM columns for trading setups

b. Real-time updating of charts, indicators and SuperDOM columns while trading

c. Accessing more data than is currently displayed within either a chart range or the DOM price ladder

d. Using Tick Replay

As far as I understand, there are 3 approaches:

1. A multi-series approach:

- using AddDataSeries(xxx,BarsPeriodType.Tick) within OnStateChange() and State == State.Configure

- using GetClose(), GetVolume(), GetBid() and GetAsk() in OnBarUpdate() with Calculate == Calculate.OnEachTick in State == State.SetDefaults

- there could be is a synchronising problem with backfill as historical data is stored as separate data series for Last/Bid/Ask

- this synchronisation problem maybe overcome if historical L1 tick data is available/loaded from your data provider

- dependent on how much historical L1 tick has either been downloaded or is available from your data provider

2. Tick Replay approach (uses Market Replay data)

- with the Tick Replay option set, accessing Last,Volume, Bid/Ask Price data within OnMarketData() for MarketDataType.LastMarketDataType.Last

- there is no synchronisation problem with this approach

- this approach handles be more CPU intensive as higher-level bars would also be build from tick data

- dependent on how much Market Replay has been downloaded

3. Using BarsRequest

- creating a BarsRequest for a desired start/end datetime and a BarsPeriodType.Tick

- using OnBarUpdate() and BarsUpdateEventArgs to process real-time updates

- using OnRender() to display data (in either a chart or a SuperDOM column)

- dependent on how much historical L1 tick has either been downloaded or is available from your data provider


My questions:

1. Is my understanding in general correct? Or have I missed something?

2. If historical L1 tick data is available from your data provider (I'm using Continuum), will the L1 tick data always be provided & processed correctly ie synchronised ?

3. Are there any further requirements to make sure this works correctly - eg within the various OnStateChange() states?

4. When using BarsRequest, where should you perform an initial load of data ? I'm thinking State.Configure...

Many thanks.

[NinjaTrader_BrandonH]

Hello pmbi00,

Thanks for your post.

It seems that most of your general understanding is correct.

That said, Tick Replay is an option that allows you to ensure that the market data (bid/ask/last) that went into building a bar is loaded in the exact sequence of market data events. It is not dependent on the Market Replay data you have downloaded.

Please see the forum post from my colleague Manfred linked below detailing the differences between Market Replay data, Historical Data, and the Tick Replay option to fully understand each of these.

https://ninjatrader.com/support/forum/forum/ninjatrader-8/platform-technical-support-aa/1139772-historical-data-market-replay-and-tick-replay?p=1139787#post1139787

This help guide page details the use of AddDataSeries() to add additional bars objects to a script: https://ninjatrader.com/support/help...dataseries.htm

See the help guide documentation below for more information about Tick Replay and developing scripts for Tick Repaly.

TickReplay — https://ninjatrader.com/support/help...ick_replay.htm
Developing for Tick Replay -
https://ninjatrader.com/support/helpGuides/nt8/developing_for__tick_replay.htm?

You could view this forum thread for more information about using BarsRequest in a custom NinjaScript: https://ninjatrader.com/support/foru...46#post1124546

And, see this help guide page about BarsRequest: https://ninjatrader.com/support/help...ub=barsrequest

Let me know if I may assist further.

[pmbi00]

Hi,

Thank you for your reply.

Actually, I understand that Tick Replay throws either downloaded market replay data or historical data from your data provider in the same way as real-time ticks are received - consequently OnMarketData() and OnBarUpdate() events are in effect in sync.

This means you can use either OnMarketData() or AddDataSeries(BarsPeriodType.Tick, 1) to process L1 tick data - but, if using AddDataSeries(), you need to provide additional code/logic in OnBarUpdate() to handle the differences between real-time / historical processing and Calculate.OnBarClose / OnEachTick.

I found this link to your BuySellVolumeOneTick example which I'm working through to develop a script that can handle Tick Replay, real-time or historical data:

https://ninjatrader.com/support/forum/forum/ninjatrader-8/indicator-development/1182308-onmarketdata-during-issuspendedwhileinactive

I think I'm on track now, right?

Many thanks

[NinjaTrader_BrandonH]

Hello pmbi00,

Thanks for your note.

The BuySellVolumeOneTick reference sample would use the 1-Tick data series to calculate logic and it is not recommended to use the script with TickReplay.

Tick Replay shouldn't alter the result for a time-based historical bar. We are aware there are discrepancies between the OneTick approach and tick-based historical bars, but time-based bars should be consistent when compared to the default BuySellVolume indicator.

Please let me know if I may assist further.

[pmbi00]

Hi,

Ok, if we exclude TickReplay, what is the recommended way to process real-time and historical L1 tick data ? Is there a best practice sample script to do this?

Many thanks.

[NinjaTrader_BrandonH]

Hello pmbi00,

Thanks for your note.

After further reviewing the version of BuySellVolumeOneTick that you linked in post # 3, I see that the script is already programmed to support Tick Replay so this script may be used with Tick Replay enabled.

Note that this script will calculate historical and real-time tick data by default and does not require Tick Replay to be enabled since it calculates from a one-tick series. This could be seen by enabling the script onto a chart and testing this script with Tick Replay disabled. Then you could enable Tick Replay and view the indicator on a chart to see that the historical and real-time values are the same.

Let me know if I may assist further.

[pmbi00]

Hi,

Thank you for your note and clarification.

I have this sample script working now for real-time, historical and Tick Replay data. :-)

Though I will probably have a few questions about the logic shortly...

Many thanks.