Get current Bid/Ask with tick replay OR get main indicators to update on the 1-tick

[markdshark]

I feel like I've been around the world and back with Ninjatrader development this past year. I've made tons of progress but in a way I'm also back where I started though ahead by a bit. Kinda like following along the coils of a spring (which seems at least to be pointed in the right direction).

Long story shot, using tick replay, my indicators update properly unlike by simply adding a 1-tick series where all trades are being opened at the end of the main series bar. I feel as though I had solved this problem earlier by just adding the 1 tick series but it's back with a vengeance. They are at least closing intrabar but I think that's cos closing is price based and not indicator driven.

I would love to ditch tick replay and have the strategy follow along on the 1 tick series however I need the indicators on the higher timeframe (tickframe?) to react to the way the 1 tick series moves as opposed to just using OHLC of the main series - for 2 reasons:

1. I believe I want to access Bid and Ask market events and not only when a trade occurs (as limited by tick replay) and also do this in a backtest in which indicators follow market movement at the 1 tick level.
2. Speed

I tried adding bid and ask historical series in tick replay but of course discovered they can't be used *with* tick replay enabled. I then backed out of tick replay to using the 1-tick series and found the indicators on the main series don't appear to be responding to the granular level of the 1-tick series. So I feel a bit stuck..

Without using tick replay trades go off at the wrong time (end of bar)
Using tick replay I cannot get at the Bid/Ask series historically.

It's possible I'm just confused at this point but I thought I'd put it all on the forum and see if I could find some help and clarification.

I think perhaps another way of phrasing this is: When not using tick replay, considering indicators on the main series (higher timeframe) and an added 1 tick series, at what point do the indicators on the main series update when they are set to "Calculate.OnEachTick"

And btw *all* indicators and the strategy itself are all set to Calculate.OnEachTick

Thanks!

[NinjaTrader_BrandonH]

Hello markdshark,

Thanks for your post.

OnMarketData() could be used to get Bid/Ask data, such as Bid/Ask price in a NinjaScript indicator or strategy.

From the OnMarketData() help guide:

"By default, this method is not called on historical data (backtest), however it can be called historically by using TickReplay

If used with TickReplay, please keep in mind Tick Replay ONLY replays the Last market data event, and only stores the best inside bid/ask price at the time of the last trade event. You can think of this as the equivalent of the bid/ask price at the time a trade was reported. As such, historical bid/ask market data events (i..e, bid/ask volume) DO NOT work with Tick Replay. To obtain those values, you need to use a historical bid/ask series separately from TickReplay through OnBarUpdate(). More information can be found under Developing for Tick Replay."

The Developing for Tick Replay help guide linked below demonstrates how to access historical Bid and Ask prices with Tick Replay


For backtesting a strategy, you may use Tick Replay along with an added 1-tick series to have logic processed intra-bar and have orders filled intrabar.

When in historical data (backtesting), only the Open, High, Low, and Close will be available and there will be no intra-bar data. This means actions cannot happen intra-bar, fills cannot happen intra-bar. All prices and actions come from and occur when the bar closes as this is all the information that is known.

Because of this, OnBarUpdate will only update 'On bar close' as it does not have the intra-bar information necessary for 'On price change' or 'On each tick' and the script will not have the intra-bar information to accurately fill an order at the exact price and time.

Tick Replay should be used to have the logic process OnEachTick or OnPriceChange with historical data, but this does not allow for intra-bar order fills. You would need to also add a single-tick data series and submit orders to that single-tick data series for a strategy that uses Tick Replay.

See the SampleIntrabarBacktest example and the following Help Guide links for more information.

SampleIntrabarBacktest 'Backtesting NinjaScript Strategies with an intrabar granularity': https://ninjatrader.com/support/helpGuides/nt8/backtesting_ninjascript_strate.htm

TickReplay: https://ninjatrader.com/support/help...ick_replay.htm

Historical Bid/Ask Series: https://ninjatrader.com/support/help..._ask_serie.htm

Developing for Tick Replay: https://ninjatrader.com/support/helpGuides/nt8/developing_for__tick_replay.htm

[markdshark]

Thanks Brandon. I do appreciate the quickness of your reply but in this case you've merely reiterated exactly what I already know and said I was already doing in my post above.

I have been using Tick Replay with OnMarketData. But if I want the Bid Ask events besides those associated with the last trade, I can't have them.

As an alternate approach I moved to a regular backtest and then trades are going off at the wrong place.

Given I have included a 1-tick series I know that my OnBarUpdate is firing with each tick now for the secondary 1-tick series. My remaining confusion is why my indicators on the main series do not appear to be updating at the granular level too. Shouldn't they be?

Mark

[NinjaTrader_BrandonH]

Hello Mark,

Thanks for your notes.

If you do not want to get the bid/ask prices from the last trade and want to get the bid/ask price all the time then you would need to use a historical bid/ask series in your script and get those values from those series and not use Tick Replay and OnMarketData().

Or, you could choose to get the bid/ask prices from the last trade in OnMarketData() and use Tick Replay.

You would not be able to get the historical bid/ask prices other than from the last trade in OnMarketData() when using Tick Replay.

Please review the Developing for Tick Replay help guide page for more information: https://ninjatrader.com/support/helpGuides/nt8/developing_for__tick_replay.htm

Historical Bid/Ask Series: https://ninjatrader.com/support/help..._ask_serie.htm

[markdshark]

Thanks Brandon. But that's the crux of the problem isn't it?

I have been developing using tick replay for the last 5 months so believe me I have reviewed the Developing for Tick Replay guide very thoroughly already.

But...

I can't get the data I want using Tick Replay (which otherwise works for my strategy and backtesting)

AND

If I don't use Tick Replay I can't get my indicators to update on the 1 tick granular level.

You've restated the problem I'm facing... But I'm not hearing a solution.


Best,
Mark

[NinjaTrader_BrandonH]

Hello Mark,

Thanks for your notes.

Unfortunately, that is not how Tick Replay works with OnMarketData(). You are only able to get the bid/ask prices from the last trade when using OnMarketData() with Tick Replay.

If that does not have the data you need then you would have to modify your strategy to use a historical bid/ask series in your script and get those values from those series and not use Tick Replay and OnMarketData().

Historical Bid/Ask Series: https://ninjatrader.com/support/help..._ask_serie.htm

There are only two data formats for tick replay data (second or sub-second granularity) which can be seen on the help guide page below and this is specifically the data it uses.

Understanding File and Data Formats: https://ninjatrader.com/support/help...ngImportFileAn dDataFormats

1. I believe I want to access Bid and Ask market events and not only when a trade occurs (as limited by tick replay) and also do this in a backtest in which indicators follow market movement at the 1 tick level.

The only way to do this is by having a data provider that offers historical ask/bid data as well.

Data by provider: https://ninjatrader.com/support/help...y_provider.htm

[markdshark]

Thanks Brandon. I'm not sure why but your replies do appear to be consistently missing what I'm getting at. Let me try and restate:

1. I know how Tick Replay works and I understand the limitations. I am not trying to get it to do something it does not do.
2. I also know what data I am looking for and I believe I can access it easily when using historical data in a non tick replay setting.

Hopefully we can be on the same page up to this point so we can avoid links to how tick replay works or data providers etc. None of that is relevant

3. Understanding the limitations of Tick Replay I have reworked my code into using historical data in a non tick replay setting. It is at this point that I am finding that the indicators which are running on the main series do not seem to be updating at the level of granularity of the 1 tick series which I have added and my entries are all occurring at the close of the bar.

There should be no confusion thus far.

Here is where the problem lies. There is no point in proceeding to modify the strategy to use a historical bid/ask series until I can assure myself that the indicators on the main series in the strategy are being updated at the level of the 1-tick series.

Maybe answer that question first: Is that not how they should be working?

If there's still confusion, is there someone else at Ninjatrader who can chime in please? Or you can ask me to restate it.

BTW the link you provided to File and Data Formats above is broken.

Best,
Mark

[NinjaTrader_BrandonH]

Hello Mark,

Thanks for your notes.

I have fixed the Data By Provider link on post # 6.

Having the indicators in the strategy update on the 1-tick interval would require setting the strategy to Calculate.OnEachTick and running the strategy in realtime.

To have indicators work on the 1-tick historically you would need a BarsInProgress 1 condition in the indicator and have it add a 1-tick series. Then execute its logic based on BarsInProgress 1.

Plots are still based on BarsInProgress 0 and what the strategy uses so any logic in BarsInProgress 0 or plots will relate to the strategy's primary series.

Please create a reduced sample script that shows the problem in question by using prints and the steps you are using to reproduce the behavior so I may investigate this matter further.

Note that a reduced copy refers to a copy of the script that contains the minimum amount of code needed to reproduce the issue. All other code is commented out or removed.

To export the script to share, go to Tools > Export > NinjaScript AddOn.

[markdshark]

Thanks for continuing to work with me on this Brandon. You may have jogged an idea in my head by talking about adding a 1 tick series in the indicator. I will follow up on that.

I would like to return to my original issue one more time though as I have had some thoughts on how I might progress between these posts. Hopefully you will be able to provide insight after reading this.

Let me restate my goals in a different manner:

1. I developed and backtested a strategy using tick replay that performs at a certain level. However given the limitations of tick replay I am certain it performs better than real market conditions as it does not take slippage into account. I also don't want to hard code a slippage level as that is also an artificial constraint that will perform differently than realtime. Especially if one uses Limit orders.

2. So I would like to test this same strategy by implementing Limit orders instead of market orders.

... All good so far...

3. The problem is (and here I hope is where I am simply missing a piece of information that you or someone else can help with) I can't implement limit orders when using tick replay because I have no way of accessing real Bids and Asks to place a limit order. B'cos of the way tick replay works, the only bids and asks I can access are related to trades already placed and the market may have moved.

This is why I am attempting to go non tick replay to get that info. Am I overthinking this?

Do you see a way around problem 3 (placing effective and relevant limit orders) while remaining in tick replay?

Any other thoughts about my chain of reasoning?

I will add that I do believe one way forward would be to simply develop using tick replay for an accurate representation of how the strategy would perform assuming one will place limit orders in realtime and then simply resort to non-tick replay in real time (since that's tick "replay" by default no?) and add GetCurrentBid/Ask code to use Limit orders with actual current market conditions. Doable yes but then you have two pieces of a strategy historical and real-time and I'd have to test the realtime part separately etc. I was hoping to arrive at a unified solution.

Thanks for fixing that link btw. Yes I'm familiar with that page. Ninjatrader is my connectivity provider which provides historical bid/ask tick data which is why I am not interested in exploring other providers at the moment.


Best,
Mark

[NinjaTrader_BrandonH]

Hello Mark,

Thanks for your notes.

If you are using Tick Replay and are backtesting the strategy in the Strategy Analyzer then the only option would be to get the bid/ask prices from the last trade in OnMarketData().

It would not be possible to get every Bid/Ask price from Historical data. Every bid/ask event is not provided in Historical data from any provider.

The closest information for Historical data is the bid/ask from stamped last data which is what Tick Replay uses to simulate OnMarketData().

A possible workaround you could consider would be to program you script to write every market data event to a file in real-time and then read that data from that file.

StreamWriter: https://ninjatrader.com/support/help...o_write_to.htm
StreamReader: https://ninjatrader.com/support/help...o_read_fro.htm

[markdshark]

Hi Brandon,

Thank you for your response. That is informative.

As it turns out I was spinning my wheels a bit since replacing the custom indicator I was using with standard EMA crosses shows me that indeed the additional 1 tick series causes the indicators to be updated intra-higher-tickframe-bar. So the issue is with my indicator. I thought I had resolved this dang nearly a year ago. Indeed I had.

EDIT: No scratch that. Intrabar still only works with tick replay enabled. Adding a 1 tick series to non tick replay does not update the indicators and allow an intrabar fill. Though it really should. After all a 1 tick series basically IS tick replay.

However to round out this line of thinking and put it to rest, I was under the impression that bid/ask events might occur and move the price up and down without trades necessarily occurring. In effect the Bid drops a tick and sits but no trades so it drops another tick and then a trade occurs. That type of stuff. Vice-versa for Ask. It was these current best Bid/Ask offers that I was hoping to capture at specific moments in time regardless of whether a trade occurred. Is this not the case? Does price ONLY move on trades? Or is it simply that yes price does fluctuate like that but historical data does not capture it.


Best,
Mark

[NinjaTrader_BrandonH]

Hello Mark,

Thanks for your notes.

Correct, Historical data does not capture the bid/ask events outside of the stamped last event.

The bid/ask events available in Historical data are only available from the last events and this is what Tick Replay uses to simulate OnMarketData().