Incorrect Fill Price even when using tickreplay and having bid/ask.

[DaveE]

Hello rspine,

I dont see anyone answering your question in post#3

1) Can you point me or explain exactly how the fill price is calculated. With formulas and snippets etc.

Here is what I deduced about market order fills after some testing of the fill engine (IsHistorical):
Assuming you have OrderFillResolution.Standard and Slippage=0:
a) If you are in the context of a Ask data series then a market buy order will get filled at GetOpen(CurrentBar+1). This is a correct simulation of what would likely have happened in Realtime, ie the broker would probably have filled at the opening Ask price of the bar-being-built which is when OnBarUpdate gets executed.

b) If you are in the context of a Bid data series then a market sell order will get filled at GetOpen(CurrentBar+1). This is a correct simulation of what would likely have happened in Realtime, ie the broker would probably have filled at the opening Bid price of the bar-being-built which is when OnBarUpdate gets executed.

c) However if you are in the context of a Ask data series then a market sell order gets filled at the same price as a buy. This is obviously inaccurate and I suppose that the idea of the Slippage property was to set this to some 'average' spread and then you might get a slightly more realistic sell order.
Similar problem applies to executing market buy on a Bid series.

So for instance optimizing SampleMACrossOver with default Slippage=0 will give an illusion of higher profits (or lower losses) that you would have got Realtime, simply because the Bid Ask spread has not been taken into account.
With the fill engine as at RC2: The only way I can see to get accurate inclusion of the historical Bid Ask spread (which might vary wildly) into backtests, is by ensuring that all Buy orders get executed on a Ask data series and all Sell orders get executed on a Bid data series. But then your code is going to look a lot more complicated than SampleMACrossOver! Has anyone got a better idea?

[Trader212310]

Thanks for the response Ray.

You said

Signals are generated on the close of a bar thus the next tick (which is the open of the next bar) is where the order is filled at

The price of the next tick you mentioned can be extremely different then what the price was at the close of the bar that generated the signal, especially with thinly traded symbols.

Why not have it fill at whatever the price is at the the nanosecond after the bar closes? Or better yet just at whatever the (ideally Bid or Ask) price was at the moment the signal happened?

You've built such a great platform, I fail to understand why you undermine it with inaccurate backtesting
Ninjatrader is superior to many of the $5k-$10k a month platforms I've tested in my career, but its fatal flaw is backtesting/optimization accuracy, which is such a shame because it seems like such a simple thing for you to fix.

[NinjaTrader_Ray]

Hi NT8,

You NEED to add a setting to your Strategy Analyzer that will let it:

1. Fill orders at whatever the price was at the moment the order signal was generated. Not at the opening price of the following bar or following tick.

2. Same as item 1, but must fill on Bid/Ask data if the user has Bid/Ask data. Especially Bid/Ask data that happened in between Last ticks and on Last ticks (if the user has this data).

Thank you for your feedback.

1. Only market orders can be filled immediately and they are filled at the time the order is generated. Signals are generated on the close of a bar thus the next tick (which is the open of the next bar) is where the order is filled at. Limit and stop orders are "working" on the next bar and evaluated against this bars data to determine if it indeed has been filled and at what price.

2. Thanks for the suggestion.

[Trader212310]

Hi NT8,

You NEED to add a setting to your Strategy Analyzer that will let it:

1. Fill orders at whatever the price was at the moment the order signal was generated. Not at the opening price of the following bar or following tick.

2. Same as item 1, but must fill on Bid/Ask data if the user has Bid/Ask data. Especially Bid/Ask data that happened in between Last ticks and on Last ticks (if the user has this data).

3. Your Strategy Analyzer's Chart needs to have the same features as your regular chart (such as buttons in the GUI that can be used to display BID/ASK data, multiple instruments and so on.)

PERIOD!

I'm sorry to be harsh, but not having these features is just downright irresponsible and reckless. I can't image how much money your users have lost due to the misleading backtests results the lack of these features has caused.

I've been in the industry a while and am actually in the process of writing a letter to the CFTC suggesting they require all commercial trading/backtesting software have the above mentioned features 1 and 2 to prevent retail traders from misunderstanding backtest results.

SFT-132 does seem like a good idea.

[NinjaTrader_Brett]

Great feedback, it's appreciated.

I edited the SFT to be clear on possible solutions you point out. I've had requests for either solution you point out over time and it is not clear yet which one we would implement based on demand. Right now our focus is getting NT8 into the wild so it will be some time before this is reviewed.

-Brett

[Matheyas5]

Great Brett.

Regarding your quote below, can you please verify that you are talking about Bid prices and Ask prices that existed both on and/or in between Last Ticks, rather then just the Bid/Ask prices that existed On Last Ticks? I highly suggest the former.

As to the item of wanting the fill engine to also use bid/ask prices to fill when in Tick Replay mode then I will add your vote to our suggestions and feedback tracking.

Also, I highly suggest adding feature request SFT-1324 (shown here http://ninjatrader.com/support/forum...280#post456280) which would probably solve all the problems brought up in this thread (expect for the ability to customize/control the fill engine).

I truly feel Ninjatrader 8's and 7's main weakness is how complicated it is to run an accurate backtest/optimization in the Strategy Analyzer. I think Ninjatrader is a far more superior platform then Multicharts, but Multicharts does make it much easier to run a accurate backtest/optimization.
If any other users reading this agree, please vote for SFT-1324 as well.

P.S.
As somewhat mentioned in the SFT-1324 link above, Ninjatrader's support forums seem to be filled with an enormous amount of threads (such as this thread) regarding trying to understand/fix/workaround accuracy issues within the Strategy Analyzer. I think adding SFT-1324 will highly cut down on the time Ninjatrader's support staff (and its users) have to spend on tech support questions.

[NinjaTrader_Brett]

Thanks for letting me know.

We have increased our documentation on how the fill engine works, will be pushed with our next release.

As to the item of wanting the fill engine to also use bid/ask prices to fill when in Tick Replay mode then I will add your vote to our suggestions and feedback tracking. SFT-982

[Matheyas5]

Hi NinjaTrader_Brett,

I'd just like to say I completely agree with everything rspine has said in this thread.

[NinjaTrader_Brett]

us8lander,

NT7 is completely different then NT8 with fill logic. Please post a thread in the NT7 forums so we don't confuse the two. We will help you understand NT7 logic there.

[NinjaTrader_Brett]

ggggg,

Thanks for the response.

1) No way to manually import in the bid/ask attached properties to the last. Nor can you view them in the Tools > Historical Data.

As to what providers provide the data I will have this page update to reflect it: http://ninjatrader.com/support/helpG...y_provider.htm

Primarily its Kinetick, IQ Feed, and any provider utilizing the NinjaTrader historical data server such as CQG, Continuum, Rithmic.

2) The use is bid/ask trade analysis for indicators and strategies. Was never intended for use during backtests.

3) I will look into improving the documentation in this area. As I agree we could add some more detail to the doc then what we had in NT7 since we no longer have it 'open source'. Which I will discuss that decision internally as well.

[ggggg]

Brett,

Thank you very much for your answer.

Accessing historical bid/ask via the OnMarketData() event

What the document does not describe is where the bid/ask data in tick replay comes from. As shown by our experiment, importing historical bid, ask and ticks does not generate the "Last" data with bid/ask at the time of the trades mentioned by you. This is quite counterintuitive.

It is not clear how one can create the desired "Last" with correct bid and ask data. In the historical data viewer, there are no bid and ask columns associated with "Last" so I am not quite sure what you mean by

So if you are manually creating the data you would need to only create the last and populate the bid/ask fields on the last data.

I am wondering if there a way to import such data from a single CSV file having additional columns for bid and ask or from separate CSV files with trades, bids and asks.

Historical Fills Processing

Since the fill simulation algorithm does not use the bid/ask data the usefulness of importing it is quite limited in the first place. I think the fact that bid/ask data is not used for calculating fills is worth mentioning explicitly in the documentation both in the section on Tick Replay and the one on Historical Fills Processing.

Speaking of the historical fills, the very detailed understanding of how it works is absolutely necessary for any serious user since it is a key assumption of a backtest. In my view, the documentation should make it extremely clear how limit and market orders are simulated given the data at hand and the backtest settings. The required level of detail can be demonstrated by the following examples

• Limit orders: if "fill on touch" is false, the standing buy (sell) limit order is filled when the current bar's low is strictly lower (current's bar high is strictly higher) than the order price. If "fill on touch" is true, the rules are similar with the strict equalities ("strictly lower" or "strictly higher") replaced with their non-strict versions ("lower or equal" and "higher or equal")".
• Market orders: once a market order is submitted it is filled at the end of the next bar. The fill price equals ... plus the slippage parameter.

The current documentation on fill resolution setting is insufficient:

Selecting order fill resolution of "High" will allow you to set a secondary bar series to be used as the price data to fill your orders, this allows you to bring in more granular data then you are currently running the strategy on. For example you may have a strategy that you run on "Daily" bars but then want to bring in "Minute" bars for the historical fill algorithm to be based on.

It is not exactly clear what happens when one uses, say, daily data for signals and minute data for fill simulations given that daily data lacks intraday timestamps. If one submits an order in OnBarUpdate() what is the time it will be considered submitted?

Another unclear point how historical fills processing works when one uses tick data granularity (since tick data is different from OHLC bars). Instead of high, low, open and close we have only one price so personally I am not quite sure what happens in this case.

Thank you.

[us8lander]

NT7 Insight

I am dealing with this same issue, but am still on NT7. Can someone point me to the details on Fillprice algo/approach that SImulator uses.... I am getting fills that don't make sense based on bid, ask and last tick data received through OnMarketData event processing ...
Thanks
us8lander

[NinjaTrader_Brett]

ggggg,

http://ninjatrader.com/support/helpGuides/nt8/en-us/developing_for__tick_replay.htm

Please see the resource above for some guidelines to follow. Let me know if you felt that didn't go into enough details and ask questions and I will update the document.

For you question though there is actually physically Last (With Bid/Ask), Ask, and Bid physical series on the disk and saved. Tick replay plays back only the Last series which has its bid/ask stamped at the time the trade occurred. So if you are manually creating the data you would need to only create the last and populate the bid/ask fields on the last data.

[ggggg]

Hi Brett,

Unfortunately, we cannot make OnMarkedData() work the way you have described. A minimal working example of strategy working with historical bid/ask would be extremely helpful.

We created the following artificial data

Bid:
20151211 060101;99;1
20151211 060104;100;1
20151211 060106;101;1
Ask:
20151211 060101;101;1
20151211 060104;102;1
20151211 060107;103;1
Trades (Last):
20151211 060102;100;1
20151211 060103;100;1
20151211 060105;101;1

and created a strategy with the following OnMarketData() handler.

protected override void OnMarketData(MarketDataEventArgs marketDataUpdate)
{
	Print(string.Format("{0} Bid={1} Ask={2} Price={3} Type={4}",
	  marketDataUpdate.Time,  marketDataUpdate.Bid, marketDataUpdate.Ask,
	  marketDataUpdate.Price, marketDataUpdate.MarketDataType)
	);
}

If we do not subscribe to any additional streams besides the primary one specified in the GUI the output is

11.12.2015 07:01:02 Bid=100 Ask=100.01 Price=100 Type=Last
11.12.2015 07:01:03 Bid=100 Ask=100.01 Price=100 Type=Last
11.12.2015 07:01:05 Bid=101 Ask=101.01 Price=101 Type=Last

If, in addition, we subscribe to the bid ask streams in OnStateChange()

protected override void OnStateChange()
{
	#...
	else if (State == State.Configure)
	{
		AddDataSeries("GOOG", Data.BarsPeriodType.Tick, 1, Data.MarketDataType.Ask);
		AddDataSeries("GOOG", Data.BarsPeriodType.Tick, 1, Data.MarketDataType.Bid);
	}
}

the output becomes

11.12.2015 07:01:01 Bid=101 Ask=101.01 Price=101 Type=Last
11.12.2015 07:01:01 Bid=99 Ask=99.01 Price=99 Type=Last
11.12.2015 07:01:02 Bid=100 Ask=100.01 Price=100 Type=Last
11.12.2015 07:01:03 Bid=100 Ask=100.01 Price=100 Type=Last
11.12.2015 07:01:04 Bid=102 Ask=102.01 Price=102 Type=Last
11.12.2015 07:01:04 Bid=100 Ask=100.01 Price=100 Type=Last
11.12.2015 07:01:05 Bid=101 Ask=101.01 Price=101 Type=Last
11.12.2015 07:01:06 Bid=101 Ask=101.01 Price=101 Type=Last
11.12.2015 07:01:07 Bid=103 Ask=103.01 Price=103 Type=Last

In either case, we cannot access bid/ask data in a meaningful manner. Could you please indicate if there are errors in what we are doing or possibly provide a minimal working example. Thank you

[NinjaTrader_Brett]

1) http://ninjatrader.com/support/helpG...ical_fill_.htm

This page here describes the fill processing. If you have further questions let me know and I will answer them and potentially update the page to have more detail as needed. I would not be able to provide code snippets or formulas however.

2) You are speaking to the Product Manager. I will note your feedback and discuss it internally. Early on in the design phase of NT8 we decided to not expose fill algorithms as NinjaScript since in NT7 we found low to no usage of the exposed fill algorithms and we felt we could innovate there with a proprietary fill engine which satisfied the use cases and just 'worked'. Even with NT7 we did not use bid/ask data for the fill of orders. This is an enhancement request which comes with the penalty of requiring having the bid/ask data (Most people don't have and even if you did have there is due diligence to do to know is the bid/ask data filtered) and comes at the cost of triple the cost in time/memory/etc since there is more data to process. With the cost and benefit analysis and the scope of clients who actually have access to the data we have decided not to implement this at this time.

3) Thanks for explaining, this is actually expected. Since Tick Replay is only designed to work with last. Instead you always monitor 'Last' events and utilize the "MarketDataUpdate.Ask" and "MarketDataUpdate.Bid" properties to get the Ask/Bid price when the last trade occurred. We store the ask/bid with the actual last data as it occurs so we only have to process a single series.

If your saying that we should also throw OnMarketData events and support tick replay for bid/ask dataseries then I would record this feedback so we can look into in in a future major release.