Multi Instrument Backtest Problem

[SoulSurfer]

I can't find a reference to this problem using the search facility, but if there is one please feel free to point me in the right direction!

The symptoms are as follows.

1. The strategy uses daily bars, and unmanaged orders with both a stop and limit exit
2. Those exit orders are updated every bar - A sort of custom "advancing target/trailing stop"
3. Backtest results vary depending upon the order in which the instruments are assigned to the various BarsInProgress.

As far as can tell from experimenting with a custom fill type inside the VS debugger, the strategy analyzer checks each open order for each new bar for EVERY instrument, but OnBarUpdate() is only called once per new bar per instrument, as you would expect. This means that the check for a fill is called BEFORE OnBarUpdate() modifies the exits for all the instruments where BarsInProgress > 0.

If my explanation makes sense, has anyone come across this issue before? If so, is there a simple solution?

[NinjaTrader_CalH]

Hello SoulSurfer,

Thank you for your post.

This is all expected behavior with the backtest.

What is occurring is that at the end of the bar, the OnBarUpdate() gets called and processed. Any orders are thus handled right then and processed at the next bar.

At the same time, the Strategy Analyzer is checking your exit orders and seeing if the next bar they would have gotten filled and if so to fill them.

Example, on bar 20 when OnBarUpdate() is called your entry and exit orders will get processed before OnBarUpdate() on bar 21

Let me know if I can be of further assistance.

[SoulSurfer]

Hi Cal,

Hopefully you can be of further assistance

This is all expected behavior with the backtest.

What you describe is not what I'm seeing in a multi-instrument backtest. First assume no new positions need opening today. I'm guessing slightly here, but if I have 4 instruments, and a position in the instrument corresponding to BarsInProgress = 3 then the exit orders are checked 4 times. My assumption is the first 3 occurrences are triggered by new bars on BarsInProgress = 0,1,2. The sequence seems to be:

OnBarUpdate(0)
CheckOrders(3) => "Yesterday's" exits for BarsInProgress = 3 potentially filled
OnBarUpdate(1)
CheckOrders(3)
OnBarUpdate(2)
CheckOrders(3)
OnBarUpdate(3) => Exits for BarsInProgress = 3 updated here,
CheckOrders(3) => but the orders may have already been filled at "stale" prices

Does that make more sense? It's certainly not the behaviour I would expect, and it certainly leads to unexpected backtest results, where changing the order of the instruments changes the prices at which the exits are filled

[NinjaTrader_CalH]

SoulSurfer,

How are you checking the CheckOrders that you are printing out?

Do you have a sample of how you submitting and tracking this?

[SoulSurfer]

How are you checking the CheckOrders that you are printing out?

I started out seeing inconsistent results in NinjaTrader's strategy analyzer output.
I'm currently not printing out anything whilst trying to ascertain the cause. I'm just setting breakpoints in the Visual Studio debugger, inside "CustomFillType.cs". I've no idea how the fill checking code operates, so I've given it the arbitrary name "CheckOrders()", which will at some point call Fill(Order order) and hit my breakpoint;

I take it the behaviour I'm describing isn't familiar to you?

[NinjaTrader_CalH]

SoulSurfer,

A fill type will not affect when it gets called.
Have you tested this with another fill type such as Default Fill Type?

[SoulSurfer]

Have you tested this with another fill type such as Default Fill Type?

Yes - It behaves in exactly the same way. My CustomFillType.cs is just a clone of DefaultFillType where I can add debugging code without fear of messing up the original

[SoulSurfer]

FYI I have stumbled across another thread where it seems like somebody else has experienced the same issue. See: http://www.ninjatrader.com/support/f...ad.php?t=62903

To repeat blunt's question:

I just don't know why OnBarUpdate for BarsInProgress 0 is called before the exits are handled, but OnBarUpdate for BarsInProgress 1 to BarsInProgress N are handled after the exits are processed.

As far as I can tell he never received an adequate answer.

[NinjaTrader_CalH]

SoulSurfer,

This is how the NinjaTrader Core is designed. Each BarsInProgress will have its own run through of the FillType if applicable. Meaning, that if I have multiple data series or BIP in my script, NinjaScript will run through each BIP and process any order or filltype requests at that time before moving on to the next BIP.

For instance -
Running one strategy with 2 additional BarsInProgress for different instruments, So BIP 0, 1 ,2. I then added a OnOrderUpdate() again check the fill and BIP to see when in the run through it is processed.

I ran a quick script that submits orders based on BIP, see the attached sample script, and got the expected results.

I attached a screenshot of the output window which shows this happening.

[SoulSurfer]

Hi Cal,

This is how the NinjaTrader Core is designed. Each BarsInProgress will have its own run through of the FillType if applicable. Meaning, that if I have multiple data series or BIP in my script, NinjaScript will run through each BIP and process any order or filltype requests at that time before moving on to the next BIP.

I have simultaneously been creating an example to demonstrate my (and blunt's) problem. See the attached .ZIP. Here's an extract from the output window:

Strategy bars 0 - new GBPUSD bar: OHLC 1.5584, 1.5622, 1.5499, 1.552 at 05/07/2012 22:00:00
Strategy bars 1 - new GBPAUD bar: OHLC 1.5166, 1.5227, 1.5069, 1.5082 at 05/07/2012 22:00:00
Strategy bars 0 - new GBPUSD bar: OHLC 1.5519, 1.555, 1.546, 1.5486 at 06/07/2012 22:00:00
Strategy bars 1 - new GBPAUD bar: OHLC 1.5081, 1.5201, 1.5078, 1.5164 at 06/07/2012 22:00:00
Fill bars 1 - Open = 1.5169, High = 1.5251, Low = 1.5159, Stop = 1.5151
Strategy bars 0 - new GBPUSD bar: OHLC 1.5471, 1.5534, 1.5469, 1.5522 at 09/07/2012 22:00:00
Fill bars 1 - Open = 1.5202, High = 1.5255, Low = 1.5157, Stop = 1.5151
Strategy bars 1 - new GBPAUD bar: OHLC 1.5169, 1.5251, 1.5159, 1.5202 at 09/07/2012 22:00:00
Fill bars 1 - Open = 1.5202, High = 1.5255, Low = 1.5157, Stop = 1.5151
Strategy bars 0 - new GBPUSD bar: OHLC 1.5522, 1.5548, 1.5477, 1.5514 at 10/07/2012 22:00:00
Fill bars 1 - Open = 1.522, High = 1.5242, Low = 1.5115, Stop = 1.5151
Strategy bars 1 - new GBPAUD bar: OHLC 1.5202, 1.5255, 1.5157, 1.5219 at 10/07/2012 22:00:00
Strategy bars 0 - new GBPUSD bar: OHLC 1.5514, 1.5577, 1.5485, 1.5498 at 11/07/2012 22:00:00
Strategy bars 1 - new GBPAUD bar: OHLC 1.522, 1.5242, 1.5115, 1.5115 at 11/07/2012 22:00:00

This is a modified version of the standard NT sample multi-instrument strategy. You will note that the check for a fill of the trailing stop for BarsInProgress=1 are first processed BEFORE OnBarUpdate() for BarsInProgress=1, and then checked again AFTER the matching OnBarUpdate().

Is this really "how the NinjaTrader Core is designed"? If so, is there any way to detect within the CustomFillType which BarsInProgress actually triggered the fill check?

[NinjaTrader_CalH]

SoulSurfer,

Yes, this is how the Core is designed to work.

I would recommend that you add other print statements outside the just the stop order statement and you can then see where and when the Fill Type is called and how that can translate for you to the strategy Analyzer.

I ran your script with just a print in the Class for FillType at the beginning and got three calls.
One for the Entry, one for an evaluation of the TrailStop, and the last one for the TrailStop fill

[SoulSurfer]

Yes, this is how the Core is designed to work.

If that is indeed how the NT Core logic is designed to work then at the moment I fail to see the rationale, though maybe there is one? What possible benefit is it to evaluate potential fills BEFORE calling OnBarUpdate() for a particular instrument, and then repeating that evaluation for every instrument in the basket?

As I said at the outset, for my use case the strategy analyzer produces different results depending on the order in which the multiple instruments are presented to it, which strikes me as being a major disadvantage!

To restate my earlier question, is there any way for my CustomFillType to determine which instrument triggered the current invocation, since evidently for any particular order Fill(Order order) is invoked for every instrument in the basket?

[NinjaTrader_CalH]

SoulSurfer,

It does not call before the OnBarUpdate() but after the OBU and only when a order is considered in an OrderState of Working.

Output example -

Order='NT-00003/Backtest' Name='Trail stop' State=PendingSubmit Instrument='GOOG' Action=Sell Limit price=0 Stop price=1031.94 Quantity=100 Strategy='test1' Type=Stop Tif=Gtc Oco='NT-00002-452' Filled=0 Fill price=0 Token='8e954e9c43d24b2db6d4561f648a1384' Gtd='12/1/2099 12:00:00 AM'
OOU 1 3/26/2014 2:00:00 PM
Order='NT-00003/Backtest' Name='Trail stop' State=Accepted Instrument='GOOG' Action=Sell Limit price=0 Stop price=1031.94 Quantity=100 Strategy='test1' Type=Stop Tif=Gtc Oco='NT-00002-452' Filled=0 Fill price=0 Token='8e954e9c43d24b2db6d4561f648a1384' Gtd='12/1/2099 12:00:00 AM'
OOU 1 3/26/2014 2:00:00 PM
Order='NT-00003/Backtest' Name='Trail stop' State=Working Instrument='GOOG' Action=Sell Limit price=0 Stop price=1031.94 Quantity=100 Strategy='test1' Type=Stop Tif=Gtc Oco='NT-00002-452' Filled=0 Fill price=0 Token='8e954e9c43d24b2db6d4561f648a1384' Gtd='12/1/2099 12:00:00 AM'
Custom Fill being called 1
OBU 0 3/27/2014 2:00:00 PM
Custom Fill being called 1
OBU 1 3/27/2014 2:00:00 PM
Custom Fill being called 1
OBU 0 3/28/2014 2:00:00 PM
Custom Fill being called 1

Each bar update when a working order is identified will send a request to the FillType, as shown above

It is per Working Order at the end of the OBU that calls the FillType.
So, when OBU is called and is a Working Order it will call the FillType to evaluate the order.

[SoulSurfer]

Cal - I've added your OnOrderUpdate() to my example, and added an extra Print() at the start of my custom Fill(), and changed my terminology slightly to match yours. Here's an extract from the output:

OBU 1 - new GBPAUD bar: OHLC 1.5166, 1.5227, 1.5069, 1.5082 at 05/07/2012 22:00:00
OBU 0 - new GBPUSD bar: OHLC 1.5519, 1.555, 1.546, 1.5486 at 06/07/2012 22:00:00
OBU 1 - new GBPAUD bar: OHLC 1.5081, 1.5201, 1.5078, 1.5164 at 06/07/2012 22:00:00
OOU 1 06/07/2012 22:00:00
Order='NT-00002/Backtest' Name='RSI' State=PendingSubmit Instrument='$GBPAUD' Action=Buy Limit price=1.5164 Stop price=0 Quantity=1 Strategy='ExampleMultiInstrument' Type=Limit Tif=Gtc Oco='' Filled=0 Fill price=0 Token='c5059ebd461a442b9b0235630b9342c0' Gtd='01/12/2099 00:00:00'
OOU 1 06/07/2012 22:00:00
Order='NT-00002/Backtest' Name='RSI' State=Accepted Instrument='$GBPAUD' Action=Buy Limit price=1.5164 Stop price=0 Quantity=1 Strategy='ExampleMultiInstrument' Type=Limit Tif=Gtc Oco='' Filled=0 Fill price=0 Token='c5059ebd461a442b9b0235630b9342c0' Gtd='01/12/2099 00:00:00'
OOU 1 06/07/2012 22:00:00
Order='NT-00002/Backtest' Name='RSI' State=Working Instrument='$GBPAUD' Action=Buy Limit price=1.5164 Stop price=0 Quantity=1 Strategy='ExampleMultiInstrument' Type=Limit Tif=Gtc Oco='' Filled=0 Fill price=0 Token='c5059ebd461a442b9b0235630b9342c0' Gtd='01/12/2099 00:00:00'
Fill bars 1 - Order = NT-00002, Action = Buy, State = Working, Type = Limit
OOU 1 06/07/2012 22:00:00
Order='NT-00003/Backtest' Name='Trail stop' State=PendingSubmit Instrument='$GBPAUD' Action=Sell Limit price=0 Stop price=1.5151 Quantity=1 Strategy='ExampleMultiInstrument' Type=Stop Tif=Gtc Oco='NT-00002-982' Filled=0 Fill price=0 Token='d8237addfe9042cfad4d5035b6d44b12' Gtd='01/12/2099 00:00:00'
OOU 1 06/07/2012 22:00:00
Order='NT-00003/Backtest' Name='Trail stop' State=Accepted Instrument='$GBPAUD' Action=Sell Limit price=0 Stop price=1.5151 Quantity=1 Strategy='ExampleMultiInstrument' Type=Stop Tif=Gtc Oco='NT-00002-982' Filled=0 Fill price=0 Token='d8237addfe9042cfad4d5035b6d44b12' Gtd='01/12/2099 00:00:00'
OOU 1 06/07/2012 22:00:00
Order='NT-00003/Backtest' Name='Trail stop' State=Working Instrument='$GBPAUD' Action=Sell Limit price=0 Stop price=1.5151 Quantity=1 Strategy='ExampleMultiInstrument' Type=Stop Tif=Gtc Oco='NT-00002-982' Filled=0 Fill price=0 Token='d8237addfe9042cfad4d5035b6d44b12' Gtd='01/12/2099 00:00:00'
OOU 1 06/07/2012 22:00:00
Order='NT-00002/Backtest' Name='RSI' State=Filled Instrument='$GBPAUD' Action=Buy Limit price=1.5164 Stop price=0 Quantity=1 Strategy='ExampleMultiInstrument' Type=Limit Tif=Gtc Oco='' Filled=1 Fill price=1.5164 Token='c5059ebd461a442b9b0235630b9342c0' Gtd='01/12/2099 00:00:00'
Fill bars 1 - Order = NT-00003, Action = Sell, State = Working, Type = Stop
Fill bars 1 - Open = 1.5169, High = 1.5251, Low = 1.5159, Stop = 1.5151
OBU 0 - new GBPUSD bar: OHLC 1.5471, 1.5534, 1.5469, 1.5522 at 09/07/2012 22:00:00
Fill bars 1 - Order = NT-00003, Action = Sell, State = Working, Type = Stop
Fill bars 1 - Open = 1.5202, High = 1.5255, Low = 1.5157, Stop = 1.5151
OBU 1 - new GBPAUD bar: OHLC 1.5169, 1.5251, 1.5159, 1.5202 at 09/07/2012 22:00:00
Fill bars 1 - Order = NT-00003, Action = Sell, State = Working, Type = Stop
Fill bars 1 - Open = 1.5202, High = 1.5255, Low = 1.5157, Stop = 1.5151
OBU 0 - new GBPUSD bar: OHLC 1.5522, 1.5548, 1.5477, 1.5514 at 10/07/2012 22:00:00
Fill bars 1 - Order = NT-00003, Action = Sell, State = Working, Type = Stop
Fill bars 1 - Open = 1.522, High = 1.5242, Low = 1.5115, Stop = 1.5151
OOU 1 10/07/2012 22:00:00
Order='NT-00003/Backtest' Name='Trail stop' State=Filled Instrument='$GBPAUD' Action=Sell Limit price=0 Stop price=1.5151 Quantity=1 Strategy='ExampleMultiInstrument' Type=Stop Tif=Gtc Oco='NT-00002-982' Filled=1 Fill price=1.5151 Token='d8237addfe9042cfad4d5035b6d44b12' Gtd='01/12/2099 00:00:00'
OBU 1 - new GBPAUD bar: OHLC 1.5202, 1.5255, 1.5157, 1.5219 at 10/07/2012 22:00:00
OBU 0 - new GBPUSD bar: OHLC 1.5514, 1.5577, 1.5485, 1.5498 at 11/07/2012 22:00:00
OBU 1 - new GBPAUD bar: OHLC 1.522, 1.5242, 1.5115, 1.5115 at 11/07/2012 22:00:00

Note that at line 21 there's an OBU 0 followed by a Fill 1. Then there's an OBU 1 followed by an identical Fill 1. Then there's an OBU 0 => Fill 1 => OOU 1, eventually followed by a OBU 1. Can you see that? If so, can you now understand the point I am endeavouring to make?

[NinjaTrader_CalH]

SoulSurfer,

Please attach the script that you ran this test on so I can further investigate this