Backtesting with intrabar granularity

[poseidon_sthlm]

Hi!
I try to achieve an intrabar fill for my strategy by adding a daily BarsObject to a weekly chart. (I use Yahoo for data feed). It works well for entries if you use market orders, i.e. "EnterLong(int barsInProgressIndex, int quantity, string signalName)".

But if I use a limit order, i.e.
"EnterLongLimit(int barsInProgressIndex, bool liveUntilCancelled, int quantity, double limitPrice, string signalName) "
it seams as if I have to set the liveUntilCancelled parameter to True, to make it work. Otherwise the Weekly bar series (BarsInProgress =0) cancells the orders that are submitted to the daily bar series. Is this expected?

My seccond question is, how can I achieve an intrabar fill for the exit method SetStopLoss() ?

My trace file:

2010-01-08 15:34:11 CancelAllOrders: BarsInProgress=0
2008-01-31 00:00:00 Entered internal PlaceOrder() method at 2008-01-31 00:00:00: Action=SellShort OrderType=Limit Quantity=1 LimitPrice=36,98 StopPrice=0 SignalName='Short 1a' FromEntrySignal=''
2008-02-01 00:00:00 Cancelled expired order: BarsInProgress=0: Order='NT-00000/Sim101' Name='Short 1a' State=Working Instrument='A' Action=SellShort Limit price=36,977697345507 Stop price=0 Quantity=1 Strategy='TestingIntrabarBacktest' Type=Limit Tif=Gtc Oco='' Filled=0 Fill price=0 Token='b7dcd4f2d0944a58a338ad130f78302f' Gtd='2099-12-01 00:00:00'
2008-06-26 00:00:00 Entered internal PlaceOrder() method at 2008-06-26 00:00:00: Action=Buy OrderType=Limit Quantity=1 LimitPrice=33,37 StopPrice=0 SignalName='Long 1a' FromEntrySignal=''
2008-06-27 00:00:00 Cancelled expired order: BarsInProgress=0: Order='NT-00001/Sim101' Name='Long 1a' State=Working Instrument='A' Action=Buy Limit price=33,3674306480076 Stop price=0 Quantity=1 Strategy='TestingIntrabarBacktest' Type=Limit Tif=Gtc Oco='' Filled=0 Fill price=0 Token='ba1b2c1ecc4b45aea67e813f81d55266' Gtd='2099-12-01 00:00:00'
2008-09-18 00:00:00 Entered internal PlaceOrder() method at 2008-09-18 00:00:00: Action=SellShort OrderType=Limit Quantity=1 LimitPrice=35,46 StopPrice=0 SignalName='Short 1a' FromEntrySignal=''
2008-09-19 00:00:00 Cancelled expired order: BarsInProgress=0: Order='NT-00002/Sim101' Name='Short 1a' State=Working Instrument='A' Action=SellShort Limit price=35,4570762732273 Stop price=0 Quantity=1 Strategy='TestingIntrabarBacktest' Type=Limit Tif=Gtc Oco='' Filled=0 Fill price=0 Token='68f885381e2c46298dda28b968d40e7e' Gtd='2099-12-01 00:00:00'
2009-09-10 00:00:00 Entered internal PlaceOrder() method at 2009-09-10 00:00:00: Action=Buy OrderType=Limit Quantity=1 LimitPrice=21,40 StopPrice=0 SignalName='Long 1a' FromEntrySignal=''
2009-09-11 00:00:00 Cancelled expired order: BarsInProgress=0: Order='NT-00003/Sim101' Name='Long 1a' State=Working Instrument='A' Action=Buy Limit price=21,4026151299527 Stop price=0 Quantity=1 Strategy='TestingIntrabarBacktest' Type=Limit Tif=Gtc Oco='' Filled=0 Fill price=0 Token='dc2b742608b04bf098a59da560e6a95f' Gtd='2099-12-01 00:00:00'

/Andreas

[NinjaTrader_Bertrand]

Andreas, expected the next OnBarUpdate() call would make those orders expire, thus you have to resubmit or use the liveUntilCancellled overload.

The SetStopLoss would fill submit / intrabar on the incoming execution.

http://www.ninjatrader-support.com/HelpGuideV6/SetStopLoss.html

[poseidon_sthlm]

Thank you for your quick reply.

I forgot to mention in my last post that I'v set the parameter "Simultated" = True in the SetStopLoss() method. Is there a way to make a simulated SetStopLoss order fill intrabar in a backtest?

[NinjaTrader_Bertrand]

Should give you the same behavior, as this feature would only be of use in realtime trading 'hiding' the stop locally until triggered.

[poseidon_sthlm]

My resaon to use the simulated SetStoLoss() is that I want to adjust the StopLoss order to the trend dynamically within the OnBarUpdate() method, that is programing my own trail stop. By using a simulated order I want to avoid extra comisson for order modification that i.e. Interactive Brokers charge. Is the simulated SetStoLoss() the right method to use to avoid comisson for order modification in this case?

In your reply to my first post you wrote that I have to resubmit when I use the "EnterLongStopLimit(int barsInProgressIndex, bool liveUntilCancelled, int quantity, double limitPrice, double stopPrice, string signalName) " method. Would this be the correct way to do this?

 protected override void OnBarUpdate()
        {
            if (BarsInProgress == 0)
            {
.....
                   if (Position.MarketPosition == MarketPosition.Flat)
                        
                    {
                        // Submit entry order to BarsInProgress = 0
                        EnterLongLimit(0, false, Orderquantity, Enter, "Long");
                                                
                        //Submit entry order to BarsInProgress = 1
                        EnterLongLimit(1, false, Orderquantity, Enter, "Long");
                           
                    }
               
..........
            }         
            else
            {
                return;
            }
        }

[NinjaTrader_Ben]

Hello,

I'm kind of jumping in here, but did you try this link?:

http://www.ninjatrader-support2.com/vb/showthread.php?t=10344&highlight=food+thought

Lots of good trailing stop code!

[poseidon_sthlm]

Thank you for your suggestion, NinjaTrader_Ben. Intresting code, but I realize now that I should use the ExitLongStop() and ExitShortStop() instead of a simulated SetStopLoss() to achive what I intended. Sorry for the confusion.

But I still would like to know what the best way is to resubmit entry- and exitorders when you backtest with intrabar granualrity. Does the code in post #5 look correct?

[NinjaTrader_Ben]

Hello,

I will have someone reply to you on Monday. Thank you for your patience.

[NinjaTrader_Bertrand]

Just work alongside this reference sample then below -

http://www.ninjatrader-support2.com/vb/showthread.php?t=6652

As your conditions for adjusting stops trigger (i.e. last close being higher than entryPrice by x tics), just resubmit the exit order to the finer BarsInProgress to simulate an intrabar fill.

[poseidon_sthlm]

Stratgies with limit orders and backtesting with Intrabar granualrity

Hi! I really want to backtest my strategies with intrabar granularity, but I can't make it work. After several hours of testing I suspect that limit orders and stop orders are causing the problem. Limit orders seam to get cancelled and never fill when backtesting with intrabar granualrity. I have also tested the liberal fill algorithm and that is not the problem. (If you only use market orders in a strategy, as in your reference example, it sems to work fine. But my strategies need limit orders.)

I'v done my tests on a chart with a weekly bar series as the primary BarsObject and added a daily bar series as the seccondary BarsObject. To describe the problem I used a simple moving average cross over strategy (based on your round2ticksize reference example). I'v attached a diagram with the strategy running on a weekly chart without intrabar granularity as reference point. At least two entries are expected.

How can I use limit ordes when backtesting with intrabar granularity? Is it correct to submit the limitorder to both barseries as in the code below?


The code for the strategy with intrabar granularity that doesn't fill:

/// This namespace holds all strategies and is required. Do not change it.
namespace NinjaTrader.Strategy
{
    /// <summary>
    /// Simple moving average cross over strategy.
    /// </summary>
    [Description("A simple strategy to demonstrate the use of limit orders in backtesting with granualrity.")]
    public class TestIntrabarBacktestWithLimitorder : Strategy
    {
        #region Variables
        private int        fast        = 10;    // This variable represents the period of the fast moving average.
        private int        slow        = 25;    // This variable represents the period of the slow moving average.
        private double    eMAprice    = 0;    // This variable represents the unrounded value for entries.
        #endregion

        /// <summary>
        /// This method is used to configure the strategy and is called once before any strategy method is called.
        /// </summary>
        protected override void Initialize()
        {
            // Note: The primary bar series is a weekly bar series. 
            // We add a secondary daily bar series. 
            Add(PeriodType.Day, 1);
            
            TimeInForce = Cbi.TimeInForce.Gtc;
            TraceOrders = true;
            BarsRequired = slow;
            
            // Entry handling
            EntriesPerDirection = 1;
            
            // Plots for visual reference.
            EMA(Fast).Plots[0].Pen.Color = Color.OrangeRed;
            EMA(Slow).Plots[0].Pen.Color = Color.Black;

            Add(EMA(Fast));
            Add(EMA(Slow));

            CalculateOnBarClose    = true;
        }

        /// <summary>
        /// Called on each bar update event (incoming tick).
        /// </summary>
        protected override void OnBarUpdate()            
        {
            if (BarsInProgress == 0)
            {
                // First, calculate the fast EMA value.
                eMAprice = Instrument.MasterInstrument.Round2TickSize(EMA(Fast)[0]);
                
                Print(ToDay(Time[0]) +" |  BarsInProgress=0"+ " |  eMAprice " + Instrument.MasterInstrument.Round2TickSize(eMAprice));
                
                // The position is flat, enter a position if necessary.
                if (Position.MarketPosition == MarketPosition.Flat)
                {                                    
                    // Entry conditions for limitorders.
                    if (EMA(Fast)[0] > EMA(Slow)[0])
                    {                            
                        // Sumbit limit entry order for 1 contract to daily barseries (BarsInProgress = 1)
                        EnterLongLimit([COLOR=Blue]1[/COLOR], false, 1,eMAprice, "Long");
                        
                        // Resumbit limit entry order for 1 contract to weekly barseries (BarsInProgress = 0)
                        EnterLongLimit([COLOR=Blue]0[/COLOR], false, 1,eMAprice, "Long");
                    }
                    else if (EMA(Fast)[0] < EMA(Slow)[0])
                    {                        
                        // Sumbit limit entry order for 1 contract to daily barseries (BarsInProgress = 1)  
                        EnterShortLimit([COLOR=Blue]1[/COLOR], false, 1,eMAprice, "Short");
                        
                        // Resumbit limit entry order for 1 contract to weekly barseries (BarsInProgress = 0)
                        EnterShortLimit([COLOR=Blue]0[/COLOR], false, 1,eMAprice, "Short");
                    }
                }
                
                // Simple crossover exit conditions.
                if (CrossAbove(EMA(Fast), EMA(Slow), 1))
                {
                    // Sumbit market order order for 1 contract to daily barseries (BarsInProgress = 1)
                    ExitShort(1, 1, "Exit Short", "Short");
                }
                else if (CrossBelow(EMA(Fast), EMA(Slow), 1))
                {
                    // Sumbit market order order for 1 contract to daily barseries (BarsInProgress = 1)
                    ExitLong(1, 1, "Exit Long", "Long");
                }
            }
            else
            {
                Print(ToDay(Time[0]) +" |  BarsInProgress=1"+ " |  eMAprice " + Instrument.MasterInstrument.Round2TickSize(eMAprice));
                return;
            }
        }

        #region Properties
        /// <summary>
        /// </summary>
        [Description("Period for fast EMA")]
        [Category("Parameters")]
        public int Fast
        {
            get { return fast; }
            set { fast = Math.Max(1, value); }
        }

        /// <summary>
        /// </summary>
        [Description("Period for slow EMA")]
        [Category("Parameters")]
        public int Slow
        {
            get { return slow; }
            set { slow = Math.Max(1, value); }
        }
        #endregion
    }
}

I have attached the the traceoutput in the next post.As you can see in the trace code from the output window, the entry orders always are cancelled and never fill. Why is this?

TO BE CONTINUED IN THE NEXT POST....

[poseidon_sthlm]

Continued from previous post: Stratgies with limit orders.....

As you can see in the trace code from the output window, the entry orders always are cancelled and never fill. Why is this?

20090227 |  BarsInProgress=0 |  eMAprice 37,16
2009-02-27 00:00:00 Entered internal PlaceOrder() method at 2009-02-27 00:00:00: Action=SellShort OrderType=Limit Quantity=1 LimitPrice=37,16 StopPrice=0 SignalName='Short' FromEntrySignal=''
2009-02-27 00:00:00 Entered internal PlaceOrder() method at 2009-02-27 00:00:00: Action=SellShort OrderType=Limit Quantity=1 LimitPrice=37,16 StopPrice=0 SignalName='Short' FromEntrySignal=''
2009-02-27 00:00:00 Ignore order amendment: Action=SellShort OrderType=Limit Quantity=1 LimitPrice=37,16 StopPrice=0 SignalName=Short' FromEntrySignal='' Reason='Order already has this stop price/limit price/quantity'
20090227 |  BarsInProgress=1 |  eMAprice 37,16
2009-02-27 00:00:00 Cancelled expired order: BarsInProgress=1: Order='NT-00000/Sim101' Name='Short' State=Working Instrument='AZN' Action=SellShort Limit price=37,16 Stop price=0 Quantity=1 Strategy='TestIntrabarBacktestWithLimitorder' Type=Limit Tif=Gtc Oco='' Filled=0 Fill price=0 Token='d352507b7ea848d7ad7a10bd20f75b09' Gtd='2099-12-01 00:00:00'
20090302 |  BarsInProgress=1 |  eMAprice 37,16
20090303 |  BarsInProgress=1 |  eMAprice 37,16
20090304 |  BarsInProgress=1 |  eMAprice 37,16
20090305 |  BarsInProgress=1 |  eMAprice 37,16
20090306 |  BarsInProgress=0 |  eMAprice 35,98
2009-03-05 00:00:00 Entered internal PlaceOrder() method at 2009-03-05 00:00:00: Action=SellShort OrderType=Limit Quantity=1 LimitPrice=35,98 StopPrice=0 SignalName='Short' FromEntrySignal=''
2009-03-06 00:00:00 Entered internal PlaceOrder() method at 2009-03-06 00:00:00: Action=SellShort OrderType=Limit Quantity=1 LimitPrice=35,98 StopPrice=0 SignalName='Short' FromEntrySignal=''
2009-03-06 00:00:00 Ignore order amendment: Action=SellShort OrderType=Limit Quantity=1 LimitPrice=35,98 StopPrice=0 SignalName=Short' FromEntrySignal='' Reason='Order already has this stop price/limit price/quantity'
20090306 |  BarsInProgress=1 |  eMAprice 35,98
2009-03-06 00:00:00 Cancelled expired order: BarsInProgress=1: Order='NT-00001/Sim101' Name='Short' State=Working Instrument='AZN' Action=SellShort Limit price=35,98 Stop price=0 Quantity=1 Strategy='TestIntrabarBacktestWithLimitorder' Type=Limit Tif=Gtc Oco='' Filled=0 Fill price=0 Token='613e21a9a1964cdb8af5066cae78e356' Gtd='2099-12-01 00:00:00'
20090309 |  BarsInProgress=1 |  eMAprice 35,98
20090310 |  BarsInProgress=1 |  eMAprice 35,98
20090311 |  BarsInProgress=1 |  eMAprice 35,98
20090312 |  BarsInProgress=1 |  eMAprice 35,98
20090313 |  BarsInProgress=0 |  eMAprice 35,43
2009-03-12 00:00:00 Entered internal PlaceOrder() method at 2009-03-12 00:00:00: Action=SellShort OrderType=Limit Quantity=1 LimitPrice=35,43 StopPrice=0 SignalName='Short' FromEntrySignal=''
2009-03-13 00:00:00 Entered internal PlaceOrder() method at 2009-03-13 00:00:00: Action=SellShort OrderType=Limit Quantity=1 LimitPrice=35,43 StopPrice=0 SignalName='Short' FromEntrySignal=''
2009-03-13 00:00:00 Ignore order amendment: Action=SellShort OrderType=Limit Quantity=1 LimitPrice=35,43 StopPrice=0 SignalName=Short' FromEntrySignal='' Reason='Order already has this stop price/limit price/quantity'
20090313 |  BarsInProgress=1 |  eMAprice 35,43
2009-03-13 00:00:00 Cancelled expired order: BarsInProgress=1: Order='NT-00002/Sim101' Name='Short' State=Working Instrument='AZN' Action=SellShort Limit price=35,43 Stop price=0 Quantity=1 Strategy='TestIntrabarBacktestWithLimitorder' Type=Limit Tif=Gtc Oco='' Filled=0 Fill price=0 Token='e0e7e0490d3d4c6fb279e08c510bf912' Gtd='2099-12-01 00:00:00'
20090316 |  BarsInProgress=1 |  eMAprice 35,43
20090317 |  BarsInProgress=1 |  eMAprice 35,43
20090318 |  BarsInProgress=1 |  eMAprice 35,43
20090319 |  BarsInProgress=1 |  eMAprice 35,43
20090320 |  BarsInProgress=0 |  eMAprice 34,97
2009-03-19 00:00:00 Entered internal PlaceOrder() method at 2009-03-19 00:00:00: Action=SellShort OrderType=Limit Quantity=1 LimitPrice=34,97 StopPrice=0 SignalName='Short' FromEntrySignal=''
2009-03-20 00:00:00 Entered internal PlaceOrder() method at 2009-03-20 00:00:00: Action=SellShort OrderType=Limit Quantity=1 LimitPrice=34,97 StopPrice=0 SignalName='Short' FromEntrySignal=''
2009-03-20 00:00:00 Ignore order amendment: Action=SellShort OrderType=Limit Quantity=1 LimitPrice=34,97 StopPrice=0 SignalName=Short' FromEntrySignal='' Reason='Order already has this stop price/limit price/quantity'
20090320 |  BarsInProgress=1 |  eMAprice 34,97
2009-03-20 00:00:00 Cancelled expired order: BarsInProgress=1: Order='NT-00003/Sim101' Name='Short' State=Working Instrument='AZN' Action=SellShort Limit price=34,97 Stop price=0 Quantity=1 Strategy='TestIntrabarBacktestWithLimitorder' Type=Limit Tif=Gtc Oco='' Filled=0 Fill price=0 Token='93c69f476c6044eb9752f16af18699b2' Gtd='2099-12-01 00:00:00'
20090323 |  BarsInProgress=1 |  eMAprice 34,97
20090324 |  BarsInProgress=1 |  eMAprice 34,97
20090325 |  BarsInProgress=1 |  eMAprice 34,97
20090326 |  BarsInProgress=1 |  eMAprice 34,97
20090327 |  BarsInProgress=0 |  eMAprice 34,5
2009-03-26 00:00:00 Entered internal PlaceOrder() method at 2009-03-26 00:00:00: Action=SellShort OrderType=Limit Quantity=1 LimitPrice=34,50 StopPrice=0 SignalName='Short' FromEntrySignal=''
2009-03-27 00:00:00 Entered internal PlaceOrder() method at 2009-03-27 00:00:00: Action=SellShort OrderType=Limit Quantity=1 LimitPrice=34,50 StopPrice=0 SignalName='Short' FromEntrySignal=''
2009-03-27 00:00:00 Ignore order amendment: Action=SellShort OrderType=Limit Quantity=1 LimitPrice=34,50 StopPrice=0 SignalName=Short' FromEntrySignal='' Reason='Order already has this stop price/limit price/quantity'
20090327 |  BarsInProgress=1 |  eMAprice 34,5
2009-03-27 00:00:00 Cancelled expired order: BarsInProgress=1: Order='NT-00004/Sim101' Name='Short' State=Working Instrument='AZN' Action=SellShort Limit price=34,5 Stop price=0 Quantity=1 Strategy='TestIntrabarBacktestWithLimitorder' Type=Limit Tif=Gtc Oco='' Filled=0 Fill price=0 Token='b7ca6df861bc4c649bb19494ac7f5fe0' Gtd='2099-12-01 00:00:00'

[NinjaTrader_Bertrand]

Andreas, as you do not set them to liveUntilCancelled they expire on the next OnBarUpdate() call which is expected behavior, so either use this option or resubmit until you're filled.

[poseidon_sthlm]

Thank you for your quick reply. I really appreciate your customer support.

Since I'm rather new to NT I thought that I would avoid the advanced option "Live until cancelled". Don't want to risk missing to cancel an order. But I see that it seams to work as expected.

But if I would choose the other alternative "resubmit until you're filled", how would I do this? What code is missing in the strategy that I posted below?

/Andreas

[NinjaTrader_Bertrand]

The basic idea is to check for your entry still being valid, but your MarketPosition does not reflect the trade, thus you would need to resubmit the order on the next bar until you're filled.

I'm not sure why you try submitting entry orders to both bars objects...also if you target to enter long at the long EMA for example, this would indeed never fill on biggest portion of the screenshot as the price is never touching it.

Is your strategy working as you would expect if you reduce it to one timeframe only? Then you could proceed from there to streamline your coding efforts.

[hawks67]

This is all terrific information, but I seem (as usual) to have an issue that can't be fully resolved by the examples herein.

I've studied the code in SampleIntrabarBacktest, primarily because the description of that example says "Like in live trading, taking entry conditions from a 5min chart means executing your order as soon as possible instead of waiting until the next 5min bar starts building. ", but it doesn't quite point me in the right direction.

I have a Range(6) chart as primary, and a 377T chart as secondary. The majority of the time, a range bar will close before the secondary bar. [Perhaps the biggest question here is, does that make the 377T the primary?] Any OnBarUpdate calculation (referencing values on the secondary timeframe) made at the close of the range bar can only refer back to the last closed secondary timeframe bar, not the one in-progress at the time the range bar closed. This is causing me to miss out on more than a few trades, and is preventing me from applying tight filters.

Is it possible to "hand the ball off" to the secondary (377T) timeframe, in the form of some sort of flag that says, in effect, "the range chart says enter a trade if your current 377T bar closes within specified parameters"?

Thank you in advance,
Gary