backtesting discprenency with forward testing

[jonmoron]

Hi all,

I am having some difficulty using NT's built in backtester. Here's some background. I have a fairly sophisticated position-management based NT7 strategy that scales in and scales out on multiple legs (separate IOrders, unmanaged=true). The strategy works perfectly fine during forward testing and executes all orders/traders perfectly as to the rules. However every time I run the backtester, over whatever period (1 day, 1 week, 1 month, etc), the backtester takes a lot of time (which I understand) and comes back with exactly 2 trades. One entry and One exit for the FIRST leg. The entry is the first trade that is executed on the first day (if it is triggered) and the last trade is the exit at the end of the period (end of day exit)).

Again during forward testing the strategy works perfectly fine, but when I run the backtest (the standard NT7 backtest), I get 2 trades.

I realized a few things while going through this process:

1. CalculateOnBarClose has to be set to true (although the strategy is a tick-based strategy)
2. I need to comment out the Historical check (obviously)

Are there any other tidbits of information I need to implement specifically for backtesting (I'm hoping its something simple I'm overlooking). Also, are the above two realizations true realizations?


thanks!

[sev888]

I'm having the same problem... Any of the NT staff have any ideas on how to resolve this?

[NinjaTrader_Bertrand]

The 2 statements are correct, as the intrabar formation is not known in backtesting, CalculateOnBarClose if defaulted to 'true'.

Do you get any log errors as the behavior occurs in the SA?

Are you debugging already with TraceOrders = true?

For scaling in and out - which Entry Handling are you using?

[jonmoron]

Bertrand,

Thank you for the response.

1. What do you mean that the CalculateOnBarClose is defaulted to true? Do I not need to manually set COBC to true for backtesting purposes?

2. I should comment out the lines below for ANY backtest runs

    if(Historical)
       return; 

3. I have TraceOrders = true in the Initialize function of the strategy. What effect does this have on the backtest?

4. This is my Entry Handling options:

EntriesPerDirection = 44;
EntryHandling = EntryHandling.UniqueEntries;
Unmanaged = true; 

However it should be noted that the 44 lots are broken down in this order:
15,15,5,5,4 lot orders. So there are 5 orders (With the quantities detailed earlier) that sum up to 44.


Should I be changing any of these settings?

[NinjaTrader_Bertrand]

jonmoron, there's only one CalculateOnBarClose setting in backtesting (true). Setting it to false would not have an influence, as backtesting uses OHLCV info and the intrabar formation is not known.

Correct, for backtesting, you would need to take the historical lock out, otherwise the OnBarUpdate() returns out on running through historical data.

TraceOrders are helpful in debugging, you would need open a output window to see the results.

Have you named your scale ins and outs all uniquely? Then you should be ok with EntriesPerDirection to 1 and unique handling on.

[jonmoron]

jonmoron, there's only one CalculateOnBarClose setting in backtesting (true). Setting it to false would not have an influence, as backtesting uses OHLCV info and the intrabar formation is not known.

Correct, for backtesting, you would need to take the historical lock out, otherwise the OnBarUpdate() returns out on running through historical data.

TraceOrders are helpful in debugging, you would need open a output window to see the results.

Have you named your scale ins and outs all uniquely? Then you should be ok with EntriesPerDirection to 1 and unique handling on.

Bertrand,

Thanks for the info. At any one time I could have upto 5 orders (with a total quantity of 44) entered and working. Each of the 5 orders is uniquely named order1, order2, order3, order4, or order5. Keep in mind that this strategy works PERFECTLY fine when run in any forward test.

I suspect there is a problem with my EntryHandling setings for backtesting.

EntriesPerDirection = 44;
EntryHandling = EntryHandling.UniqueEntries;
Unmanaged = true; 

Can backtests be run on 'Unmanaged = true' strategies?

[NinjaTrader_JoshP]

jonmoron,

You can backtest unmanaged strategies. What will not work with unmanaged though are those specific EntryHandling options. Unmanaged means you have to keep track of everything. All those handling that existed with managed orders are not applicable for unmanaged orders.

[jonmoron]

jonmoron,

You can backtest unmanaged strategies. What will not work with unmanaged though are those specific EntryHandling options. Unmanaged means you have to keep track of everything. All those handling that existed with managed orders are not applicable for unmanaged orders.

Josh, Ok that answers my question. So then what is wrong with my strategy? How come backtesting results in 2 trades over a 1 week period and forward-testing results in tens, if not hundreds?

[NinjaTrader_JoshP]

jonmoron,

When you are forward testing you can have lots of different signals versus backtesting. Backtesting is done on a CalculateOnBarClose = true only stand point while forward testing, this could be set to false.

Other things to consider is turning on TraceOrders = true to see what is happening under the hood with your orders. What I suggest is find a trade you have in real-time and do the same date in backtest and print the values of your conditions and things like that so you can analyze what is happening differently.

[jonmoron]

jonmoron,

When you are forward testing you can have lots of different signals versus backtesting. Backtesting is done on a CalculateOnBarClose = true only stand point while forward testing, this could be set to false.

Other things to consider is turning on TraceOrders = true to see what is happening under the hood with your orders. What I suggest is find a trade you have in real-time and do the same date in backtest and print the values of your conditions and things like that so you can analyze what is happening differently.

Ok. I'll do that. But if I set the bar to 1-tick on the backtest, wouldn't it simulate a tick-based backtest run (essentially looking at every tick)?

[NinjaTrader_JoshP]

jonmoron,

If you set it to 1-tick that would be a 1-tick backtest. That is not what you would want to do. Instead you would want to run your regular backtest with an additional bars object through a multi-time frame strategy. The multi-time frame strategy would then allow you to have added granularity to let you get better prices.

[jonmoron]

jonmoron,

If you set it to 1-tick that would be a 1-tick backtest. That is not what you would want to do. Instead you would want to run your regular backtest with an additional bars object through a multi-time frame strategy. The multi-time frame strategy would then allow you to have added granularity to let you get better prices.

Right, I intend to run the backtest on a large time frame (say 1 month in 2009) but on a 1-tick data series.

[jonmoron]

I think I might have found the problem. I have this line of code that checks for the current time:

if(DateTime.Now > DateTime.Parse(endTime))
                ExitAll("End of TradeDate"); 

As soon as this code is called the strategy turns off for the day. However, the problem is that I want to dynamically set the endTime. Do you think if this line is called during the backtest, the NT engine uses the current-computer time and not the bar's time? If so, how can I access the bar's time to compare against the "endTime" variable I set?

thanks!

[NinjaTrader_JoshP]

DateTime.Now is your PC system clock's time. To get bar's time you need to use Time[0].

[jonmoron]

DateTime.Now is your PC system clock's time. To get bar's time you need to use Time[0].

Thanks! I've made that change to my code. However now the strategy doesn't run at all. I get 0 trades or executions. Also, nothing is written to the output window (before it did).

I also simply swapped out the DateTime.Now check with the Time[0] check...and it never works with Time[0] and always (wrongly) works with DateTime.Now.

Any clues?