Achieving accurate backtesting results on short time frames

[GoldStandard]

I'm seeking some advice from those experienced with backtesting on shorter time frames with Ninjatrader.

I'm hiring a programming company (haven't picked one yet, so if anyone has recommendations or would like to bid on the job themselves please let me know.) to put the indicators I use into a strategy as trade triggers and filters in order to backtest some of the setups I use.

I plan to use it to improve my discretionary trading by testing my setups and indicators, to alert me to opportunities, to test different stop loss and take profit strategies, and perhaps eventually to do some automated trading.

In preparing the specifications for the programmers, the part I could use some help with is regarding overcoming some of the known issues with achieving accurate results in backtesting short term trades where a few ticks difference in entry and exit can make a big difference in the results.

Below is the portion of the specs regarding backtesting and overall design quality I've put together so far. Is there anything important I've left out? I would be grateful for any comments and suggestions from those with experience. What other technical criteria do you think are important?

Strategy design requirements checklist:

Item #1 Checking bid/ask records to get accurate fill prices for entries and exits. The strategy needs to include a script that checks a file containing bid/ask information that is not normally included in backtesting, and supplies the correct fill prices according to the best bid and ask prices available at the moment the trade was placed. For more information see this thread: http://www.ninjatrader-support2.com/...ad.php?t=24167

Item #2 Order placement should occur on a separate time frame that can be adjusted by the user in the parameters. 1 tick and 1 range will be the most commonly used choices for short term trading. See the following links for more information: http://www.ninjatrader-support2.com/...ead.php?t=6652

http://www.ninjatrader-support2.com/vb/showthread.php?p=108743

http://www.ninjatrader-support2.com/vb/showthread.php?p=136515

http://www.ninjatrader-support2.com/vb/showthread.php?t=5787

example framework here: http://www.ninjatrader-support2.com/...catid=2&id=183

Item #3 - be aware of this known issue with calling indicators from strategies http://www.ninjatrader-support2.com/...ad.php?t=18653

Item #4 cobc lines need to be removed from any individual indicators and moved to the strategy itself. See http://www.ninjatrader-support2.com/...d.php?p=140458 for more information.

Item #5 be aware of ninja 7 compatibility requirements: http://www.ninjatrader-support2.com/...ad.php?t=21016

Item #6 all text indicator parameters need to be changed into numbers so they can be optimized


Item #1 looks the trickiest. I would be interested in hearing from anyone who has pulled it off and would be willing to offer me some code samples or consulting on the subject.

If you have experience with using strategies and would like to collaborate or help test this one, PM me. It will have some interesting features, including a set of rules to define and take trades off of various upport and resistance lines and volume profile areas,a simple rule set for using multiple indicators or support and resistance lines to confirm or filter trade setups, and some volume delta, level 2, and order flow indicators.

[NinjaTrader_Ben]

Hello,

If you have a question I am happy to help.

[GoldStandard]

Thanks Ben! Mainly I'm just looking for confirmation that accurate backtesting is possible through the methods I'm describing above, and to see if those who have gone through the process already have any advice to offer. I have read about people like Maxima who are working on it, but I haven't actually heard if anyone has succeeded or not.

I guess thats not surprising though; if they were successful at achieving accurate backtesting on short time frames, they are probably busy running their backtests and improving their strategies

[NinjaTrader_Bertrand]

GoldStandard, backtesting is what it is, meaning it's not meant to give you an exact sequence of events that will happen again with the same 'precision' - it just gives you an idea if your edge is worth your time monitoring it realtime going forward, or better first execuing it against Market Replay data.

The first point is indeed hard to accomplish in backtesting, with NT7 you could execute to a added bid and ask series then if your data provider offers the needed historical data. Couldn't you just include a reasonable slippage thus ensuring that your edge persists after spread costs?

[GoldStandard]

Mainly I want accurate backtesting so that I can rely on it to test things more quickly than I can live. If I can't rely on backtesting, and I still have to test everything on market replay, then backtesting, and especially optimization, is not nearly as useful as it could be.

I'm happy to report that a programmer has contacted me that says he can accomplish the backtesting using bid/ask data, so I'm going to give it a try.

I'm a little mystified as to why Ninja isn't doing this work themselves. It doesn't seem as hard as a lot of what you guys have already pulled off, and it would be an incredibly attractive feature. Right now, all the experienced Ninja auto traders I have heard from say you can't rely on Ninja backtesting, but it seems they work with Ninja anyway because it is so extensible. Accurate backtesting would probably bring the serious auto traders on board as customers in droves.

I'm not complaining, though. Ninja is a great application, your business model and resultant developer community is a great idea, and c# has a lot of programmers available for it, so it is a practical platform for me.

[NinjaTrader_Bertrand]

Thanks for the feedback and great you got a programmer located - NT7 will add more features to work with bid / ask data directly and has historical storing for it as well.

[GoldStandard]

Thanks Bertrand, I'm really glad you guys added stored bid/ask data for NT7. Great feature.

I'm still hoping some experienced strategy developers will drop by this thread and leave some comments about their own experiences with backtesting accuracy. Anyone?

[naxo79]

Thanks Bertrand, I'm really glad you guys added stored bid/ask data for NT7. Great feature.

I'm still hoping some experienced strategy developers will drop by this thread and leave some comments about their own experiences with backtesting accuracy. Anyone?

Looking at the same thing.

Did you finally pull it off?

My experience adding different timeseries to same chart/strategy with NT is that I usually ran into some other bugs making the whole process unreliable.

[NinjaTrader_Bertrand]

Can you please clarify what you wish to do? NT7 does allow access to Bid / Ask series as well for executing backtest trades to those for more accurate results - http://www.ninjatrader.com/support/h...s/nt7/add3.htm

[naxo79]

I would basically be trying to accomplish what the OP here tried:

http://www.ninjatrader.com/support/forum/showthread.php?t=40832

[NinjaTrader_Bertrand]

Yes that would be possible with NT7 - his main issue was likely working with the Set's tied to BIP0 while he needed to employ the Exit methods allowing him to specify the traded to BIP as well.

[naxo79]

Do I really have to specify the instrument name in the syntax if I want to add Bid?

this works:
Add(PeriodType.Minute, 1);

this does not:
Add(PeriodType.Minute, 1, MarketDataType.Bid);

[NinjaTrader_Bertrand]

Yes, for the adding a specific market data type please use the extended offered overload including the instrument as well.

[naxo79]

Ok tnx.

I intend to run backtests on a list of symbols.
How can I automatically insert the symbolname into the overload?

[NinjaTrader_Bertrand]

We would unfortunately not support dynamically adding of instruments, so best would be setting up a user string input and then entered your symbol as needed.

Some users have found success using Instrument.FullName here as input for the overload -

http://www.ninjatrader.com/support/helpGuides/nt7/instrument_fullname.htm