What are the problems with backtesting?

[jeffbg123]

Hey,

I am relatively new to Ninjatrader and am using it for backtesting some strategies.

Besides for the obvious fact that curve fitting is evil and the past is not the future, what are the problems that can come out of backtesting?

In other words, if you run a backtest (with default filling) and it shows profit, then what are the reasons that it would not work in a live trading environment?

This is assuming a strategy is run with all limit orders off the market, so slippage should be negligible. I currently have a backtest for a winning strategy but it shows avg time in market = 0.0......how is this possible?

Thanks
-Jeff

[NinjaTrader_Jason]

Hello Jeff,

Please see the link below for information regarding discrepancies between backtesting and running the strategy real-time.

http://www.ninjatrader-support.com/HelpGuideV6/DiscrepanciesReal-TimeVsBacktest.html

I am not sure why 'Average time in market' is reported as zero. If you backtest the SampleMACrossOver strategy on the same historical data, does the issue persist?

Could it be the strategy is in a position for only a few seconds?

Please see the following link for the calculation of the value.

http://www.ninjatrader-support.com/HelpGuideV6/AverageTimeInMarket.html

Could it be all orders were submitted on the same date?

[jeffbg123]

I am running a strategy that goes for has a profit target of 2 ticks and a stop loss of 11 ticks. There are no other exits. So there are certainly times that it is only in the market for a few seconds, but the results are extremely unrealistic.

It says that I have 1000+ trades and 1 loser (which I know isn't true)

I basically want to know what I did wrong in my code to make it seem like it is the holy grail when it is not.

It says that default fill is when the price is penetrated.....Does that mean that the whole price needs to trade (meaning that you will definately get filled) or that only a certain number needs to be traded? AKA is this the worst case scenario where as liberal is the best case scenario?

What do you mean by on the same date? There are about 15 trades per day in the backtest. The chart looks reasonable...

Thanks
-Jeff

[NinjaTrader_Jason]

Unfortunately I am not sure what needs to be changed in your code. I suggest to debug your strategy so it functions as you intend. General debug information can be found at the link below.

http://www.ninjatrader-support2.com/vb/showthread.php?t=3418

The two system fill algorithms are:

Default
An algorithm that takes a conservative and more realistic approach to filling limit and stop limit orders.

• Limit orders only fill if the limit price was penetrated
  
• Limit orders are always filled at the limit price specified never better (for example, if a limit order is submitted on bar n, NT will check if the order is filled on bar n+1, if this bar gaps down and the limit order was a buy, the order would be filled at the limit price and NOT the high of bar n+1)

Liberal
An algorithm that takes a liberal approach to filling limit and stop limit orders.

• Limit orders fill if the limit price was touched
  
• On gap down bars, buy limit orders will fill at the high of the gap down bar
  
• On gap up bars, sell limit orders will fill at the low of the gap up bar

Using the Default algorithm, it will fill the limit order when the market penetrades (traded through) the order price.

Please disregard my last inquiry as per the date.

[jeffbg123]

One problem that I see is that in the log I am getting one of those "Enter() is ignored" messages. I have read through the suggested help topic.

However I do not understand where the problem is. My logic is very simple. I am checking if I am flat, and entering on conditions. I will never enter if I am not flat and I never enter more than once on a single bar.

Where else could this be coming from?

Thanks
-Jeff

[NinjaTrader_Jason]

Unfortunately I am not sure. What is the outcome if you backtest the strategy on a different instrument or interval?

If you check the individual trades under the Trades-tab, did they occur as you intend?

If the strategy does not work as you like, please debug your strategy as per my previous post.

[jeffbg123]

In the trades section it says that my entry and exit time for all of the trades is the same and that they are all 2 tick winners (except for the 1 loser). They are the same time to the second.....

Any idea why this is? I am just using profit target and stop loss....

[NinjaTrader_Jason]

This does clarify the zero 'Average time in market'.

Average Time In Market
SUM(exit date – entry date) of all trades / # of trades

Since the exit and entry are exact the same, the equasion above results in zero.

I am not sure why your strategy sent orders this way. You will need to debug your strategy to check why it does not function as you like.

In addition, you could contact any of the supported NinjaScript consultants at the link below as well.

http://www.ninjatrader.com/webnew/partners_onlinetrading_NinjaScript.htm

[jeffbg123]

Isn't this a ninjatrader bug? How could I possibly get 2 ticks profit 1000/1001 times that takes under 1 second? Wouldnt this be something to do with the software, not my code?

[NinjaTrader_Jason]

Can you please backtest the SampleMACrossOver strategy and check if it tests as expected.

What do you see for the second the orders executed in the chart? Right click the trade in the Trades-tab and select 'Chart'

Did you try other instruments and intervals?

[jeffbg123]

I tried to run the sample MA crossover and it seems to work fine.

What do you mean by this: What do you see for the second the orders executed in the chart? Right click the trade in the Trades-tab and select 'Chart'

I did what you said but I am not sure what "second the orders executed" means.

I am running this all on a 3500 tick chart. I have not tried other intervals or instruments because I do not have data for other instruments and only wanted to test on a tick chart

Thanks
-Jeff

[NinjaTrader_Jason]

The order executed on a certain second (as in time). I suggested to check if the price candle that holds this second contains both the entry and exit price of your order.

[jeffbg123]

In my strategy all of them are executed on the same candle because they take 0 seconds. I am running it on 3500 bars and every trade entry occurs on the same bar as it's exit.....Basically it is exiting me at prices that I would not be able to get and I am not sure why it is giving me those prices.....

How can I fix this?

[NinjaTrader_Jason]

As mentioned, you will need to debug your strategy and check why it does not function as you like. Tips can be found at the link below.

http://www.ninjatrader-support2.com/vb/showthread.php?t=3418

Unfortunately I am not sure why your strategy behaves this way.

[Baruch]

Hi Jeff,
You need to run it on a smaler chart, so your exits would be not on the same bar. You can get your trigers from the current chart.

Baruch