More backtesting problems

[lookOutBelow]

This is another head scratcher.

If I backtest one of my strategies from 7/1/2010 to 11/12/2010, then backtest that same strategy from 8/1/2010 to 11/12/2010 on a 1-minute chart, I get different trades firing off.

I didn't really notice this until I saw on 11/10/2010, I had a large trade gain if I run the backtest on 7/1 to 11/12. But that trade is missing when I do 8/1 to 11/12.

There is nothing in my strategy that looks at anything very far back in time. I'm using the exact same settings.


Why would a simple change of date cause one backtest to skip some trades?

[lookOutBelow]

I should note that this isn't the only trade that is different. There are quite a few of them.

[NinjaTrader_RyanM1]

Hello lookoutbelow,

Yes, you will get different trades if the date range is different. This is expected and can be for a few different reasons.

You will want to use plenty of Print() statements to verify values are what you expect.

Add TraceOrders = true to your Initialize method and you can then view output related to strategy submitted orders through Tools > Output window.

It may also help to add drawing objects to your chart for signal confirmation. Additional help for these items is available at the following links:

Debugging your NinjaScript code.
TraceOrders
Drawing Objects.

[lookOutBelow]

Buy why would it be different? That doesn't make sense to me.

If I'm backtesting from 7/1 to 11/14 and then 8/1 to 11/14. Why would data be different enough in one or the other to skip a trade on 11/10 when it has plenty of run up time for the 11/10 trade on either backtest date?

I can understand if the trade times in a particular day started differently.. i.e. the indicators haven't had enough data to work from for that day.

I've even gone so far to test it between 9/16 and 9/17 as the start dates (both ending on 11/14). The differences are night and day.



I have draw objects to show me when conditions have been met. And traceorders won't help when a trade shows in one backtest and not another. The obvious difference is that the data is different (or the indicators are looking at that data differently). But I just don't get why it would be.

[NinjaTrader_RyanM1]

Minimum bars required may factor in here. You can have a different sequence of trades that leads to different positions. Comparing two different date ranges is two different tests and we don't expect the trades to be the same.

Best is to evaluate behavior for one given date range. You can evaluate this with print statements and trace orders output. If you find something that doesn't work as you expect, please let us know.

[lookOutBelow]

Best is to evaluate behavior for one given date range. You can evaluate this with print statements and trace orders output. If you find something that doesn't work as you expect, please let us know.

Well, yes I am finding things that doesn't work as expected. That was the intent of this thread.


So let me ask you this..

When I'm trying to evaluate a strategy. Should I go back 6 months to see what happened in the month of October? Or do I need to go back 12 months? Or can I simply go back to September?

Do you see why this is an issue? How in the world are we supposed to get any sort of feel for how a strategy is performing if we get different results like this? Optimizing is pretty much impossible if a simple date change of 1 day can change things so drastically. Again, I would understand if I were backtesting 11/10 to 11/10 then switched to 11/1 to 11/14 and get different results for 11/10. But this isn't the case.

Minimum bars is not a factor. It has remained the same for each of these backtests.

Minimum bars required may factor in here. You can have a different sequence of trades that leads to different positions. Comparing two different date ranges is two different tests and we don't expect the trades to be the same.

Best is to evaluate behavior for one given date range. You can evaluate this with print statements and trace orders output. If you find something that doesn't work as you expect, please let us know.

[NinjaTrader_RyanM1]

We do not expect a backtest with different dates to result in the same trades, even if there is some overlap in the date range between the tests.

Minimum bars required can factor, even if they're the same. If set to anything > 0, then you're ignoring bars on the 8/1 test that you wouldn't on the 7/1 test.

Even without minimum bars factoring, here's a basic scenario where the trades can't be in the same sequence:

Date range 7/1 to 11/1
Enters a long position on 7/31. Current position is long rolling into 8/1. This trade never has a chance to enter for the test started on 8/1.

[lookOutBelow]

Okay. So I'm using 0 bars. And none of my trades are allowed to carry over out of a session. They all enter/exit between 6:30am (pst) and 12:30pm (pst). My session template is 24/7 (although changing that doesn't fix that problem).

So what then would be a cause?

[NinjaTrader_RyanM1]

I suggest you use TraceOrders output and print statements to test strategy behavior with one date range at a time. You'll then have a better understanding of what you're strategy is doing and can identify the specific reason why your results are different.

[lookOutBelow]

I suggest you use TraceOrders output and print statements to test strategy behavior with one date range at a time. You'll then have a better understanding of what you're strategy is doing and can identify the specific reason why your results are different.

I feel like Bill Murray in Ground Hog day...

As a trade is either happening or not, I cannot evaluate with TraceOrders. I've compared every value I'm using to enter the trade. And they are identical from one bar to another that the trade appears or does not (depending on backtest dates).

[NinjaTrader_RyanM1]

We don't expect backtest results to match if you're using a different date range across two tests. Different Test = Different Results.

You should use TraceOrders and Print statements because these are your tools for identifying strategy behavior. Print the values of your indicators. If you're using min bars required = 0, then they might not have enough bars to calculate fully and you're looking at incomplete values.

Since we expect there to be differences between the two tests you're running, any analysis on correct behavior needs to be done one strategy run at a time.

[lookOutBelow]

We don't expect backtest results to match if you're using a different date range across two tests. Different Test = Different Results.

You should use TraceOrders and Print statements because these are your tools for identifying strategy behavior. Print the values of your indicators. If you're using min bars required = 0, then they might not have enough bars to calculate fully and you're looking at incomplete values.

Since we expect there to be differences between the two tests you're running, any analysis on correct behavior needs to be done one strategy run at a time.

I understand why to use Traces and Prints. But in this instance, there doesn't seem to be an answer in them. There are no errors in the TraceOrder. And the values on the same 1min time bar and the two indicators are identical when comparing the instance where the trade happens and where it doesn't. The only changing factor is the date range of the backtest. Never are my backtests start date ranges near the offending trade problem.

If I switch the session template to something else, it either lessens the effect or not. But no matter which one I use, it still happens with some combination of switching date ranges.

I hear what you're saying. Changing the date range changes things. But the indicator values and the bar values are the same from one to another. Minimum # bars has no affect either way (my sessions are either 24/7 or starting hours before I start trading. (My strategy won't start until 6:30am (pst) no matter what session I'm using.)

[NinjaTrader_RyanM1]

Focus on one strategy run at a time. We can look at issues where code doesn't behave as expected, but only within one given strategy run.

The only way to verify if the code is working as you expect is with prints and TraceOrders output.

If you feel one of these tests indicates an error or bug, then the issue will stand alone. If one of these tests is an error: Identify the factors for reproducing it, strip down as simple a strategy as possible, and provide the steps needed to see it.

[mart331]

Because I also have major problems with backtest results, my thoughts:

1 - If I never go overnight with trades, every day starts blank.
2- All the data/indicator values after (lets say) a month, MUST then be the same.
3 - strategy has no number of trades logic, nor loss or profit, so any trades before today DO NOT affect todays trades.
4 THAN it CANT be that trades are different a mont after the starting date if you change start date 1 day....... or am I wrong?

[NinjaTrader_RyanM1]

Hello mart331,

That's a good checklist of things discussed so far. Ultimately a different date range = a different test and we don't expect things to be the same.

If you run into strategy behavior that you can't follow or would like us to shed light on, please break it down into a simple strategy and let us know the steps needed to see what you're seeing.