More backtesting problems

[mart331]

Help!

Hello mart331,

That's a good checklist of things discussed so far. Ultimately a different date range = a different test and we don't expect things to be the same.

If you run into strategy behavior that you can't follow or would like us to shed light on, please break it down into a simple strategy and let us know the steps needed to see what you're seeing.

Hello Ryan,

I uploaded an example in the other thread that shows what is (very) wrong. Please check.

And sorry, but I have the feeling that you dont take this serious.

The point is: I try to help you with what is wrong. Too many paople have such big differences.
There must be SOMETHING wrong.
Results backtest/replay are not even far from equal.

QUOTE
1 - If I never go overnight with trades, every day starts blank.
2- All the data/indicator values after (lets say) a month, MUST then be the same.
3 - strategy has no number of trades logic, nor loss or profit, so any trades before today DO NOT affect todays trades.
4 THAN it CANT be that trades are different a mont after the starting date if you change start date 1 day....... or am I wrong?
END QUOTE

It's too easy to say that you cant expect them to be the same. When your startdate is 1 day later, and 1 month later you get VERY different trades (taken into account the above), then something is wrong. Period.
All bardata in that month are the same, so all indicator values should be the same too. (5 -15 minute bars).

NinjaTrader is very usefull because of strategy/indicator programming capacities. But if backtest versus replay gives so different results, what is the value than?

Sorry, but I invested very much time on optimizing, and it seems all useless.
Really, I have read threads, i am an experienced programmer in C++, and read the manual.

[NinjaTrader_RyanM1]

This thread was created because the original poster felt that backtesting two date ranges with overlapping periods should lead to the same trades for the overlapping periods.

We provided several reasons why this should not be expected.

Please share a simple strategy and details of two different tests if you would like these reasons to be expanded or clarified.

[lookOutBelow]

Since I get the feeling that no one believes this is a problem, I've stripped the issue down to its basics and made a very simple script that illustrates what is going on - on one particular moment in time.

If there is a simple explanation or something that I'm overlooking. I'm all ears.


1) Run this on "TF"
2) Use the settings in the attached picture.
3) Once you see the output in the output window, change the date from 9/7 start to 9/8 start.
4) Compare the values.

You'll see that the values that enter a trade are identical. But on 9/7 start date, the trade doesn't trigger, but on 9/8 start date, it does. Nothing seems to have changed other than the date range.


9/7 date results
Low: 723.2 SMA: 723.2
exampleProblem is running.
K[0]: 62.5000000000022 > D[0]: 62.1593915343914
K[1]: 60.2564102564112 < D[1]: 63.8204275704273
exampleProblem is running.


9/8 date results
Low: 723.2 SMA: 723.2
exampleProblem is running.
K[0]: 62.5000000000022 > D[0]: 62.1593915343914
K[1]: 60.2564102564112 < D[1]: 63.8204275704273
Trade::: Low: 723.2 SMA: 723.2
K[0]: 62.5000000000022 > D[0]: 62.1593915343914
K[1]: 60.2564102564112 < D[1]: 63.8204275704273



Identical results. But on one date, the trade triggers, on the other it does not.

//





#region Using declarations
using System;
using System.ComponentModel;
using System.Diagnostics;
using System.Drawing;
using System.Drawing.Drawing2D;
using System.Xml.Serialization;
using NinjaTrader.Cbi;
using NinjaTrader.Data;
using NinjaTrader.Indicator;
using NinjaTrader.Gui.Chart;
using NinjaTrader.Strategy;
#endregion

// This namespace holds all strategies and is required. Do not change it.
namespace NinjaTrader.Strategy
{
    /// <summary>
    /// 
    /// </summary>
    [Description("No name for this strategy yet.")]
    public class exampleProblem : Strategy
    {
        #region Variables
        
        private int     drawDown = 1000;

        private int     contracts = 1; 
        private int        contractMultiplier = 1;
        private int        contractCeiling = 3;
        private double    atRisk = 300;
        

        private bool     ProfitLossForDayReached = false;
        private double     PandL = 0;
        private bool     OneBarSinceExit = true;
        private int     _tradeTimeBegin = 630; // 
        private int     _tradeTimeEnd = 1250; // 
        private bool     addIndicators = true; // 
        
        private double     stopPrice = 0;
        private double     entryPrice = 1;
        private    double    targetPrice = 0;
        private double    shortRecoverEntryPrice = 0;

        private double     quickDouble = 0;
        private int     quickInt = 0;
        
        private int        tempOpt = 47;
        private double    tempOptDouble = 0;
        
        
        
        #endregion

        /// <summary>
        /// 
        /// </summary>
        protected override void Initialize()
        {    
            if (addIndicators == true)
            {
                Add(Stochastics(12, 20, 3));
                Add(SMA(14));
            }

            CalculateOnBarClose = true;
            TraceOrders = false;
            Unmanaged = false;
            
            Print ("exampleProblem is running.");    
        }

        /// <summary>
        /// Called on each bar update event (incoming tick)
        /// </summary>
        protected override void OnBarUpdate()
        {
                
            
                if (Position.MarketPosition == MarketPosition.Flat)
                    LookForLongEntry();
                        
            
        }
        


        // ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
        private void LookForLongEntry()
        {
            
            
            if (High[0] == 723.7 && Low[0] == 723.2)        
            {
                Print ("");
                Print ("+++++++++++++++++");
                Print ("Low:  " + Low[0] + "    SMA:  " + SMA(14)[0]);
                
                Print ("K[0]:  " + Stochastics(12, 20, 3).K[0] + "  >  D[0]:  " + Stochastics(12, 20, 3).D[0]);
                Print ("K[1]:   " + Stochastics(12, 20, 3).K[1] + "  <  D[1]:  " + Stochastics(12, 20, 3).D[1]);
                
                
                if (
                    Low[0] < SMA(14)[0] 
                    && Stochastics(12, 20, 3).K[0] > Stochastics(12, 20, 3).D[0] 
                    && Stochastics(12, 20, 3).K[1] < Stochastics(12, 20, 3).D[1]        
                    )
                {        
                
                    SetStopLoss("Long", CalculationMode.Ticks, 2, false);
                    SetProfitTarget("Long", CalculationMode.Ticks, 2);
                    EnterLong(1, "Long");    
                    DrawArrowUp("InitialStop" + CurrentBar, true, 0, Low[0], Color.Pink);
                    
                    Print ("Trade:::  Low:  " + Low[0] + "     SMA:  " + SMA(14)[0]);
                    Print ("K[0]:  " + Stochastics(12, 20, 3).K[0] + "  >  D[0]:  " + Stochastics(12, 20, 3).D[0]);
                    Print ("K[1]:  " + Stochastics(12, 20, 3).K[1] + "  <  D[1]:  " + Stochastics(12, 20, 3).D[1]);
                }
            }
        }

            


                                
        
        

        #region Properties
        
        
        #endregion
    }
}

[NinjaTrader_RyanM1]

This strategy works the same on my end with either date range. There's one trade on 11/ 10 in both tests. Sorry, I'm not able to provide any clues.

Since you're comparing double values, are you perhaps running into issues with floating point arithmetic:
Floating-Point Arithmetic

[lookOutBelow]

This strategy works the same on my end with either date range. There's one trade on 11/ 10 in both tests. Sorry, I'm not able to provide any clues.

Since you're comparing double values, are you perhaps running into issues with floating point arithmetic:
Floating-Point Arithmetic

Maybe that is the problem in this case. Thanks for suggesting that. Reading the floating point link made me change my code slightly.


I replaced:

Low[0] < SMA(14)[0] 
&& Stochastics(12, 20, 3).K[0] > Stochastics(12, 20, 3).D[0] 
&& Stochastics(12, 20, 3).K[1] < Stochastics(12, 20, 3).D[1]

With:

Low[0] <= SMA(14)[0] 
&& CrossAbove(Stochastics(12, 20, 3).K, Stochastics(12, 20, 3).D, 1)

And both trades showed up here too. The <= was the difference maker. But, put in the CrossAbove just to be certain with that too.

I put that in my real strategy code and it seems to have fixed that particular trade problem. Not sure at this point if the others are fixed too (there were quite a few that were off). At least it is a start and something to work off of.

I followed the floating point link and will investigate more into it. I had no idea that this could be a potential problem.



Thanks

[NinjaTrader_RyanM1]

Thanks for looking into this as a potential cause for what you're seeing. Glad you're able to find some consistency.