Different backtest results on different systems

[rodriguesdanny]

I have a strategy that I have been testing. I was wondering why I would get different results testing the strategy on the same instrument and timeframe but on 3 different computers.

[NinjaTrader_ChelseaB]

Hello rodriguesdanny,

Generally this means the data is different. I've also seen scripts modified on one computer without the script being re-exported transferred to and imported on the other computer.

Writing the information out to file can let us pinpoint exactly what is different.

The support article below provides direction in the section 'Comparing strategy performance between two computers'.

https://support.ninjatrader.com/s/article/Developer-Guide-Comparing-Strategy-Results

An example script is linked where you can copy the code to write to file and modify this to fit your conditions.

I would be happy to assist with analyzing the output files and discovering what is causing differences.

[rodriguesdanny]

I have been racking my brain trying to get this to work with the code I already had. Can you help me out with that part below is my strategy code:

region Using declarations
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.ComponentModel.DataAnnotations;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Windows;
using System.Windows.Input;
using System.Windows.Media;
using System.Xml.Serialization;
using NinjaTrader.Cbi;
using NinjaTrader.Gui;
using NinjaTrader.Gui.Chart;
using NinjaTrader.Gui.SuperDom;
using NinjaTrader.Gui.Tools;
using NinjaTrader.Data;
using NinjaTrader.NinjaScript;
using NinjaTrader.Core.FloatingPoint;
using NinjaTrader.NinjaScript.Indicators;
using NinjaTrader.NinjaScript.DrawingTools;
#endregion

//This namespace holds Strategies in this folder and is required. Do not change it.
namespace NinjaTrader.NinjaScript.Strategies
{
public class KeltnerChannels : Strategy
{
private int SavedBar;

private KeltnerChannel KeltnerChannel1;

protected override void OnStateChange()
{
if (State == State.SetDefaults)
{
Description = @"Enter the description for your new custom Strategy here.";
Name = "KeltnerChannels";
Calculate = Calculate.OnBarClose;
EntriesPerDirection = 1;
EntryHandling = EntryHandling.AllEntries;
IsExitOnSessionCloseStrategy = true;
ExitOnSessionCloseSeconds = 300;
IsFillLimitOnTouch = false;
MaximumBarsLookBack = MaximumBarsLookBack.TwoHundredFiftySix;
OrderFillResolution = OrderFillResolution.Standard;
Slippage = 1;
StartBehavior = StartBehavior.WaitUntilFlat;
TimeInForce = TimeInForce.Gtc;
TraceOrders = false;
RealtimeErrorHandling = RealtimeErrorHandling.StopCancelClose;
StopTargetHandling = StopTargetHandling.PerEntryExecution;
BarsRequiredToTrade = 20;
// Disable this property for performance gains in Strategy Analyzer optimizations
// See the Help Guide for additional information
IsInstantiatedOnEachOptimizationIteration = true;
Stop = 1;
Profit = 1;
Input3 = 1;
SavedBar = 1;
}
else if (State == State.Configure)
{
}
else if (State == State.DataLoaded)
{
KeltnerChannel1 = KeltnerChannel(Close, 1.5, 80);
SetStopLoss("", CalculationMode.Ticks, Stop, false);
SetProfitTarget("", CalculationMode.Ticks, Profit);
}
}

protected override void OnBarUpdate()
{
if (BarsInProgress != 0)
return;

if (CurrentBars[0] < 1)
return;

// Set 1
if (
// Time
((Times[0][0].TimeOfDay < new TimeSpan(16, 0, 0))
|| (Times[0][0].TimeOfDay > new TimeSpan(18, 0, 0)))
// Market Position
&& ((Position.MarketPosition == MarketPosition.Flat)
|| (Position.MarketPosition == MarketPosition.Short))
&& (CurrentBars[0] != SavedBar)
&& (Open[0] < KeltnerChannel1.Upper[0])
&& (Close[0] > KeltnerChannel1.Upper[0]))
{
EnterLong(Convert.ToInt32(DefaultQuantity), "");
SavedBar = Convert.ToInt32(CurrentBars[0]);
}

// Set 2
if (
// Time
((Times[0][0].TimeOfDay < new TimeSpan(16, 0, 0))
|| (Times[0][0].TimeOfDay > new TimeSpan(18, 0, 0)))
// Market Position
&& ((Position.MarketPosition == MarketPosition.Flat)
|| (Position.MarketPosition == MarketPosition.Long))
&& (CurrentBars[0] != SavedBar)
&& (Open[0] > KeltnerChannel1.Lower[0])
&& (Close[0] < KeltnerChannel1.Lower[0]))
{
EnterShort(Convert.ToInt32(DefaultQuantity), "");
SavedBar = Convert.ToInt32(CurrentBars[0]);
}

}

region Properties
[NinjaScriptProperty]
[Range(1, int.MaxValue)]
[Display(Name="Stop", Order=1, GroupName="Parameters")]
public int Stop
{ get; set; }

[NinjaScriptProperty]
[Range(1, int.MaxValue)]
[Display(Name="Profit", Order=2, GroupName="Parameters")]
public int Profit
{ get; set; }

[NinjaScriptProperty]
[Range(1, double.MaxValue)]
[Display(Name="Input3", Order=3, GroupName="Parameters")]
public double Input3
{ get; set; }
#endregion

}
}

[NinjaTrader_ChelseaB]

Hello rodriguesdanny,

Be sure you are looking at the code in the RealtimeReplayHistoricalComparisonsExample_NT8 linked in my previous post.

The output you are writing to file for the conditions should match the conditions.

To get you started, lets take a few conditions from the first condition set and add this to the output message string.

if (
((Times[0][0].TimeOfDay < new TimeSpan(16, 0, 0))
|| (Times[0][0].TimeOfDay > new TimeSpan(18, 0, 0)))
&& ((Position.MarketPosition == MarketPosition.Flat)

message = string.Format("{0:dd/MM/yyyy H:mm:ss.ffff} |(Times[0][0].TimeOfDay: {1} < 160000 || Times[0][0].TimeOfDay: {1} > 180000) && (Position.MarketPosition: {2} == Flat)", {0}, Times[0][0].TimeOfDay, Position.MarketPosition);

Try adding the rest of the values in this conditions set to the output message string.

[rodriguesdanny]

I imported the recommended file and tried to run a test but I am not getting any output.

[NinjaTrader_ChelseaB]

Hello rodriguesdanny,

If this was enabled for historical may I confirm the Backtest property is checked and enabled?

Also, if this was enabled for real-time may I confirm Backtest is unchecked and the strategy was actually enabled?

May I have a screenshot of the Documents\NinjaTrader 8\ folder to confirm the text files were not written?

[rodriguesdanny]

It was for historical, I tried with and without Backtest being selected

[NinjaTrader_ChelseaB]

Hello rodriguesdanny,

Below is a link to a video of testing the script.

https://drive.google.com/file/d/1nGMnPgOOypjO9YPaqbICc-LytwyAfoRA/view?usp=drivesdk

If you perform the exact steps shown in the video is the behavior different?

[rodriguesdanny]

I had to completely remove the file, restart and reload it but got it to work now.

I assume I run the test on the same instrument and timeframe that I was having the issue with?

[NinjaTrader_ChelseaB]

Hello rodriguesdanny,

This example would just be a demonstration of how to use a C# StreamWriter to write information data to file that you can compare between systems.

You will want to adapt this to your script's conditions in your script. But also write out the price information, condition information, and execution information.

Be sure to watch the Comparing Real-Time Performance with Historical or Playback Data video which goes through how the code is setup in the script.

The output will show us if the data is the same, and if the conditions have evaluated the same at the same times, and if the orders are being filled at the same price.

Most of the setup code you can copy and paste over into your script, such as declaring and initializing the sw variable holding the StreamWriter, as well as the WriteToFile() method declaration.

But the message string should contain the information from your conditions.

You can also decide if you want to print all data or limit the data being printed depending on the amount of days being tested over.
I would recommend doing very small tests first, such as a single day, or even a single hour so the test is quicker and there is less to review in the output.

Your script appears to be using Calculate.OnBarClose, so it may not be necessary to write out every tick of data, but just the bar close information and condition values.

Once the debugging output is coded in, be sure to export the script on the development computer then transfer and import on the second computer to ensure the code is the same.

Then provide the output text files that are written out and we can look at what is different in them. I would mostly suspect the data, but it might be a difference in settings such High Order Fill Accuracy or something not identified yet. If the data is the same, the execution output would show us differences and something to further investigate.