Basket Tests Inconsistent Results, Same Data, Dates, Strategy

[ChainsawDR]

Hi,

I have a strategy (MySupportResistance) that I am running a basket backtest for and experiencing different results from the same settings. The strategy uses both minute and tick data. The inconsistent results can be recreated by: I open NT8 > open strategy analyzer > run from 03/01/21 to 05/31/21 for a basket of 500 stocks. Once complete, I fully exit NT8 and confirm via Task Manager that the system has closed before opening NT8 again, and following the exact same steps. The first run produced results of -$3714.30, the second run produced results of -$2853.93 (a 3rd run after a full restart resulted in -$4581). You can see from the summary screenshots that the settings are identical, and some tickers are present in both results (such as 'WOOF' with 1.24 performance) while some are missing (such as ZGNX being present in the first run but not in the second run).

To try and debug, I am using Streamwriter to write the individual trades to a file. Run 1 has 1142 trades not present in Run 2. Run 2 has 728 trades not present in Run 1 (with ~11k trades, this is a 10% difference on each run). When trades are missing (I am matching by ticker and timestamp on trades) they are outright missing, aka not delayed, fully missing. When I look at specific examples I see a consistent behavior... ticker OPK had 6 trade pairs (buy & sell) missing from Run 2 that existed in Run 1, all towards the end of the time period (5/21 onwards) aka the last 6 trade pairs in Run 1 were missing in Run 2. For ticker ABCL, both runs had 7 trades up to 3/5 but then Run 1 had 18 trades from 3/16 onwards that Run 2 did not have. Similarly, ticker WEN had 13 trades in both runs, but then from 3/24 onwards Run 2 had 41 trades that Run 1 did not have.

In summary, there's consistent instances of A) strategies ending early on a per ticker basis, and B) some tickers not running at all (or perhaps they ran but ended very early before any trades took place). My best assumptions are that, although I'm running the program on a powerful computer (AMD Ryzen Threadripper 1950X 16-Core Processor, 32GB RAM), running 500 tickers for 3 months is causing the system to fall over (from a previous ticket, I understand that the data for all 500 tickers is being loaded into memory and the strategy is running against all 500 in parallel, which is likely breaking somewhere) OR I have something funky in my code that is causing these inconsistent results (I don't think I have anything that is causing randomization).

My main questions are:
*Are there any other options for me to debug this situation? I presume I may need to reduce the number of tickers or the date range to a level that doesn't trip the system up - are there any diagnostic tools within the system I can use for this?
*The screenshots show the settings being used. Are there any settings that may avoid this issue that I can use?
*Are there any common issues with strategy logic that can cause inconsistent results?

EDIT UPDATE: When I run the default strategies that comes with NT8 such as Sample MA Crossover, I cannot recreate this issue (tried 500 stocks for 3 months) as it shows consistent results. This implies that there's either a random element within my strategy code that changes each time, or my code is consuming materially more memory and (if it is a memory issue) and so a simple MA crossover is not stress testing the memory aspects. I've reduced my testing down to 15 stocks & one week but still have the issue. Interestingly if I reduce it to a single ticker, then even over 3 months the results are consistent. My new best guess is that sometimes the system will process a basket of stocks in different orders, and some variables are perhaps not being reset and are carrying over to other stocks (or something similar). I will keep investigating.

Many thanks

ChainsawDR

[ChainsawDR]

I added a Streamwriter step at the beginning of OnBarUpdate to write out every tick, then ran it against a basket of 15 stocks for 1 day to see if each run processed the same tick data consistently...

protected override void OnBarUpdate()
{
debugStorage = "Time: " + Time[0] + ", Ticker: " + Instrument.FullName + ", BarsInProgress: " + BarsInProgress + ", Close: " + Close[0];
sw2 = File.AppendText(pathDebug); // Open the path for writing
sw2.WriteLine(debugStorage); // Append a new line to the file
sw2.Close(); // Close the file to allow future calls to access the file again.
...

... unfortunately the strategy is not assessing tick data consistently. These were the number of tick rows written to file for a series of runs where I set it up once, clicked Run, copied the output when finished, clicked Run again (e.g. no exiting and restarting NT8 each time):
1. 120,327 rows
2. 7,849 rows
3. 120,292 rows
4. 30,093 rows
5. 38,184 rows

I'm not sure where to go from here. I am relatively new to coding so I'm aware that my code is not optimal (not DRY, using int when could use byte, storing 20+ variables instead of using an array), and assume it must be something I have coded rather than the NT8 software itself. I'll proceed with turning off code until it runs consistently to try and isolate it.

[ChainsawDR]

Oh this is maddening! So I stripped everything back to a brand new strategy which only does two things: 1) adds tick data, 2) writes each OnBarUpdate. There is no strategy logic whatsoever, yet it still gives inconsistent results! The entirety of the code being used is below. I've attached a 30sec gif where I have pre-selected a basket of 15 stocks to run on 1min data (primary, tick data is added as an additional set in the code) and I hit "Run" several times and you can see in the Strategy Analyzer that the actual instruments being evaluated each time is changing.
1st run: ARVL & HBAN
2nd run: ARVL & ZNGA
3rd run: ARVL & HBAN
4th run: ARVL & VTRS
Summary 4 runs back to back with no changes, 3 different sets of results.

It's not shown in the video but for each of these 4 runs I disconnected all connections (I'd previously downloaded all historical data) but the same thing happens if I connect to my data provider (but testing disconnected has ruled out this being a data provider issue). There's also obviously different volumes of OnBarUpdates being written to file with each run.

Code being used (very basic, no logic):

#region Using declarations
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.ComponentModel.DataAnnotations;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Windows;
using System.Windows.Input;
using System.Windows.Media;
using System.Xml.Serialization;
using NinjaTrader.Cbi;
using NinjaTrader.Gui;
using NinjaTrader.Gui.Chart;
using NinjaTrader.Gui.SuperDom;
using NinjaTrader.Gui.Tools;
using NinjaTrader.Data;
using NinjaTrader.NinjaScript;
using NinjaTrader.Core.FloatingPoint;
using NinjaTrader.NinjaScript.Indicators;
using NinjaTrader.NinjaScript.DrawingTools;
using System.IO;
#endregion


//This namespace holds Strategies in this folder and is required. Do not change it.
namespace NinjaTrader.NinjaScript.Strategies
{
public class TestMissingTicks : Strategy
{
private string pathDebug;
private StreamWriter sw2; // a variable for the StreamWriter that will be used
private string debugStorage;

protected override void OnStateChange()
{
if (State == State.SetDefaults)
{
Description = @"TestMissingTicks";
Name = "TestMissingTicks";
Calculate = Calculate.OnEachTick;
EntriesPerDirection = 1;
EntryHandling = EntryHandling.AllEntries;
IsExitOnSessionCloseStrategy = true;
ExitOnSessionCloseSeconds = 30;
IsFillLimitOnTouch = false;
MaximumBarsLookBack = MaximumBarsLookBack.TwoHundredFiftySix;
OrderFillResolution = OrderFillResolution.Standard;
Slippage = 0;
StartBehavior = StartBehavior.WaitUntilFlat;
TimeInForce = TimeInForce.Gtc;
TraceOrders = false;
RealtimeErrorHandling = RealtimeErrorHandling.StopCancelClose;
StopTargetHandling = StopTargetHandling.PerEntryExecution;
BarsRequiredToTrade = 20;
// Disable this property for performance gains in Strategy Analyzer optimizations
// See the Help Guide for additional information
IsInstantiatedOnEachOptimizationIteration = true;
pathDebug = NinjaTrader.Core.Globals.UserDataDir + "MyDEBUGAnalysisFileDR.txt"; // Define the Path to our test file
}
else if (State == State.Configure)
{
AddDataSeries(Data.BarsPeriodType.Tick, 1);
}
else if (State == State.Terminated)
{
if (sw2 != null) //Necessary to call in order to clean up resources used by the StreamWriter object
{
sw2.Close();
sw2.Dispose();
sw2 = null;
} //end of streamwriter
}
}

protected override void OnBarUpdate()
{
//Add your custom strategy logic here.
debugStorage = "Time: " + Time[0] + ", Ticker: " + Instrument.FullName + ", BarsInProgress: " + BarsInProgress + ", Close: " + Close[0];
sw2 = File.AppendText(pathDebug); // Open the path for writing
sw2.WriteLine(debugStorage); // Append a new line to the file
sw2.Close(); // Close the file to allow future calls to access the file again.
}
}
}

The only things I can think of now are:
A) Am I doing something fundamentally wrong with the primary/secondary data series setup? Do I need to flip them around (so tick is primary, minute is secondary - this would be annoying as I'd like to be able to use a strategy on a chart and see 1min bars on the chart)
B) a bug with NT8.

Thanks for any help with this issue, I'm finding it maddening!

[ChainsawDR]

I flipped the data series setup so primary was ticks and secondary was minute data, but this still has the same issue (gif attached).
Finally, I even removed the streamwriter logic so there was literally no logic whatsoever, and there is still the same issue (gif attached).

I'm at a complete loss how you can run a strategy analyzer over and again on exactly the same data and settings and get completely different results. I must be missing something basic with my setup or settings, any help would be greatly appreciated please! Copying the final basic code that fails:

#region Using declarations
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.ComponentModel.DataAnnotations;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Windows;
using System.Windows.Input;
using System.Windows.Media;
using System.Xml.Serialization;
using NinjaTrader.Cbi;
using NinjaTrader.Gui;
using NinjaTrader.Gui.Chart;
using NinjaTrader.Gui.SuperDom;
using NinjaTrader.Gui.Tools;
using NinjaTrader.Data;
using NinjaTrader.NinjaScript;
using NinjaTrader.Core.FloatingPoint;
using NinjaTrader.NinjaScript.Indicators;
using NinjaTrader.NinjaScript.DrawingTools;
#endregion


//This namespace holds Strategies in this folder and is required. Do not change it.
namespace NinjaTrader.NinjaScript.Strategies
{
public class TestMissingTicks : Strategy
{
protected override void OnStateChange()
{
if (State == State.SetDefaults)
{
Description = @"TestMissingTicks";
Name = "TestMissingTicks";
Calculate = Calculate.OnEachTick;
EntriesPerDirection = 1;
EntryHandling = EntryHandling.AllEntries;
IsExitOnSessionCloseStrategy = true;
ExitOnSessionCloseSeconds = 30;
IsFillLimitOnTouch = false;
MaximumBarsLookBack = MaximumBarsLookBack.TwoHundredFiftySix;
OrderFillResolution = OrderFillResolution.Standard;
Slippage = 0;
StartBehavior = StartBehavior.WaitUntilFlat;
TimeInForce = TimeInForce.Gtc;
TraceOrders = false;
RealtimeErrorHandling = RealtimeErrorHandling.StopCancelClose;
StopTargetHandling = StopTargetHandling.PerEntryExecution;
BarsRequiredToTrade = 20;
// Disable this property for performance gains in Strategy Analyzer optimizations
// See the Help Guide for additional information
IsInstantiatedOnEachOptimizationIteration = true;
}
else if (State == State.Configure)
{
AddDataSeries(Data.BarsPeriodType.Minute, 1);
}

}

protected override void OnBarUpdate()
{
//Add your custom strategy logic here.

}
}
}

[NinjaTrader_ChelseaB]

Hello ChainsawDR,

Are any errors appearing on the Log tab of the Control Center?

May I have the text output files?

May I have a reduced version of the test script that only writes the tick data to file to test on my end?

[ChainsawDR]

Thank you for your help Chelsea. The attached text file is the full strategy that I am running (it's the same as featured in my 1.54pm post). The attached '15 NASDAQ Stocks' image is the 15 stocks being run against for 3/1/21. The attached 'log_errors" image shows the errors for two runs of this file. Interestingly, these errors refer to a file ('MyDEBUGAnalysisFileDR.txt') that no longer exists in the strategy, however it did exist in the previous version of this strategy (posted in full in my 1.31pm post). Is NT8 caching and using an older version of my strategy? (there's no reason why this error should appear otherwise).

I didn't actually know about the log tab (sorry, I won't make that mistake again). As the log file is showing 13 errors, and only 2 on the strategy analyzer summary screen, it looks like these streamwriter errors are the root cause. I have created a brand new strategy without any streamwriter and it runs consistently. Thanks for pointing out the log file. The final two questions I have please are:
1. Is NT8 caching compiled strategies? (the log errors shouldn't have shown up as the latest version of that strategy had no streamwriter code in it)
2. Please could you suggest how streamwriter code should be structured for Strategy Analyzer initiated tests? Aka how do I use streamwriter please within a strategy that is being basket tested across 500 instruments? E.g. writing a simple 'Time[0] + Close[0]' message OnBarUpdate.

Thank you for your help Chelsea!

ChainsawDR

[NinjaTrader_ChelseaB]

Hello ChainsawDR,

It looks like you missed one of the files when uploading. May I have you provide the output file this is being compared with?

Where you have asked:
"Is NT8 caching compiled strategies? (the log errors shouldn't have shown up as the latest version of that strategy had no streamwriter code in it)"

Running instances would need to be reloaded, but the code a script uses when enabled when a new instance is run will be what is compiled.


Where you have asked:
" Please could you suggest how streamwriter code should be structured for Strategy Analyzer initiated tests? Aka how do I use streamwriter please within a strategy that is being basket tested across 500 instruments? E.g. writing a simple 'Time[0] + Close[0]' message OnBarUpdate."

Write each instance to its own file. The filename could include the instrument in the file name.


Importantly, when testing never complicate things. Always reduce the amount of variables and data and simplify as much as possible. If you are running a basket test, it may be beneficial to start with a single instrument test first.

[ChainsawDR]

Thanks Chelsea. I just did a recording of the steps to recreate so I could share with you, however this time I could not recreate it. I can only assume that I made a mistake somewhere along the way (maybe I saved but didn't compile). I will keep an eye on it for future and record a video if it repeats, but for now I think it's safe to close the thread. Thanks for your help (and sorry if I wasted your time towards the end) - the pointer for the logs confirms that my streamwriter code doesn't work for basket tests and needs adapting. I will update this thread if/when I have an alternative solution.

FYI I agree with the principle of keeping it simple. Highlighting that I went down the streamwriter route mainly because of the way NT8 runs basket backtests - which feels suboptimal in the sense that if you want to test a strategy for a large number of instruments (say, 2000 for 6 months of tick data) then all 2000 get loaded into memory which means eventually the system gets stuck (e.g. at 1700 of 2000 it slows to a crawl and will never complete). Currently it is a lot of guess work and trial/error to find out how many instruments can be processed for how long. By using streamwriter, when it slows and will not complete, then at least the performance data for the processed instruments so far (1700 in this case) is available and the time hasn't been wasted. I'll work on trying to resolve the streamwriter issue, either writing to individual files and compiling separately, or some other solution, but overall it'd be great if NT8 could be more sophisticated in how it runs large scale basket tests of strategies.

Thanks again for your help

ChainsawDR