I have a strategy (MySupportResistance) that I am running a basket backtest for and experiencing different results from the same settings. The strategy uses both minute and tick data. The inconsistent results can be recreated by: I open NT8 > open strategy analyzer > run from 03/01/21 to 05/31/21 for a basket of 500 stocks. Once complete, I fully exit NT8 and confirm via Task Manager that the system has closed before opening NT8 again, and following the exact same steps. The first run produced results of -$3714.30, the second run produced results of -$2853.93 (a 3rd run after a full restart resulted in -$4581). You can see from the summary screenshots that the settings are identical, and some tickers are present in both results (such as 'WOOF' with 1.24 performance) while some are missing (such as ZGNX being present in the first run but not in the second run).
To try and debug, I am using Streamwriter to write the individual trades to a file. Run 1 has 1142 trades not present in Run 2. Run 2 has 728 trades not present in Run 1 (with ~11k trades, this is a 10% difference on each run). When trades are missing (I am matching by ticker and timestamp on trades) they are outright missing, aka not delayed, fully missing. When I look at specific examples I see a consistent behavior... ticker OPK had 6 trade pairs (buy & sell) missing from Run 2 that existed in Run 1, all towards the end of the time period (5/21 onwards) aka the last 6 trade pairs in Run 1 were missing in Run 2. For ticker ABCL, both runs had 7 trades up to 3/5 but then Run 1 had 18 trades from 3/16 onwards that Run 2 did not have. Similarly, ticker WEN had 13 trades in both runs, but then from 3/24 onwards Run 2 had 41 trades that Run 1 did not have.
In summary, there's consistent instances of A) strategies ending early on a per ticker basis, and B) some tickers not running at all (or perhaps they ran but ended very early before any trades took place). My best assumptions are that, although I'm running the program on a powerful computer (AMD Ryzen Threadripper 1950X 16-Core Processor, 32GB RAM), running 500 tickers for 3 months is causing the system to fall over (from a previous ticket, I understand that the data for all 500 tickers is being loaded into memory and the strategy is running against all 500 in parallel, which is likely breaking somewhere) OR I have something funky in my code that is causing these inconsistent results (I don't think I have anything that is causing randomization).
My main questions are:
*Are there any other options for me to debug this situation? I presume I may need to reduce the number of tickers or the date range to a level that doesn't trip the system up - are there any diagnostic tools within the system I can use for this?
*The screenshots show the settings being used. Are there any settings that may avoid this issue that I can use?
*Are there any common issues with strategy logic that can cause inconsistent results?
EDIT UPDATE: When I run the default strategies that comes with NT8 such as Sample MA Crossover, I cannot recreate this issue (tried 500 stocks for 3 months) as it shows consistent results. This implies that there's either a random element within my strategy code that changes each time, or my code is consuming materially more memory and (if it is a memory issue) and so a simple MA crossover is not stress testing the memory aspects. I've reduced my testing down to 15 stocks & one week but still have the issue. Interestingly if I reduce it to a single ticker, then even over 3 months the results are consistent. My new best guess is that sometimes the system will process a basket of stocks in different orders, and some variables are perhaps not being reset and are carrying over to other stocks (or something similar). I will keep investigating.
Many thanks
ChainsawDR
Comment