Backtest Stuck on 97/100 - Debug Tips?

[ChainsawDR]

Hi,

When running backtests on a basket of stocks/instruments, I'm frequently finding that larger tests (either larger number of stocks, or longer period of time) are stalling out. E.g. I'm currently trying to run a backtest of a strategy across 3 months of data (I already pre-downloaded the historical data for all the stocks in advance) for 100 stocks. It has reached 97/100 after 10.5hrs (an overnight run) but stalled at this 97/100 point for the last couple of hours. My systems CPU is fine (1% being used) and memory isn't an issue (only 36% used, 4,326MB).

Are there any tips/tricks for how I can debug this? (e.g. I can't see which stock is failing to understand if its a problem with that specific stock or something else)
Any tips for how I can pre-estimate how long I can run a stock for? (waiting 10hrs to learn it stalls at 97/100 is frustrating, looking for ways to avoid this).
Any hidden ways of being able to pause/unpause a test to try and shake it free?

Many thanks

ChainsawDR

[ChainsawDR]

Note: Running the 100 stocks for 1 month works fine, running for 2months works fine.

[NinjaTrader_ChelseaB]

Hello ChainsawDR,

I'm not sure what could cause an all out stall unless there was an error hit or something specific to that computer such as running out of memory.

If the stall occurred, the CPU usage would then drop and the memory would be cleaned up by garbage collection. Likely this is why you are seeing these low, as they would not be low during the backtesting.

Are you able to reproduce this behavior using the SampleMACrossover included with NinjaTrader?

[ChainsawDR]

Thanks Chelsea. I ran the SampleMACrossover strategy against 100, 500 & 1000 stocks with no issue. I also re-ran the strategy that had twice stalled out, and it completed this time (sigh - apparently twice wasn't enough to make sure it was recreatable!). I will run a few more tests to determine if I can recreate the issue consistently. Hoping to ask a few related questions please.

I have added "Print(String.Format("{0} is being used as the input series", Instrument.FullName));" within StateDataLoaded state, and "Print(String.Format("{0} state terminated", Instrument.FullName));" within State.Terminated - to try and understand where it is stalling. I was kind of expecting the system to load say 32 data series (multi-threading) and once a stock was finished being processed, it's state would terminate and a new stock be added. It doesn't look to operate that way. Eg. I'm running a 3 month test against 1000 stocks and it is currently on iteration 432/1000 and the ninja output is showing only stock add messages like 'FITB is being used as the input series" aka it's loaded 432 stocks tick data into memory and doesn't appear to have terminated any of them. As tests like this go on, they start out very fast and then get slower, and slower, and slower.

Q1. Is NT8 keeping all of the series data for all of the stocks in memory until all 1000 complete? If Yes, is there a way to avoid this (as performance will obviously degrade as the 500th stock is added, 600th stock is added etc)? If no, are there any print statements I could use to see the stocks being added/removed from memory? FYI I'm using IsInstantiatedOnEachOptimizationIteration = true.

Q2. I have pre-downloaded 6 months of tick & day data for 2,000 stocks. When running tests, is there any debug code (Prints etc) that I can use to understand if NT8 is using the historical data vs downloading from my data provider (iqfeed)? Asking because perhaps there was an unknown error during the historical download and my first two runs were stalling because they were trying to download fresh data, whereas the run I did today may have just pulled the data from the cache (I've no data to support this theory, hence trying to understand how to debug it/rule it out).

Q3. I am seeing an inconsistent error message: "Strategy 'MyTestStrat': Error on calling 'OnStateChange' method: Object reference not set to an instance of an object." in the NinjaScript Output. Do you know if there is anyway of understanding which stock/instrument was triggering this message? (I presume it's a trial and error of going stock by stock but wanted to double check).

Thanks again for your help Chelsea

ChainsawDR

[NinjaTrader_ChelseaB]

Hello ChainsawDR,

Unfortunately, IsInstantiatedOnEachOptimizationIteration applies to optimizations and does not apply to backtests.
https://ninjatrader.com/support/foru...39#post1069339

I'm not aware of a way to tell within a script if data is being downloaded or pulled from the cache. If the data is downloaded, the cache will be used. If there is data missing it will try and pull from an active connection. But you can predownload the data and then disconnect from the connection and only the cache can be used.


I don't have details on the inner workings of when garbage collection is able to clean up basket test instances. I'll ping our development and report back any information I am able to find. I can say that you cannot change the inner workings.


'Object reference not set to an instance of an object.' is a variable with a null value that tried to get used.

If there is an error in OnStateChange occurring this could explain the hang. Unfortunately, the script would have to at least progress to State.DataLoaded before the instrument it is applied to can be found with a print. If the error is occurring before the Instrument object is prepared, the script would never reach that code.

You could check to see which instruments are not getting prints of the instrument and progressing to State.Historical. And narrow down which ones are not causing an issue.

Also I recommend reducing the script and finding the line of code causing the error. If you comment out all of the added logic in OnBarUpdate() does the behavior continue?
Comment out all extra logic in OnStateChange(). Does the behavior continue?

[ChainsawDR]

Thanks Chelsea, yes knowing a bit more of the inner workings may help. I'm imagining that I may need to use StreamWriter to pinpoint at which step it stalls. I'll look into that specific error. In the meantime, attaching two screenshots as an example of the stall. While running a strategy against 1,000 stocks for 3 months, it reached 999 after 2hrs4mins (NT8 1.png) and has now stayed stalled at this same progress for the last 30mins (NT8 2.png showing the system is still running but no progress). I did a run last night that stalled at 99/100 and went to bed for 7hrs and got up and it didn't complete, so it seems to be a pattern - at least with this strategy, but it's odd that it runs ok when I break the runs up into smaller batches (aka I've never not been able to run a strategy against certain stocks or timeframes, it just comes down to how many I can run without it stalling out like this each time!)

If you are able to share more info on Q1 please that'd be a big help.

Many thanks

ChainsawDR

[NinjaTrader_ChelseaB]

Hello ChainsawDR,

To summarize the information I've received back from our development:

1. The historical data does stay in memory until all consumers have been terminated. (Think of scripts as children of the data thread as the data processing events (like OnBarUpdate) come from the data thread). From there memory is marked for cleanup by the garbage collection. As the scripts do not terminate until the test completes, this means the data stays in memory until the scripts terminate.

2. The TradeHistory collection will be a big part of memory consumption. The impact of this can be lessened by setting IncludeTradeHistoryInBacktest to false. This can be set to false if the strategy is not using the SystemPerformance collection internally in the code.

https://ninjatrader.com/support/helpGuides/nt8/includetradehistoryinbacktest.htm