Backtests of an entire Instrument List skips a few contracts at random each backtest.

[ChiTrader2000]

Hello,

When I run a backtest on an entire Instrument List of many (>1000) contracts by right clicking on the Instrument List (in NT7 Beta 4) and choosing "Backtest...", a few contracts show 0 trades.

This wouldn't surprise me except that the list of contracts that is missing is different on each backtest.

Any ideas on what may be causing this?

Thanks in advance,

ChiTrader2000

[NinjaTrader_Austin]

ChiTrader2000, thanks for the observation. We will look into this.

Just checking, but can you run the backtest on the individual instruments that don't appear in the larger backtest (do they have data)?

[ChiTrader2000]

NinjaTrader_Austin,

1. Great question. I forgot to mention the most important part of the problem in my last post... any contract that randomly has no data when I backtest an entire Instrument List always has trades when I backtest it by itself. Also useful information is that this problem does not happen with NT 6.5.

2. I had a thought. Since NT7 backtests now support multiple processors, I wonder if I have a threading issue. I set lots of contract-specific class-level member variables in the Initialize() method and on the first OnBarUpdate() in my strategy, and I haven't attempted to put a lock around class-level member variables.

I wonder if the NT backtesting engine starts a backtest of one contract on one thread, then starts a backtest of a second contract on a second thread before the first contract is done being backtested, which then overwrites my class-level contract-specific member variables before the first contract is done being backtested.

Is this threading issue possible?

Or does NT7 create separate instances of a strategy class for each thread on each processor when an entire Instrument List of 1000's of contracts is being backtested at once?

Or does NT 7 have some other way to make sure that a new contract doesn't start getting backtested on a thread until the contract currently being backtested is done?

Thank you very much in advance.

ChiTrader2000

[NinjaTrader_Dierk]

Ok, if you can reproduce on a standard strategy like SampleMaCrossOver then we could try this:
- please try find a setup which would use minute (or daily) data
- zip:
* the directory <my documents>/NinjaTrader7/db/minute
* the file <my documents>/NinjaTrader7/db/NinjaTrader.sdf
- upload the zip archive to some file server (e.g. our file server by Help->Mail to support)
- please send me download and instructions on how to reproduce

Thanks

[ChiTrader2000]

Helllo,

I have been trying to reproduce this problem all day with a simple strategy on the daily Yahoo data but have so far been unable to reproduce the problem that is happening with my strategy and my 1 minute bar data.

Unfortunately, I will not be allowed by my employer to give out our data or exact strategy.

#1. Is there any way to force NinjaTrader to backtest contracts one at a time, synchronously, on a single thread?

#2. If I can locate a single processor machine in our office, I will try to see if this makes a difference. Should a single processor machine cause backtests to occur as described in my question #1 above?

Thanks in advance,

ChiTrader2000

[kdoren]

If you are running Windows Vista (or probably Windows 7) you can use the Windows Task Manager to set Processor Affinity for a Process. I think if you uncheck all processors except one for NinjaTrader.Exe it should do what you want.

Although you are probably seeing a software problem, you should also verify that you don't have a hardware problem; stressing your hardware can cause hardware problems to surface that look like software problems. I suggest downloading "prime95" from http://www.mersenne.org/freesoft/
and running a stress test overnight.

I've run some backtests & long optimizations on instrument lists with NT7 and I haven't seen anything like what you're seeing, although I'm not running nearly as many instruments. The main problems I've had were that I had a few cases where my strategy threw exceptions (visible in the log) that caused certain instruments to not show any trades. I believe the same strategy under NT6.5 didn't throw any exceptions. But the problem was in the strategy, and the behavior was consistent whether I ran the strategy by itself or as part of a list.

[NinjaTrader_Dierk]

We needed a reproducible scenario in order to analyse. Unfortunately there is no way around.

[ChiTrader2000]

1. kdoren,

Thanks a lot for the info. It helped. I tried backtests with NinjaTrader.exe's affinity set to only one processor. While this didn't fix my problem, now I know and don't need to hunt down a single core machine. Also, I have been checking the logs for errors. Unfortunately, there are no errors and this problem is still happening.

2. NinjaTrader_Dierk,

I understand completely. Since I can't give over our strategy and data, I realize that I am on my own to determine the cause of this problem.

3. Everybody (anybody, please),

Question: Where can I place code that should occur once and only once per contract in a large backtest of many contracts, at the beginning of the backtest of each contract?

In NT6.5 I put this code in the Initialize() method, but it seems that this method is no longer works in this way. Also, I have moved this contract-specific initialization code into the OnBarUpdate() event, like this:

Variation 1:

if (this.CurrentBar == 0 && this.BarsInProgress == 0)
{
InitializeNewContract(); // Does a pre-backtest initialize for each contract.
}

Variation 2:

// outside OnBarUpdate() method:
privatestring _lastInstrumentFullName = null;

// inside OnBarUpdate() method:
if (this.Instrument.FullName != _lastInstrumentFullName)
{
_lastInstrumentFullName = this.Instrument.FullName;
InitializeNewContract(); // Does a pre-backtest initialize for each contract.
}

Does anybody see any weaknesses with either of these two variations, and does anybody have a better way to do what I am trying to do?

Thank you very much in advance,

ChiTrader2000

[NinjaTrader_Austin]

Question: Where can I place code that should occur once and only once per contract in a large backtest of many contracts, at the beginning of the backtest of each contract?

if (CurrentBar == 0)
{
    // do something once
}

[ChiTrader2000]

THE SOLUTION to NT7 multi-threaded backtesting issues: CAREFUL with use of &quot;STATIC&quot;

Hello everybody,

I figured out the issue that I explained earlier in this post!

Be very careful with your use of static classes and member variables in NT7 strategies, even if they worked in NT6.5.

In NT7, unlike NT6.5, backtesting occurs on multiple threads at once, which makes backtests on multi-core machines go significantly faster.

This multi-threaded nature of NT7 backtesting means that on a multi-instrument backtest, backtesting can occur on multiple contracts at the same time.

You can observe this by writing to the Log on the first bar of a new contract then writing to the Log every so many bars in the course of a backtest. You will see that the Log entries of different contracts are not 100% in time order.

If you have static variables or classes in your strategies, know that all backtests that are occurring at the same time may be accessing (and changing the values of) the static variables and classes simultaneously.

Hope this helps somebody out there.

ChiTrader2000

[NinjaTrader_Ray]

Thanks for posting that, I am sure that will come in useful.