multi-timeframe Strategy stops

[Shansen]

NT Team,

I’ve been trying to track down a persistent problem where a multi-timeframe Strategy (on ES 09-22) (using NinjaTrader Continuum connection) stops processing. Most commonly the program stops shortly before or after NYSE open (9:30am ET). The problem does not occur when using a Playback Connection.

Attempting to isolate where this problem is coming, from I’ve implemented a log (log4net.ILog) and a 1min timer (System.Timers.Timer) which writes to the log on each tick (i.e. each 1min).

Symptom 1) Timer ticks continue while OnBarUpdate events (and other events) stop. The below shows 1) "Timer" ticks each minute and 2) (OnBarUpdate) "OBU FirstTick" on the Primary BarsArray ES 06-22 (1m).
23:25:00,091 OutputTestLog ES 06-22 (1m) OBU FirstTick MEM 1004.8MB
23:25:41,864 OutputTestLog ES 06-22 (1m) Timer MEM 1004.5MB
23:26:00,364 OutputTestLog ES 06-22 (1m) OBU FirstTick MEM 1018.7MB
23:26:41,877 OutputTestLog ES 06-22 (1m) Timer MEM 995.0MB
23:27:02,445 OutputTestLog ES 06-22 (1m) OBU FirstTick MEM 1014.7MB
23:27:41,874 OutputTestLog ES 06-22 (1m) Timer MEM 1014.8MB
23:28:00,081 OutputTestLog ES 06-22 (1m) OBU FirstTick MEM 1028.3MB
23:28:41,889 OutputTestLog ES 06-22 (1m) Timer MEM 994.9MB
23:29:00,251 OutputTestLog ES 06-22 (1m) OBU FirstTick MEM 1003.4MB
23:29:41,906 OutputTestLog ES 06-22 (1m) Timer MEM 1002.7MB
23:29:59,992 OutputTestLog ES 06-22 (1m) OBU FirstTick MEM 1016.9MB
23:30:41,923 OutputTestLog ES 06-22 (1m) Timer MEM 995.2MB
23:30:59,973 OutputTestLog ES 06-22 (1m) OBU FirstTick MEM 1003.3MB
23:31:41,930 OutputTestLog ES 06-22 (1m) Timer MEM 1003.6MB
23:32:00,003 OutputTestLog ES 06-22 (1m) OBU FirstTick MEM 1017.8MB
23:32:41,927 OutputTestLog ES 06-22 (1m) Timer MEM 996.1MB
23:33:00,073 OutputTestLog ES 06-22 (1m) OBU FirstTick MEM 1003.2MB
23:33:41,942 OutputTestLog ES 06-22 (1m) Timer MEM 1003.5MB
23:34:00,001 OutputTestLog ES 06-22 (1m) OBU FirstTick MEM 1017.0MB
23:34:41,938 OutputTestLog ES 06-22 (1m) Timer MEM 996.2MB
23:35:00,043 OutputTestLog ES 06-22 (1m) OBU FirstTick MEM 1004.2MB
23:35:41,940 OutputTestLog ES 06-22 (1m) Timer MEM 1005.0MB
23:35:59,946 OutputTestLog ES 06-22 (1m) OBU FirstTick MEM 1018.4MB
23:36:41,955 OutputTestLog ES 06-22 (1m) Timer MEM 995.9MB
23:37:41,971 OutputTestLog ES 06-22 (1m) Timer MEM 996.3MB
23:38:41,990 OutputTestLog ES 06-22 (1m) Timer MEM 999.2MB
23:39:41,986 OutputTestLog ES 06-22 (1m) Timer MEM 996.3MB
23:40:41,986 OutputTestLog ES 06-22 (1m) Timer MEM 998.5MB
23:41:41,988 OutputTestLog ES 06-22 (1m) Timer MEM 997.7MB
23:42:42,000 OutputTestLog ES 06-22 (1m) Timer MEM 998.6MB

Symptom 2) Memory Usage grows after OBU appears to stop (hourly excerpt below)
00:00:30,425 OutputTestLog ES 06-22 (1m) Timer MEM 1116.0MB
01:00:30,806 OutputTestLog ES 06-22 (1m) Timer MEM 1613.8MB
02:00:31,235 OutputTestLog ES 06-22 (1m) Timer MEM 1991.1MB
03:00:31,697 OutputTestLog ES 06-22 (1m) Timer MEM 2366.4MB
04:00:32,065 OutputTestLog ES 06-22 (1m) Timer MEM 2731.1MB
05:00:32,459 OutputTestLog ES 06-22 (1m) Timer MEM 3096.3MB
06:00:32,835 OutputTestLog ES 06-22 (1m) Timer MEM 3477.0MB

Please advise :
- any ideas on what can stop OBU events.
- any ideas on how to identify what could be causing this issue.

To date,
- the NinjaScript Utilization Monitor has been of no help. At a guess, with OnBarUpdate stopping(?), any Strategy associated utilization stops. There is no standout utilization for any NinjaScript.
- the log is inconclusive, it is as though the application grinds to a halt at different locations each time.
- At no time is there an exception (i.e. null object, index error, anything) montored via VS.

As always any guidance is welcome.

[NinjaTrader_ChelseaB]

Hello Shansen,

I would start very simple.

Create a test script that does nothing but add the series, and prints the time of the bar and BarsInProgress in OnBarUpdate().

Are you able to reproduce? (This would mean its not the other code in the script and something about the data or connection)

Are you seeing any connection losses in the log?


"any ideas on what can stop OBU events"

Over utilizing the processor.
Connection losses.
Freezing threads.