Disconnection leaves strategy in limbo

[village]

so today i paid the money to lease NT and went live. First strategy is running and has opened a position fine. It has even set a stop loss which is showing up as a pending order on Interactive Brokers TWS. All good so far.

a few hours later i get an internet disconecction for a couple of minutes. when it reconnects, NT reconnects fine, but we have the following situation;
- the stategy has been disabled and the open p/l has gone. re enabling it doesnt bring anything back, it just restarts it fresh
- there is a live open position with IB. however in NT the strategy doesnt seem to know anything about it so presumably is not going to be doing anything about closing it
- NT can 'see' the position in the 'positions' tab, and it is updating, so it isnt an interface/connection issue
- as if not being able to remember what it is doing isnt bad enough, it has cancelled the stop order at IB, leaving the stategy unprotected. why would it do that????? why?

Can it really be the case that something as common as a brief disconnection basically f**** any strategy that is open, wipes out the stops for good measure, and requires manual closing and/or resetting of stops for any open positions

Please please tell me it aint so.

[NinjaTrader_RyanM1]

Hi Village,

I'm sorry you are frustrated by this situation. NinjaTrader strategies are always driven by their position so there is the potential to see a mismatch between account and strategy position. Background information for this is available here:

http://www.ninjatrader.com/support/helpGuides/nt7/strategy_position_vs_account_p.htm

There is built disconnect handling for strategies you can set in Tools > Options > Strategies > NinjaScript tab. You may be able to adjust the settings there to accommodate the type of disconnections you receive and the best handling for your strategy. These settings are expanded here:

http://www.ninjatrader.com/support/helpGuides/nt7/connectionlosshandling.htm

More control on connection loss is available by working in OnConnectionStatus() handler:

http://www.ninjatrader.com/support/helpGuides/nt7/onconnectionstatus.htm

[village]

Hi Ryan

Ok sorry for the swearing. edited.

In the Tools > Options > Strategies > NinjaScript tab, "keep running" is already the selected option, which I would have preferred it did , but it didnt. So clearly NT is supposed to be able to handle disconnections up to a point

I have checked the log, and there are multiple disconnections listed, many of which it survived, but the one that seems to have closed them down says "strategyxxxx lost price connection more than 4 times in past 5 minutes and will be disabled'. is that parameter able to be changed?

there was also a special one a few minutes later highlighted in brown with the words 'HMDS data farm connection is broken:ushmds2a" but by that stage i think they had already shut down. Does that provide any clues?

I will try dialing up the other numbers there to see if it makes any difference.
have also unchecked the 'cancel exit orders when stategy disabled' , which hopefully will at least stop the stop being cancelled.

thanks

[NinjaTrader_RyanM1]

Yes, 4 times in 5 minutes is the default setting there, so may work better to increase this if the disconnections can take longer and you want it to keep trying.

It's also good to find out if anything can be done to improve connection stability. You can check firewall/os settings, network hardware and the connection api to ensure everything is in good working order. The message sounds like an IB one, so make sure you are using the latest supported TWS version: 917.

[village]

have had no further problems. There were storms about that night which in retrospect must have been afftecting the connection, because it has been much better since. I also changed the settings to 10 times for good measure

[fxeconomist]

i can confirm the limbo...

I made myself a tick collector application. After disconnection and reconnection no tick seems to come thru anymore (database didn't receive anything new for days). Also, I couldn't find out the exact disruption time (it was supposed to be recorded in the database) because there might be something wrong with OnConnectionStatus. This is my code:

protected override void OnConnectionStatus(ConnectionStatus orderStatus, ConnectionStatus priceStatus)
{
  DateTime t,utc; 
  t=DateTime.Now;
  utc=t.ToUniversalTime();
  nowConState = priceStatus;
  if (nowConState!=oldConState)
  {
   if (nowConState==ConnectionStatus.Connected)
     {
       Print("Connected at "+TimeImage(t));
      }
   if (nowConState==ConnectionStatus.Disconnected)
     {
      Print("Disconnected at "+TimeImage(t)+", UTC time: "+TimeImage(utc));
      if (OdbcOk==true)
	{
	  if (use0==true)
	    {
	     for (int i=0;i<SymbolCount;i++)
		 {
		   Insert(i,"LOST",0,0,utc);
		  }
	    }
	 else
	    {
	      for (int i=1;i<=SymbolCount;i++)
		 {
		  Insert(i,"LOST",0,0,utc);
		 }
	    }
	}//if (OdbcOk==true)  
    }//if (nowConState==ConnectionStatus.Disconnected)
    oldConState=nowConState;
   }//if (nowConState!=oldConState)
 }

My OnConnectionStatus() is supposed to display something when it has either a connection established or it gets disconnected. Disconnection is reported by NT, but OnConnectionStatus() doesn't seem to fire:

**NT** Enabling NinjaScript strategy 'TickCollector/ba16810b180343e08e3c2545c4c770c6' : On starting a real-time strategy - StrategySync=WaitUntilFlat SyncAccountPosition=False EntryHandling=AllEntries EntriesPerDirection=1 StopTargetHandling=PerEntryExecution ErrorHandling=StopStrategyCancelOrdersClosePositio ns ExitOnClose=True/ triggering 30 before close Set order quantity by=Strategy ConnectionLossHandling=KeepRunning DisconnectDelaySeconds=10 CancelEntryOrdersOnDisable=False CancelExitOrdersOnDisable=True MaxRestarts=4 in 5 minutes
Initializing SQL connection
SQL connection initialized.
Connected at 2011/8/16 1:56:19
**NT** Strategy 'TickCollector/ba16810b180343e08e3c2545c4c770c6' lost price connection but will keep running.
Connected at 2011/8/16 11:36:12

As you can see, it fired for the connection event, but not for the disconnection event. Furthermore, the database didn't receive any "LOST" item. You might think I coded it wrong, and perhaps OnConnectionStatus() works, but my code is wrong; then why does it recognize that status changes from connecting to connection, (connected at ...11:36) but not from connection to disconnection, which is not reported ? But what's the worst thing is that OnMarketData() didn't receive anything after connection upon disconnection, cause I can't code around to make it work...

[NinjaTrader_RyanM1]

Hello fxeconomist,

Best will be to simplify this down a good bit, so you can see what exactly is reported. Use something like below to see what is reported during a disconnection.
Print(priceStatus + " " + orderStatus);

[fxeconomist]

Ok Ryan, I did as you say here. Now I play the waiting game to see what happens. But I still wonder, even if we correct the OnConnectionStatus(), how can we make OnMarketData() receive tick events again?

[sledge]

No need to wait, then the power off to your modem..

I have my cable modem and router on UPS. On my old UPS, cyberpower 900?, I get about an hour.

Ok Ryan, I did as you say here. Now I play the waiting game to see what happens. But I still wonder, even if we correct the OnConnectionStatus(), how can we make OnMarketData() receive tick events again?

[fxeconomist]

No need to wait, then the power off to your modem..

I have my cable modem and router on UPS. On my old UPS, cyberpower 900?, I get about an hour.

NinjaTrader is an AUTOMATIC trading platform. Means that I don't have to check when it stops receiving quotes, even if connection is green, means that I don't have to do anything with the router, because my system may be trading on a VPS server, means that if the network uptime is 99.9%, my system will work for 99.9% of the time, and I will have the safety that a short disconnection of a few seconds or a few minutes will not put my account in jeopardy, and, when connection is restored, event behaviour returns back to normal. If there is no way to get normal event behaviour after connection restore, then this is not a reliable trading platform...and it seems that OnMarketData() is not receiving anything after connection restore...

[NinjaTrader_Bertrand]

fxeconomist, NinjaTrader is not a black box, but an advanced automatic trading assistant / tool that you would need to supervise a regular intervals. Are you checking in your script both the order and price status connections?

[fxeconomist]

Now I inserted the print line as you said and I'm waiting for the connection to drop. If it doesn't happen today, I will resume tests monday.

[fxeconomist]

Now will shut the internet down to see the events.

[fxeconomist]

Simulation results:

Initializing SQL connection
SQL connection initialized.
Connected Connecting
Connected at 2011/8/19 14:49:51
Connected Connected
**NT** Strategy 'TickCollector/e7b371ef59544952a770b93d88466fd4' lost price connection but will keep running.
ConnectionLost Connected
**NT** Strategy 'TickCollector/e7b371ef59544952a770b93d88466fd4' lost order connection but will keep running.
ConnectionLost ConnectionLost
ConnectionLost Connected
Connected Connected
Connected at 2011/8/19 17:49:21

I can confirm that OnMarketData() is still working, as the database continues to grow.
However the disconnection was not reported, and it seems that Disconnected is not reported. Rather, ConnectionLost. And, connection resumed on the machine running Zen-Fire. Machine running PFG did not reconnect, stays in Connection Lost. I had to manually disconnect and connect back.

But this was a test. We just cut internet out from the host. I want to see what happens when it loses connection without the internet being cut.

[NinjaTrader_Bertrand]

Correct, it would report ConnectionLost and not disconnected, this is the state you see when manually disconnect from the provider.

What NT7 version and PFG API do you use here in your testing?