Using a multi instrument indicator in a strategy

[arnonmoscona]

Hi!

I am having quite a bit of confusion trying to use my custom multi-instrument indicator with a strategy.

Basically I have a multi-instrument indicator that looks at a watch list and is able to select which instrument(s) is most interesting to look at of a group. It displays fine as an indicator and I've been using it for several weeks. When used for display only on the charts it does not create any undue load.

Now I want to use it in a strategy. I created a do-nothing strategy to first see how this would basically work. I tried three approaches - all of them failed.

1. I added the indicator in the Initialize() method using the Add() method, which kept a reference to the indicator in a private class variable. This failed because it seems that all historical bars were first "pumped" to the strategy, with nothing in the indicators, and only then were the bars "pumped" to the indicator. So whenever OnBarUpdate() was called in the strategy there was nothing in the indicator yet. First I saw all the OnBarUpdate() calls on the strategy, and only after they were done did I see all the OnBarUpdate() calls on the indicator. The indicator, however displayed correctly in the chart. No undue load was generated.
2. So I tried basically the same thing, but tried to call the indicator's Update() method from within the strategy before attempting to get information from the indicator. This caused NT to basically hang, using all the CPU of one core (i.e. one thread going on some infinite busy loop). Bummer.
3. Tried to create a new instance of the indicator in each OnBarUpdate() of the strategy. Again, NT got into a tizzy, spinning one thread and hanging.

If I try approach 2 with MACD as an alternative then I have no problem. Now, MACD would be using the same instrument as the strategy. My indicator adds a bunch of instruments of its own. Now that does not seem to be a problem for display (which does not have to be coordinated with the strategy per-se), however, I suspect that this is probably the core reason behind the problem here.


Any ideas what might be going wrong here?


(BTW please don't ask for the code of the indicator. It's quite complex and I do not have a "Toy example". In fact I don't thing there is such a thing as a reliable toy example for a multi-instrument indicator as any functioning indicator has to deal with the fundamental problem of bar synchronization, as NT provides no help on the issue and everybody ends up with their own solution (or a very buggy indicator))

[NinjaTrader_Bertrand]

arnonmoscona, thanks for the post. I'm not aware of general issues here, so we would need to isolate out why you're getting this unexpected behavior:

a) does it depend on how many instruments you add in the called indicator? If you trim this down, any change?

b) What CalculateOnBarClose do you work on?

c) Any log / trace file error noticed as you experience the load issues?

d) Do you try to access / print any exposed series / value from the indicator or just Add() for now?

e) Same outcome if you accessed the custom MultiSeries indicator from another indicator instead of a strategy?

Thanks

[arnonmoscona]

With 3 extra instruments - it works (zero extra load). With 7 extra instruments it totally hangs.

Yes, I do have print statements of internal state (right now a size of a list the indicator returns). When I have 3 extra instruments (using approach 2) I get the values I expect. Without calling Update() on the indicator I don't get the values I expect. With 7 extra instruments It hangs on the call to Update(). The indicator itself displays just fine without extra load even if I put 20 instruments on it, which is not surprising, as the indicator calculations are not all that complicated (most of the logic is for matching up bars).

Both the indicator and the strategy have CalculateOnBarClose=true

In the scenario when it hangs no new log or trace is generated. In the scenario when it works you see in the log the activation and deactivation of the strategy as in:

6/19/2012 7:05:19 AM|1|128|Enabling NinjaScript strategy 'MySimpleIndexArbitrage/00396da70714410ba5fd989e3730bf99' : On starting a real-time strategy - StrategySync=SubmitImmediately SyncAccountPosition=False EntryHandling=AllEntries EntriesPerDirection=1 StopTargetHandling=PerEntryExecution ErrorHandling=StopStrategyCancelOrdersClosePositio ns ExitOnClose=False Set order quantity by=Strategy ConnectionLossHandling=KeepRunning DisconnectDelaySeconds=10 CancelEntryOrdersOnDisable=False CancelExitOrdersOnDisable=False CalculateOnBarClose=True MaxRestarts=4 in 5 minutes
6/19/2012 7:07:16 AM|3|128|Failed to call method 'Initialize' for strategy 'MyDebugStrategy/b9d82e53296049caae38085f5160cefb': This is a debug only strategy and may only be used in the simulation account
6/19/2012 7:07:39 AM|1|128|Disabling NinjaScript strategy 'MySimpleIndexArbitrage/00396da70714410ba5fd989e3730bf99'
6/19/2012 7:07:42 AM|1|128|Enabling NinjaScript strategy 'MySimpleIndexArbitrage/00396da70714410ba5fd989e3730bf99' : On starting a real-time strategy - StrategySync=SubmitImmediately SyncAccountPosition=False EntryHandling=AllEntries EntriesPerDirection=1 StopTargetHandling=PerEntryExecution ErrorHandling=StopStrategyCancelOrdersClosePositio ns ExitOnClose=False Set order quantity by=Strategy ConnectionLossHandling=KeepRunning DisconnectDelaySeconds=10 CancelEntryOrdersOnDisable=False CancelExitOrdersOnDisable=False CalculateOnBarClose=True MaxRestarts=4 in 5 minutes
6/19/2012 7:08:50 AM|1|128|Disabling NinjaScript strategy 'MySimpleIndexArbitrage/00396da70714410ba5fd989e3730bf99'

[NinjaTrader_Bertrand]

Thanks Arnon, which instruments / instrument classes would the indicator Add()?

If you open charts for those separately and then starting the test dummy strategy calling your MultiSeries indicator, any difference seen?

It would really help if we could reproduce this here, so if you see a chance to submit a stripped down example failing we would appreciate it.

[arnonmoscona]

In the tests done here the case with 3 extra instruments was: ^SP500,GLD,AGQ
The test with 7 extra instruments was: ^SP500,BAC,WFC,MS,JPM,GS,C,AIG

I tried opening a chart with all 7 instruments and adding the strategy with the 7 instruments to another chart - same result. Hangs. Trying to add the strategy to the same chart window that has all 7 instruments on it also produces the same result. Hangs.

As mentioned before, a "stripped down version" would take quit a bit of work.

As you can see in the attached image, I have the indicator running in multiple instances with various stock groups (including the ones I am testing with) with no problem at all and no detectable CPU load. It only hangs in the context of the strategy when Update() is called or where, equivalently, you try to access one of the Values[].

[NinjaTrader_Bertrand]

Thanks for the screenshot Arnon, nice work.

So the instruments are all added hardcoded into the Add() methods right? No dynamic adding being used?

I will need to create an example here to check into and see if this would be reproducible on our end with the same range of instruments.

What value do you expose for access? A signal data Series?

Thanks,

[arnonmoscona]

Actually, the indicator is dynamic in the following sense:

In the Initialize() method, it reads a CSV file where the stock groups are specified. It picks up the lines that match the group that is designated in a parameter (e.g. "Major financials"). It then goes into a loop where it adds all the instruments in the list (plus creates corresponding internal structures, and adding the relevant Value series and plots).

By the time Initialize() is finished - everything is in place and remains static after that. Data is processed during the OnBarUpdate(). There is no OnStartup() or anything like that in this indicator.

[NinjaTrader_Joydeep]

Hello armonmoscona,
The initialize section is called multiple times. Thus if you do not close the stream (while reading the CSV file), then it can throw errors. However we cannot say for sure unless you send a reproducible toy code* to us.

Please append Attn:Joydeep in the subject line of the email and give a reference of this thread in the body of the email.

I look forward to assisting you further.

*The "toy" just means something that is a stripped down version that isn't necessarily the whole logic. It makes things easier to rout out.

[arnonmoscona]

For one thing, it's obvious that streams must be closed. That is clearly not the issue here. If it was, then it would be a problem for the use of the indicator in general - where Initialize() also called multiple times. It is not a problem there.

As I mentioned already several times - it would be quite a bit of work to "strip down" the indicator. It has lots of dependencies and who is to know which piece of code is relevant here. Most of the code in the indicator is about overcoming deficiencies in NT itself, which provides no mechanism to handle multiple instruments on the same timeline. The indicator would be trivial if it wasn't for the deficiencies of NT. All it does is compare momentum.

I can try to create another indicator to reproduce the problem but it will take time and there is no guarantee that the problem would be reproduced. So it doesn't make much sense.

If you really want I can send you the indicator and the strategy along with all the dependencies and supporting CSV files and you can try to debug there. I have absolutely no way to debug this as NT totally hangs as soon as the indicator Update() is called. It is also totally unexplainable for me why it should work smoothly with 3 instruments and not with 7...

[NinjaTrader_Joydeep]

Hello armonmoscona,
Using the try catch blocks, are you able to catch the exceptions, or it still crashes.

http://ninjatrader.com/support/forum/showthread.php?t=9825

[arnonmoscona]

I can give you and additional data point.

I repeated the test multiple times. Starting with a sublist of 3 and increasing by one every time. It turned out that one specific instrument: GS is causing the strategy to hang (although it does not cause the indicator when used on its own to have any problems).

I can work fine with even larger lists of instruments, but if GS is added the strategy hangs. The GS chart shows no problem. The indicator by itself (as an indicator on a chart) shows no problem if GS is included. And I double verified that both the indicator code and the strategy code have CalculateOnBarClose=true.

Note, however that GS is a relatively high tick volume instrument. So maybe just saying CalculateOnBarClose=true is somehow insufficient for a strategy...

[arnonmoscona]

There are try-catch blocks in the code. There is no evidence for any exceptions. The problem is load. The CPU is pegged on one thread (presumably the indicator thread).

[NinjaTrader_Joydeep]

Hello arnonmoscona,
Unfortunately we can do little unless we can replicate it at our end.,

[arnonmoscona]

Just FYI. I managed to find out where it was hanging. For the benefit of other users, I used an open source tool called Managed Stack Explorer (see http://mse.codeplex.com/). While this tool was written for .NET 2.0 it seems to work fine in .NET 3.5 (some people say it does not work in .NET 4.0, but I didn't check that, and that's irrelevant for now for NT)

Using this I managed to locate the thread in which NT calls the indicators. Then I induced the situation that caused the hang, and could locate where exactly it happens. From there is was a simple matter of debugging.

The bottom line is that when NT hangs, pegging the CPU, and no prints, logs, or traces are produced, you can use an external tool to obtain a stack trace externally from the running application (even when it's unresponsive). I also found a discussion where someone suggested windbg with the command ~*e !clrstack (which I did not try myself).

[NinjaTrader_Bertrand]

Glad to hear you could resolve this Arnon and thanks for sharing your tip here on our forums.