60 Instrument Test

[greentrader]

I've been plodding my way through the ins and outs of writing multi-instrument strategies w/ accompaning plots. Things work pretty much as expected with historical data, but often fail in real time. To help sort things out, I've created a very simple strategy for attachment to any 1-minute chart. This strategy adds 60 stocks (via "Add"), and has CalculateOnBarClose = true so that it should be called 61 times each minute, once for each stock, and once for plot's primary instrument.

Each time OnBarUpdate (OBU) is called for one of the 60 stocks, the strategy increments an IntSeries count to track progress, and updates an associated indicator. With historical data, this indicator displays "60" within regular trading hours (9:30 - 4:00 PM ET). Based on my experience with similar strategies, this will stop working properly in real time, which I'll return and comment on after the market opens in the morning.

I've attached a "Test60Instruments" NinjaScript package containing the script and indicator, and a screen shot of a 1-minute ER2 window with the strategy and indicator attached. If one of the Ninja folks could load this up and try it out that would be great. I'll try to use it in this thread to illustrate the types of problems I'm having, and will review some of these in the next post. Thanks!

My goal here is to get this working, one way or another (i.e., to have the indicator count to 60 near the close of each minute in realtime, verifying that all stock data is up-to-date).

[NinjaTrader_Dierk]

To save effort on your and our end I suggest providing a sample as lean as possible. If you experience issues it would be great if you could demonstrate with just a 2 or 3 instruments. Thanks in advance.

Also: There is no guarantee that OnBarUpdate would get triggered each and every minute for each stock, since there might not have been a tick for this stock in this particular minute. Thus there won't be a bar for which OnBarUpdate could get triggered.

[greentrader]

preliminary notes

Some of the issues I've run into...

- It's a nuisance having Strategies removed from charts on disconnection, as others have noted.

- I had trouble trying to modify an Indicator's (non-bar) DataSeries from within a Strategy. The solution was to keep all data and calc.s within the Strategy, and only use Indicators for display purposes.

- Josh recommended at one point using FirstTickOfBar as a way of detecting the previous bar's closing when CalculateOnBarClose = false. But this is still tricky since, when FirstTickOfBar is first seen, one still doesn't know what the state of all the other (60) symbols is, and thus doesn't know which bar should be referred to for the latest Close value. In other words, for the other symbols, the close you're looking for may be in either the [0] or the [1] bar, depending on whether the last minute's bar has been closed or not for that symbol.

- As mentioned in previous post, things pretty much work as expected with historical data, but not in real time. The biggest problem in real time is that OnBarUpdate stops being called for all symbols. Hopefully this problem will show up again with the Test60 strategy in the morning...

- Q. Does it matter what the setting of CalculateOnBarClose is for an indicator associated with a Strategy? Or does the Stategy setting take precedence?

[greentrader]

To save effort on your and our end I suggest providing a sample as lean as possible. If you experience issues it would be great if you could demonstrate with just a 2 or 3 instruments. Thanks in advance.

Also: There is no guarantee that OnBarUpdate would get triggered each and every minute for each stock, since there might not have been a tick for this stock in this particular minute. Thus there won't be a bar for which OnBarUpdate could get triggered.

Good points. If you load the Test60 Strategy you'll see that it is extremely lean, as is the indicator. To test with fewer stocks, simply comment out some of the "Add" calls. There's no dependency on "60" in the code.

The stocks added are the top 60 S&P. They'll have plenty of action during regular trading hours.

[NinjaTrader_Dierk]

>> Or does the Stategy setting take precedence?
yes

>> Test60 Strategy you'll see that it is extremely lean
My understanding of lean is 2 or 3 instruments not 60.

Also: You can add a index, you just can't trade it. Please make sure to apply the proper name like "^COMP".

[greentrader]

>> Test60 Strategy you'll see that it is extremely lean
My understanding of lean is 2 or 3 instruments not 60.

All my research to this point has been with 2 or 3 instruments. I just sent the 60 instrument Strategy so we could play around with the number by commenting out "Add"s, and because I need that number to (eventually) work. However, the code in the strategy is bare bones.

-----------
Darn! I don't know how I missed the need to use "^" when specifying indices in "Add" calls. Sure enough, if you add this to the strategy's Add list it counts to 61 (but not if you use "DJIA") ... Thanks Dierk.

Add("^DJIA", BarsPeriod.Id, BarsPeriod.Value);

[greentrader]

first results

Attached is first Plot60 chart of day.

My interpretation is that all 60 close calls get made, but my counting IntSeries is attached to the primary symbol, and so, near the end of each minute, some of the closes get added to the old minute and some to the new minute counter, thus resulting in some counts being > 60 and some < 60. (The back and forth count, averaging to 60.)

Will have to try something more clever for tracking the closes.

-----------
Another problem I'm going to have with this approach to grabbing minute bar data is that it's taking 5 to 10 seconds for all the closes to show up at the end of each minute - presumably an artifact of the way in which NT doesn't consider a bar closed until a trade occurs in following bar. Looks like I'll need to just build the bars myself (when in realtime) based on time of day.

[greentrader]

conclusions

Before calling it a day, thought I'd summarize where I'm at with respect to the use of multi-instrument strategies after working on this for several days...

I'm basically writing 2 different kinds of multi-instrument strategies: one type (like that shown earlier in this thread) relies on acquiring minute bars for multiple instruments, while the other combines results in realtime from multiple instruments into more complex plots on a second by second basis.

Strategy Type 1. The conclusion I've come to about acquiring minute bars from NT is that it's easy to do with historical data, but not-so-easy in realtime, especially if I'm more interested in speed than perfection. So what I'm going to do is to use NT's historical minute bar data when not in realtime, and construct my own minute bars when in realtime.

Strategy Type 2. With respect to the second type of strategy (second by second plotting), I determined that attempts to use various tricks to make this more efficient ended up making the results less reliable, with historical plots often looking different from realtime plots. In this case the best approach turned out to be to simply react to every call to OnBarUpdate, using the "Closes[n][0]" form to get Close data, and continuously update the plot (indicator). This simple approach gave the best correspondence between historical plotting and realtime plotting, with one exception: when such a strategy is first loaded, the plotted values are distorted because it appears that the symbol data is processed out-of-sync (i.e., data from different bar times gets combined). However, reloading the NinjaScript will correct the plot, which then agrees with realtime plotting.

Good enough! Thanks to all those who have responded with help and insights during this past week. I think NT is flexible enough to do what I need it to do. And although it makes working with "bars" more difficult when compared to a program like Tradestation, it also doesn't have many of Tradestation's limitations (and am hoping the next version of NT does better at competing with some of the charting features of TS). It's particularly satisfying working in .NET, where the resulting code is fast, and where one is far less bounded than in something like TS's EasyLanguage.

[greentrader]

script display bug example

In my last post, I described a strategy indicator display bug that occurs with my multi-instrument strategies when CalculateOnBarClose = false:

"when such a strategy is first loaded, the plotted values are distorted because it appears that the symbol data is processed out-of-sync"

The attached image shows this effect. The first is what the graph looks like when the strategy is first loaded (noisy), and the second is the same graph after reloading its NinjaScript (smoother). The second also corresponds to what the graph looks like in real time, so I assume that's the "correct" one, and that the difference is due to an error in how data is being passed to OnBarUpdate when a script is first attached to a chart. (Also, based on the nature of the data, I know that the smoother chart is the more accurate one.)

I thought that it might be useful to document this now before making my stuff too complex. If a Ninja person is interested in pursuing this, I can create an even simpler strategy to illustrate the problem (it basically just involves adding a single symbol that gets plotted as some function of the primary symbol). I can get around this now by simply reloading the NinjaScript when starting up, but I guess this "bug" would be a bigger problem for someone who is running a strategy that trades.

[NinjaTrader_JoshP]

The simplest way is for you to just make a simple print statement from each time frame. Do something like Print(Time[0] + " BarsInProgress: " + BarsInProgress + " Close: " + Close[0]);
Then you can understand the nature in which the order of bars are updated in historical and in realtime.

[greentrader]

the plot thickens

The simplest way is for you to just make a simple print statement from each time frame. Do something like Print(Time[0] + " BarsInProgress: " + BarsInProgress + " Close: " + Close[0]);
Then you can understand the nature in which the order of bars are updated in historical and in realtime.

Good idea. I'm working my way up to using the .NET debugger with NT, but simple Print statements are probably easier to deal with in this case.

However, I discovered another, related bug which more clearly shows that there is something wrong when strategies are first loaded into an existing chart: If the strategy has an attached instrument that is to be displayed as an indicator in another panel, then this panel is not displayed when the strategy is first loaded. This, again, is fixed if NinjaScript is reloaded (i.e., the indicator magically appears). Hopefully, on your end, these two bugs are related, and finding out why the second panel is not displayed at all will lead you to why the OnBarUpdate calls screw up the added instrument data when the strategy is first loaded.

Anyway, on a positive note, NT is looking a lot less constrained than I first thought it might be. Am now able to add multiple symbols displayed in multiple panels. One interesting limitation (unless I'm missing something) is that I can't create bars in indicators like I can in Tradestation. In other words, I can't specify that one indicator value should be the high, another the low, etc., of a bar, to create custom bars. That would be a useful thing to add in a later NT version. (There are a bunch of charting improvements like that that I could list, but many of these you guys seem to be already working on based on other posts.)

------------

Q. In realtime, does "Display update interval" (chart refresh rate) affect how often OnBarUpdate is called?

[NinjaTrader_JoshP]

Do you have a simple as possible reference you could attach demonstrating what you have found? If you could kindly attach them we would greatly appreciate it.

Your "bars" inside indicator is doable if I am understanding you correctly. Please check out the Help Guide for DataSeries.

"Display update interval" does not influence OnBarUpdate. If you are using CalculateOnBarClose set to false OnBarUpdate will update every tick. If you are using CalculateOnBarClose set to true OnBarUpdate will update at the close of each bar.

[greentrader]

custom bars

Do you have a simple as possible reference you could attach demonstrating what you have found? If you could kindly attach them we would greatly appreciate it.

Your "bars" inside indicator is doable if I am understanding you correctly. Please check out the Help Guide for DataSeries.

Okay, I'll put together a very simple example...

With respect to custom bars, see the attached image from a TradeStation dialog window. The indicator "CmpTick" has 10 "plots", one of which you see being assigned to "Bar High". If you then proceed to assign other plot values to "Bar Low", etc., then TS will plot these separate plot values as a single bar. In other words, they support custom bars as a form of plot line formatting. That's what I think NT is missing. BTW, a big limitation they have is that you can only create one such custom bar per indicator, so you'd be ahead of the game if you supported more.

I took a quick look again at the DataSeries doc.s, but I don't see how one can convert two or more custom DataSeries into a single bar for display - but maybe I'm missing something...

[greentrader]

bug progress

Find attached a couple more screen shots showing the Strategy loading bug. I was fortunate to have a case occur where the indicator panels got opened but the "Add"ed instrument data was still in error when the Strategy was first loaded.

The screen fragment on the left shows that the added instrument data gets stuck on 1 minute values instead of tick by tick values when the Strategy is first loaded. Upon reloading, the tick by tick values show up and the plot looks like it does in real time.

Since this strategy is written to plot everything on each call to OnBarUpdate, then any missing calls to OnBarUpdate will show up as a flat line segment, as seen on the left indicator panels, where the length of these segments corresponds to exactly 60 seconds (it's a 5-second bar chart). (The smaller flat segments on the bottom, right reflect the known delay in this index's data reporting.)

So now we know what's wrong with the "Add"ed instrument data being passed to OnBarUpdate when a Strategy is first loaded into a sub-minute chart. It looks like OnBarUpdate is being fed minute bar closes rather than tick by tick data for the added instruments.

[NinjaTrader_JoshP]

For your custom bars you can definitely make your own. Please check out the Heiken Ashi indicator. That one builds its own bars for display.

Do you have a simple as possible strategy to correspond with the screenshots? Thanks.