60 Instrument Test Update

[greentrader]

For those who followed the "60 Instrument Test" thread, here's an update...

- In general, I have no problems now loading or working with 60 or more instruments in a Strategy. These instruments can be used in calculations and plots in any combination (although I haven't yet tried to plot custom bars like those in the Heiken Ashi indicator).

- Loading of the 60+ symbols is a bit clumsy-looking given the way that NT keeps opening and closing an alert window. This should probably be replaced by messages in the Control Center window. (Also, I'm puzzled that loading is often accompanied by a "cannot be cancelled" message, since it seems that NT already has to deal with missing data, so why does it care if the user cancels a data (re)load? The user should be able to freely cancel (re)loads, and manually reload when necessary.)

- The most important "trick" for me to working with multiple instruments was to realize the value of the "Time" stamp of each bar. This is the only way to tell if bars from different instruments belong together, both in real time and with historical data. For example, even in the simple case of just 2 instruments (as in some of NT's own examples), one shouldn't be combining them in a calculation or comparison without doing a check such as,

If (Times[0][0] == Times[1][0]) ...

That's because OnBarUpdate doesn't wait for instruments to get time-synchonized in any way, either with historical or realtime data, so you never know what the status of data is without examining bar times (i.e., a "[0]" bar might be the current minute for one instrument, but be pointing to the previous minute for another, or an even earlier minute if no trade has occurred for more than a minute). However, examination of bar "Time" (via the "Times" array) always gives you enough information to see what the status is of each instrument.

- With respect to the problem I had with not wanting to wait for the first trade to occur in a new minute before seeing the close of the previous minute (a long wait in a slow market!), the solution was to set CalculateOnBarClose = false, use some other criteria to determine when the bar was closed (such as clock time), and then grab whatever data for each instrument was available at that time from bars stamped with the appropriate time. This requires searching recent bars for each instrument to find the proper bar, and dealing with any missing bars, but at least it's doable and fast.

- Within respect to CPU time (performance), the only hit I see (other than initial load time) is with complex chart drawing. Routines that simply process data from large numbers of instruments in the background do not appear to be a problem.

[NinjaTrader_Ray]

Thanks for the useful comments.

We provide a cancel reload button where data providers suppor aborting a historical data request...Most don't.

[greentrader]

Thanks for the useful comments.

We provide a cancel reload button where data providers suppor aborting a historical data request...Most don't.

Hi Ray. Sorry if I'm missing something, but if an NT user cancels data locally, why not just ignore any further results from the data provider (for that request)? Why put the entire NT app at the mercy of a (potentially) flaky data provider?

[NinjaTrader_Ray]

I don't think we are on the same page.

Are you referring to somehow aborting a single data set request or...aborting a bunch of queued requests for multiple data sets such as a multi-instrument strategy?

[greentrader]

I don't think we are on the same page.

Are you referring to somehow aborting a single data set request or...aborting a bunch of queued requests for multiple data sets such as a multi-instrument strategy?

Both cases. Why should the local state of data acquisition within NT be subject to the whims of a data provider? If the request is unreasonably delayed, we should be able to abort it locally and get on with other things, even if the request can't be undone at the data provider.

I guess its mainly the modal alert window thing that doesn't feel right (vs. having data acquistion be more of a background activity). Or at least the user should always be able to abort any modal mode they are put in.

In TradeStation, they just erase the chart window and draw a message on it saying something about the status of the data. So at least that way any delay becomes a property of the affected window(s) only, and you don't have a pesky alert window that keeps popping up.

Another example: I have a custom trading app that makes requests for data from various sources all the time, but it never puts up an alert telling me that a request cannot be aborted. That would just interfere with all the other things the app is doing. And failure of one data request isn't going to break the app or make the rest of what it's doing useless.

I can see how NT evolved into the current approach when people were using few symbols, but with many more symbols the current (data loading) interface seems a bit clumsy and unforgiving. imho

[NinjaTrader_Ray]

This is likely a moot discussion for NT7. See image.

[greentrader]

Excellent.

Here's another pesky data-loading nuisance (unless I've missed something) ...

When working with small time frame bars (1 minute or less), and lots of symbols, it pays to restrict the total amount of data loaded (i.e., to minimize the "days back"). The problem is that "days back" is currently including non-trading days such as weekends (even if "ignore weekends" is chosen where possible), so you end up having to always load at least 2 or 3 extra days of data to deal with weekends and holidays.

On some days, this just about doubles the amount of data that I'm loading, which seems like a waste of time and resources. Perhaps the next version can be made a bit smarter about this, and provide an option for truly ignoring weekends and holidays when counting "days back".

(Example: Today, Wednesday before the open, I have a chart with 5 days back, so it has loaded last Friday, Monday, and Tuesday. But I really only need 2 or 3 days back, but that wouldn't load enough data early in the week since weekend days are getting counted as days back.)

[NinjaTrader_Dierk]

Thanks for your suggestion. We'll add it to the list of future considerations.

[greentrader]

slow strategy reloads

Am continuing to create strategies with many symbols and many associated indicators without any problems.

One performance issue that I wanted to mention again (hoping it gets fixed in next version) was the time it takes to reload a strategy with multiple instruments. With markets closed, after making a minor code change to a strategy, it can take quite a while for the strategy to be reloaded due to all the data reloading going on. This doesn't make a lot of sense (NT already has the data), and slows down the trial-and-error process. Seems like there should be some way of telling NT to forget about fetching new data when reloading a strategy (something like holding control key down while selecting reload ninjascript?) - or some reworking of how strategies are reloaded that eliminates this obstacle to development.

[NinjaTrader_Ray]

Data loading will be improved in NT7, thanks for the suggestion on excluding loading data from provider. Will consider that as well.

[greentrader]

time-consuming backups

Here's another performance issue related to multi-instrument strategies...

During the past month, there have been 2 times that NT has failed when trying to save current workspace and when trying to do a backup, resulting in loss of most work done after the previous backup. This has forced me to do a backup a couple of times each day, so that I don't lose more than 1/2 days work.

This would not be such a problem, except for the fact that one must be disconnected when doing a backup, which then unloads the multi-instrument strategies, and results in requiring a very long time to reload everything after the backup is completed (as reported in previous post).

I'll try to convey more info next time as to why NT thinks it can't do a backup, but I can say that it is not due to the current NT configuration, since if I put everything back together again it will do the backup. So it seems to have some problem related to the sequence in which changes occurred.

Anyway, it would be helpful if, in the next version, I could at least get backups done without having to unload all the strategies and accompanying data.

[NinjaTrader_Ray]

You could Save the workspaces via File > Workspaces. Next time you have this problem, let us know right away.

[greentrader]

You could Save the workspaces via File > Workspaces. Next time you have this problem, let us know right away.

Yup. I do that regularly. Last night, however, that failed first followed by a failure on the backup. Then, after quitting, the workspace was missing, and I had to go back to the latest backup to recover it. So the only reliable recovery seems to be via a full (uniquely named) backup. Definitely a bug in here somewhere, but it happens so rarely that I haven't yet identified a pattern, and the identical configuration will successfully back up the next time.

Will be careful next time to send whatever diagnostic info is available... Thanks.

[NinjaTrader_Ray]

Great, much appreciated.

[greentrader]

DrawLine for other instruments?

Something I've been working up to was the ability to draw price bars for other instruments within indicator panels using DrawLine. (For those interested, as noted elsewhere on the forum, the first thing necessary to do this is to set DrawOnPricePanel = false so that the indicator doing the drawing does its DrawLines in the indicator panel.)

The next issue is getting (non-primary) price data from the strategy to the indicator (i.e., H, L, C data). For the primary instrument, these values can be referenced directly, such as "Low[0]" for the current bar. But for other instruments, I assume the data must be passed in some way to the indicator.

When working with historical data, I find that I can simply use "Set" within the stategy to set values that can then be used by the indicator:

myInd.Values[0].Set(Highs[1][0]);
myInd.Values[1].Set(Lows[1][0]);

And in the indicator's OnBarUpdate:

DrawLine("xx" + CurrentBar.ToString(), 0, Values[1][0], 0, Values[0][0], Color.Black, DashStyle.Solid, 1);

(where 2 transparent plots have been added to the indicator to support the two "Values" arrays)

However, in real time, nothing gets drawn by DrawLine, apparently because Values[1][0] and Values[0][0] do not contain valid values for the current bar, even though the Strategy has set them. If I look one bar back (Values[1][1] and Values[0][1]), then values are present, but for the previous bar. It also doesn't help to have the Strategy call myInd.Update(). (The Strategy, btw, has CalculateOnBarClose = true.)

A similar problem came up a couple of weeks ago, when I tried having a Strategy set the current values in a custom DataSeries owned by an indicator - the DataSeries never got updated in real time.

I'm sure I could find a kludgy way to work around this, but it seems like I'm just missing some small thing (hopefully not too embarassing).

-----------------
Simple solution appears to be:

if (Historical)
DrawLine(..., 0, Values[1][0], 0, Values[0][0], ...);
else
DrawLine(..., 1, Values[1][1], 1, Values[0][1], ...);

because (I guess) with CalculateOnBarClose = true, in real time the indicator and the strategy are out-of-sync with respect to the "current" bar they are dealing with. Am continuing to investigate...