Multi-Instrument bars not in chronological order

[kdoren]

I am working with multi-instrument indicators and strategies, mostly with historical data so far. I wrote an indicator to display in the output window what happens in real time, and I discovered a few cases where bars are processed out of chronological order, which I think is a bug.

I ran my KDTestSync indicator, set to 1 Min with CalculateOnBarClose = true, on YM and ES in the middle of the night when activity is low. There are some 1 min periods with no activity on instrument or the other.

Below, on line 2, Bar Series 1 closes its 1:19:00 AM bar. Then, on line 3, Bar Series 1 closes its 1:20:00 AM Bar. After that, on line 4, Bar Series 0 closes its 1:19:00 AM Bar. This has got to be a bug; if Bar Series 0 had any actvity during the 1:19:00 AM Bar, it should have been closed when the 1:19:00 AM Bar for Bar Series 1 was closed.

BarsInProgress=0: Times[0][0]: 12/18/2009 1:18:00 AM, Times[1][0]: 12/18/2009 1:18:00 AM, Closes[0][0]: 10302,  Closes[1][0]: 1098.5, Volumes[0][0]: 3, Volumes[1][0]: 95, BarsArray[0].TickCount: 1, BarsArray[1].TickCount: 2
BarsInProgress=1: Times[0][0]: 12/18/2009 1:18:00 AM, Times[1][0]: 12/18/2009 1:19:00 AM, Closes[0][0]: 10302,  Closes[1][0]: 1098.5, Volumes[0][0]: 3, Volumes[1][0]: 480, BarsArray[0].TickCount: 9, BarsArray[1].TickCount: 1
BarsInProgress=1: Times[0][0]: 12/18/2009 1:18:00 AM, Times[1][0]: 12/18/2009 1:20:00 AM, Closes[0][0]: 10302,  Closes[1][0]: 1098.5, Volumes[0][0]: 3, Volumes[1][0]: 28, BarsArray[0].TickCount: 9, BarsArray[1].TickCount: 1
BarsInProgress=0: Times[0][0]: 12/18/2009 1:19:00 AM, Times[1][0]: 12/18/2009 1:20:00 AM, Closes[0][0]: 10301,  Closes[1][0]: 1098.5, Volumes[0][0]: 15, Volumes[1][0]: 28, BarsArray[0].TickCount: 1, BarsArray[1].TickCount: 1
BarsInProgress=1: Times[0][0]: 12/18/2009 1:19:00 AM, Times[1][0]: 12/18/2009 1:21:00 AM, Closes[0][0]: 10301,  Closes[1][0]: 1098.25, Volumes[0][0]: 15, Volumes[1][0]: 23, BarsArray[0].TickCount: 13, BarsArray[1].TickCount: 1
BarsInProgress=0: Times[0][0]: 12/18/2009 1:21:00 AM, Times[1][0]: 12/18/2009 1:21:00 AM, Closes[0][0]: 10301,  Closes[1][0]: 1098.25, Volumes[0][0]: 67, Volumes[1][0]: 23, BarsArray[0].TickCount: 1, BarsArray[1].TickCount: 2

This behavior doesn't agree with my understanding of how NT7 is supposed to work, based on the following:

From the NT7 Code Breaking Changes document:
CurrentBar internal pointers are now first updated for all bar series with the same timestamp before OnBarUpdate() will be triggered. This ensures that accessing bars across multiple series will be in sync regardless of which BarsInProgress context you may be working out of.

It's already quite hard to write reliable multi-instrument indicators & strategies; if time can go backwards it will be even harder.

Thanks,
Kevin

[NinjaTrader_Ray]

Hi Kevin,

This is not a bug.

- We will re-write the text from the code breaking change document, sounds like this is misleading
- In real-time, multiple instrument series are truly independent
- A bar will close when on the 1st tick of the next time stamp comes in
- This means you can't sync bars on multi instrument series
- The only way one could do that is if you forced a bar close based on your local PC time --> The problem with this approach is that you are guarnteed to build incorrect bars since ticks that should have been in bar x can make it into bar x + 1

[NinjaTrader_Ray]

Kevin,

Here is some more information for further clarity.

o For historical, pointers for all bar series with the same timestamp will be updated before OnBarUpdate() will be triggered.
o For real-time, pointers for all bar series of the same instrument will be updated before OnBarUpdate() will be triggered.

[kdoren]

How do I force a bar to close? (I wouldn't use a local timer; when one bar closed I would force all open bars of the same timeframe to close)

Is there a risk of a race condition if I force a bar to close, while NT gets a trade and also tries to force it to close?

If there are no trades in the bar when I force it to close, will no bar be created? Or will an empty bar be created? If an empty bar will be created, can I detect this in advance and avoid forcing a close on that bar?

If, after I force a bar to close, a trade comes in which would otherwise have been in that bar, does it just get put in the next bar? Does the bar get put in the historical database that way?

Thanks,
Kevin

[NinjaTrader_JoshP]

Kevin,

You can't "force" a bar to close. I am not quite sure what you are after. The bars will close based on the bar type you are using and based on the NT logic for building that type of a chart.

[NinjaTrader_Ray]

Let me add to Josh's comments.

What I meant in my earlier post was that in order to have all same time bars in real-time close in sync --> This could only be accomplished if bars closed based on PC clock. This is not possible in NinjaTrader nor would it be accurate approach for the reason I provided in my original post.

[kdoren]

There is a lot I like about the NT platform, but it seems to me that you are so in love with the purity of your event-driven architecture that you are blind to the deficiencies it presents in the real world.

I first encountered this when I tried to write a routine that I could run at night to do end-of-day analysis. But amazingly, this was impossible, because daily bars don't close until trading starts the next morning. Other people have complained about this but you inexplicably defend it; you seem totally unable to see that this is a really basic and totally reasonable thing that some customers need to do, every other platform can do it, your platform has failed to make it possible, you haven't bothered to enable it, and you don't give your customers a way to enable it. You could easily enable this if you were able to recognize it as the architectural mistake that it is.

I'm now encountering a similar problem on a micro level. It's quite hard to make multi-instrument indicators & strategies run reliably even on historical data, due to the complexities of dealing with possible missing bars. Now what you are telling me is that it's almost impossible to do it in real time, because bars can come in not only out of order, but possibly very late and long after other bars of the same timeframe have closed.

You really shouldn't make it this hard. You should consider things like:

1) Let me work with unclosed bars. Yes, I know I can peek into a price data series with an index of [-1]. But that's not very useful unless I can also set and reference DataSeries objects with [-1], and you don't allow it. Please enable all data series to be referenced, set, etc., with an index of [-1]. This should be trivial for you to do.

2) Give me a way to force a bar to close. This is not just a multi-instrument issue. If I want to run an analysis on Saturday that uses Friday's daily data, it's just idiotic to make me wait until trading starts on Monday to do it.

3) Give me a way to create my own synthetic "Instrument" Bar Data Series that isn't synchronized to one of the existing Instruments. For example, if I could create an OHLCV data series, and then set and close a new bar for it only when I had synchronized bars from my other instruments, that would make life a lot easier. It would have fewer bars than the instruments being tracked, but the bars it had would only be synchronized bars where all instruments had data. (Such a feature would also be useful to other customers, i.e. people who wanted to plot candlesticks).

4) For real time, add an adjustable timer (defaulting to off) which would force a bar to close if no new ticks arrived during that time, where, say, minute bars could be set to close after the end of the minute if no ticks arrive in x seconds, or daily bars could be set to close after the end the session if no ticks arrived in xx hours.

5) For multi-instrument work, if there are several data series of the same timeframe, if a tick arrives which forces one bar to close, optionally close all other bars of the same timeframe. This would ensure that a bar for one instrument wouldn't arrive long after the same bar from another instrument.

These are a few things you could do, and I'm sure there are others which I haven't thought of. NT7 in its current form may be great for single-instrument daytraders (who, due to your business model, are probably most of your paying customers). But for multi-timeframe and especially multi-instrument work, there are some serious issues. Fortunately I think they can be fairly easily dealt with, but it will require you to realize that clinging to the purity of your event-driven architecture is a mistake that limits the usefulness of your product in some important ways.

Thanks,
Kevin

[NinjaTrader_Ray]

Kevin,

Thanks for sharing your thoughts.

1) I understand this issue and its on my personal list however, I have been able to get around by setting indicators to CalculateOnBarClose = false when doing analysis at night or on the weekends. Would this workaround not work for you?

On all of your other items, there is no strong stance that we are making. Simply, we chose the approach that closes a bar based on the time stamps we receive so that bars are truly built with accurate data. To appease your requirements, the suggestions you provided would work but as you know, will come at a cost that your bars likely will be built wrong and as historical data is reloaded, you will get for sure see the implications of this since the reloaded bars could very well be different. I do appreciate your comments and will take them into consideration.

[kdoren]

Ray, Thanks, that's a good suggestion, I should have thought of setting CalculateOnBarClose = false when running analysis on nights/weekend; I hadn't thought about that because it would really mess things up if I set it that way during trading hours so I have it default to true (it's a multi-instrument, multi-timeframe strategy). You should probably post a tip about it.

It still doesn't solve the other issues, so I appreciate your considering my suggestions. You should leave it up to your users to decide if they want to have the bars closed sooner at the risk that a tick might come in later that should have been included. But I for one would rather get good data sooner than possibly slightly better data some unknown time later.

Thanks,
Kevin