BidAsk Historical...

[paulg]

The only reason I am not using NT6.5 (and it looks like I will not be using NT7.0 until this is addressed) is that it is imperative for me (and a whole load of approaches and indicators) to sequence this data properly so that it can be processed from the database in the same order as it was received in real time.

It was one of the few exciting promises months ago.... now we're back to step... 6.5... making also any form of backtesting that relies on those information substantially unreliable.

[NickA]

Thanks for sharing your thoughts. I do realize that there is demand for this and this is on our list for the future. Unfortunately, this is not something we can do for NT7 at this time since technically, there is a world of impllications in doing so and we are too far along to introduce anything that would put as back at square one in terms of a beta process.

Thanks for your reply Ray. I appreciate that you must have lots of pressure to deliver NT7.0 and that this would need some architectural change to the database. I hope it is something that your developers can bear in mind so that it does not become even more difficult to implement at a later stage.

There might be interim solutions that would rely on additions to classes this would have less far reaching implications than changes. I do realise that it would still have some implications.

An example would be to add a couple of flags to each entry in the trade database indicating that a trade occurred at the best bid or the best ask. These could be returned through new methods or extensions to existing methods. Forgive me if I have my terminology wrong I am not an OO programmer.

Anyway I do appreciate the constraints that you guys must be operating under as open beta and release looms ever closer. (Having said that there is an argument for change sooner rather than later as it is likely to be harder to make these sorts of changes after release).

Cheers.

[NinjaTrader_Ray]

Thanks for sharing your thoughts. I do realize that there is demand for this and this is on our list for the future. Unfortunately, this is not something we can do for NT7 at this time since technically, there is a world of impllications in doing so and we are too far along to introduce anything that would put as back at square one in terms of a beta process.

[NickA]

Hi NT guys,

I'd like to add my 2 cents in support of this request to accurately sequence historical bid/ask/last data. You only have to look at Gomi's thread and indicators to see there is massive demand for this feature and these sorts of indicators.

The only reason I am not using NT6.5 (and it looks like I will not be using NT7.0 until this is addressed) is that it is imperative for me (and a whole load of approaches and indicators) to sequence this data properly so that it can be processed from the database in the same order as it was received in real time.

These ticks are received in certain sequence (and must be stored in the database ina certain sequence), scripts need to be able to retrieve them in that same sequence. Whether that is achieved by time stamping or sequence number or some other method is not so important.

I am not sure why NT chose to separate the 3 types of data? Globex sends them as a single data stream with a flag to signify that a tick is a bid or ask change or an actual print. The data stream is inherently sequenced so it does not matter if you get 5000 ticks with the same time stamp, they are recieved in order. Of course a data providers might mess with this but I am unaware of it if they do. (Other charting platforms can accurately sequence with a variety of data venfors so I would be surprised if this was the case).

Anyway a heartfelt plea to re consider the priority of this problem.

[NinjaTrader_Dierk]

Thanks for your feedback.

[warrior]

NT7 disappointment

Dierk and others: I couldn't agree more with feedback given by darthtrader, dnoff, funk101, rt-trader, gomi etc. etc. It seems that this major feature request that all along has been promised with NT7 where historical bid/ask volume/price data fix so that indicators that rely on it such as volume profile, gomcd would work without having to run these indicators 24x7 collecting data themselves in files, was dropped. I even remember somewhere in the feature list that such data would be collected on NT servers for past couple of weeks so that indicators could get to them. Despite all the feedback and frustration voiced on this thread, looks like it is not a priority for you. I understand you have a major version uplift with NT7, but dropping this imp fix is a major disappointment leaving NT7 still a major step behind competition. Hoping you/RayD reconsider this for NT7.

[NinjaTrader_Dierk]

Unfortunately other priority issues are still on our plate (as Ray pointed out). We will get back to that item as soon as possible.

[rt-trader]

A question to Ray...

Back in post 17 of this thread you mentioned that reference sample which deals with handling historical tick data may be forthcoming when bandwidth allows.

I have spent many hours attempting (unsuccessfully) to properly use this capability (because I havent been able to find an accurate way to sync Series0 data start with Series1 data).

My problem (to this point at least) has been that there appears to be no current way to accurately fix the start point of the opening bar of the chart when loading and processing the dataseries up to the point of !Historical.

So Ray - would it be possible to for you to outline what the reference sample(s) or any planned methods might facilitate when they become available.

I believe many others would be interested to know this as well.

Thankyou...

[dnoff]

Dierk, no offensive but I can't disagree more. What you posted makes sense from a software engineering standpoint of getting things EXACTLY right. From a trading standpoint though, the exact sequence would only matter to someone trying to develope a real high frequency strategy system in Ninja...
Anyone with any real knowledge of high frequency strategies is not going to bother with Ninja anyway.
With this design decision though you guys are throwing out the baby with the bathwater..
For alot of people, I imagine what seperates Ninja from other software is onmarketdata()...It strikes me as obvious that a less than optimal historic onmarketdata solution would have vastly higher utility than no solution at all..
A great example is Volume profile..while I simply love your software..its hard to "pimp" to my market profile type trading friends when you have to caveat how great volume profile is with that if anything happens with your computer, you lose all your information...or if you wake up at 11am to get ready for an afternoon session, you should have set your alarm earlier to start collecting data pre cash open...its absurd from a trading standpoint.

Darthtrader - thanks for writing this. This is my thoughts also. I also use something similar to market profile (a custom developed variant but a similar concept in analyzing volume and price trader relative the market bid/ask that the exact point in time of the sale).

For me, I also find it hard that once I have loaded the charts my indicator in (amongst some others) that if there are any changes, I lose all of the indicator values for the session). I also am not able to backtest and validate anything, except for using market replay.

So I what I think would be helpful for me, like the volume profile, is an historical onmarketdata or a "lite" version of the bid/ask data that is synchronized with each tick

[NinjaTrader_Dierk]

Most of the requests we got in that area or from people how wanted to maintain the EXACT sequence.

Not sure why you feel unable to restore your volume profiles on restarting your custom indicators.

[darthtrader]

-> asking for finer granularity on the timestamp is the wrong approach. The only approach which really would resolve the issue would be sequence counter. We held off on implementing this with NT7, since the ressource investment would have delayed NT7 much further than the majority of our users would be willing to accept.

Dierk, no offensive but I can't disagree more. What you posted makes sense from a software engineering standpoint of getting things EXACTLY right. From a trading standpoint though, the exact sequence would only matter to someone trying to develope a real high frequency strategy system in Ninja...
Anyone with any real knowledge of high frequency strategies is not going to bother with Ninja anyway.
With this design decision though you guys are throwing out the baby with the bathwater..
For alot of people, I imagine what seperates Ninja from other software is onmarketdata()...It strikes me as obvious that a less than optimal historic onmarketdata solution would have vastly higher utility than no solution at all..
A great example is Volume profile..while I simply love your software..its hard to "pimp" to my market profile type trading friends when you have to caveat how great volume profile is with that if anything happens with your computer, you lose all your information...or if you wake up at 11am to get ready for an afternoon session, you should have set your alarm earlier to start collecting data pre cash open...its absurd from a trading standpoint.

[NinjaTrader_Dierk]

- On our data server I believe you should see a couple of months worth of data.
- please contact your broker to clarify how far their data would go.

[trend]

Hi,
how far back the bid/ask data in the NT servers go?
Any guidance for the brokers on their bid/ask historical data
availability?

thanks,

[NinjaTrader_Dierk]

Guys, here is what you needed to understand: no whatever fine granularity would ensure that bid/ask/last could be replayed in the right sequence. There always could be bid/ask/last events having the exact same timestamp (as sent by the exchange) and then your trading system relying on the correct sequence would just fail.

-> asking for finer granularity on the timestamp is the wrong approach. The only approach which really would resolve the issue would be sequence counter. We held off on implementing this with NT7, since the ressource investment would have delayed NT7 much further than the majority of our users would be willing to accept.

[dnoff]

Not all of the data providers NT supports offers millisecond timestamping.

Thanks for the reply and this makes sense. However I would have thought as a design solution, could you have taken the opposite approach here. For example, I would include a timestamp that is down to the millisecond, and when a data provider does not support millisecond timestamps, I would add ".000" to the timestamp when it is received.

This way you are enabling a solution for those those data providers and traders wanting this. Today it seems that instead it is degraded to the lowest common denominator, you can support those data feeds that do. Then as user of the platform I can chose to go to those data vendors that support if it fits my trading needs??