Tick Filter Failed (Real Time & Historical)

[wayneFH]

There's a wild spike on the 3/20/08 USD/CHF charts that looks like a bad tick. At first I thought tick filtering failed because it hit my StopLoss (in Strategy Analyzier) for 119 pips of loss. (I use NT 6.5.1000.1)

However, I did some some more investigation. Here's the symptoms...then I have several questions:

With the chart set to 1 tick it shows there's a burst of a lot of ticks in a row that are 119 to 120 pips lower than the last market price. Then it jumps back instantly to the previous market price.

This all happens in less than one second!

Below you can see the Output window of print statements where it shows the BarsInProgress and Close[0] for the entire first minute. The strategy has 5 minute data for the primary bars and 1 tick for secondary bars.

So tick filtering failed because it assumes bad ticks only come 1 or 2 at a time. It appears more realistic they can last several milliseconds

Here's the data logged showing all the ticks within the 1st minute of 8 am.

The forum denied the amount of data here as too big for the post --greater than 10,000 bytes. Anyway, just trust me on this. Or I can post it as an attachment.

Questions:


1. Any opinion of whether a huge burst of "out of band" data that lasts only milliseconds (less than a second) can be good data? It certainly goes against any common sense.

2. On printing the data above, I tried showing Time[0].Millisecond but it only returned 0 for every one. Does that mean NinjaTrader never stores the milliseconds with tick data?

3. Maybe the built in tick filter should be enhanced in the future to allow a configuration of how many bad ticks in a row become considered "good" or a configurable millisecond timer. Otherwise, at least if they all occur in the same second. Something more flexible to cover situations like this.

4. How to handle bad ticks in live real time production? It appears that I can simply write my own tick filter since my strategy ignores the BarsInProgress = 0, 5 minute data and only trades using the BarsInProgress = 1, or 1 tick data. I can ignore ticks I don't like.

5. But what about stops in historical? I assume I have to write my own custom fill strategy right? Will it work if I add in the tick filter with a timer or whatever I want?

6. And what about stops in real time? Since this data is from eSignal, I assume the data from the broker, MB Trading, will instead be accurate? Will I ever have a "real live" stop get hit with a weird 119 pip loss on spurious ticks like this? Any thoughts are appreciated, I don't expect any guarantees.

Sincerely,
Wayne

[NinjaTrader_Ray]

1) Yes, it can be good data and likely the case.
2) I don't know 100% but if you get zero then I guess this is the case.
3) Thanks for the suggestion, will consider for future.
4) Yes, you could run your own tick filter to control OnBarUpdate() but the underlying series will still contain these ticks
5) Correct.
6) That is broker dependant for sure.

[wayneFH]

Evidence they probably are bad ticks?

Ray. Hey, guess what? This data never appears on minute charts. On minute charts the low for that minute is: 1.0129

But a 60 second charts shows the low for that second is: 0.9994

Doesn't that lead you to believe those ticks are bad?

I can't believe them. A 119 pip spike without any ticks in between is way out of reality. Even on the dates when major news came out and prices really DID spike over 100 pips, there was still at least a few ticks in between, then pips jumped around at the extremes before gradually fading back.

In this case there was zero in between ticks either coming or going. It instantly jumps 119 pips and then instantly jumps back 119 pips all in less than one second.

It appears eSignal probably corrected the minute data and didn't bother to correct the tick data.

Sincerely,
Wayne

[NinjaTrader_Ray]

Those ticks could be bad but the context of my answer was relative to the tick filter option. When I say that they are good I meant that the tick filter works as expected and does not filter the data out.

To keep it simple -->

- Incoming tick outside of range --> Filtered out
- Next incoming tick is also outside of range --> Filtered out
- Next incoming tick is also outside of range --> This is now a valid tick

In the above sequence if the second or third tick was inside range, the counter is reset.

I am not saying a tick is good or bad, I am just saying this is how ticks are filtered or not in real-time.

[wayneFH]

Okay. Thanks for clarifying. Obviously, based on the evidence below, it appears that the rules: 3 "bad" ticks equals one good tick a overly simplistic.

It seems to work as designed. I only say it "failed" in the sense that some bad ticks obviously got through that algorithm. So hopefully it can be improved.

Aren't there some good tick filtering algorithms out there in the common domain?

I was just reading up on the PDF for the Olsen filter. Wow. Filtering is a lot more complex that I imagined.

http://www.olsendata.com/index.html?filter

Wayne

[ferdinand]

Hi,

1) I have a difference in the start and the close of the bar. this is a common issue.

2) I have a fake occassional data tick, it is not much, 2 or 3 ticks in stocks. it is annoying.

3) sometimes I have huge tails in price that I know for experience they are fake. like 120 ticks.

I am a price action trader, I need an accurate data, I don´t use other indicator that price.


I use NT 7. load 5 five charts of ES, one spy and load differents days on the charts, from 50 in 5 min TF to 365 in daily TF.

How can I solve this issue?

[NinjaTrader_Bertrand]

Ferdinand, could you please open a new thread for your issue in our Charting section here?

Thanks,

[AnotherTrader]

The following link came up on another site today (EliteTrader), which might be useful for you to get ticks cleaned. ....

http://tickmaster.webs.com

Is Tickmaster an NT partner/consultant?

[NinjaTrader_Bertrand]

Thanks for sharing with us, they are not an official partner at this time but we will check into their services and product.