Getting erroneous charts.

[molecool]

In the past few days I keep getting strange erroneous candles/bars on charts running my strategy. Please take a look:

http://screencast.com/t/nTGVV1wSt

Here it is after F5/reload:

http://screencast.com/t/l7NMsSuDA

My strategy actually takes entries on those erroneous charts so it's a problem. My data feed is Kinetick by the way and the chart settings are on the first image.

Thanks,

Molecool

[NinjaTrader_Matthew]

Hello molecool,

There may be some ticks that come in off market in real-time which do not exist on the historical data server.

If you have not yet, you can employ the 'real-time tick filter' under Tools--> Options--> Data tab--> check "Filter bad ticks"

I'd suggest setting this to 50% off market.

With this set, should you receive a tick that is more than 50% away from the last traded price, it will not consider it a valid until you receive 3 consecutive ticks at that new price.

You may feel free to lower this % off market to meet your needs.

[molecool]

Hello molecool,

There may be some ticks that come in off market in real-time which do not exist on the historical data server.

If you have not yet, you can employ the 'real-time tick filter' under Tools--> Options--> Data tab--> check "Filter bad ticks"

I'd suggest setting this to 50% off market.

With this set, should you receive a tick that is more than 50% away from the last traded price, it will not consider it a valid until you receive 3 consecutive ticks at that new price.

You may feel free to lower this % off market to meet your needs.

I don't think this is caused bad ticks as it's happening ALL the time to me now. Here's one I took just a few hours later:



This is happening way too often to be caused by bad ticks. I have used Kinetick (and iQFeed) for years now and bad ticks are extremely rare. Something else is going on.

[NinjaTrader_Matthew]

It may be something else, but we cannot move to the next step until you have tried my suggestion as it is a necessary step. Please let me know after you have tried my suggestion.

[molecool]

It may be something else, but we cannot move to the next step until you have tried my suggestion as it is a necessary step. Please let me know after you have tried my suggestion.

Agreed - I did set that setting to 50% as suggested. I'll chime in here if it happens again.

[NinjaTrader_Matthew]

Thanks for the update - I'll keep an eye out on this on my end without the filter enabled and let you know what I see.

[molecool]

Just happened again!

Here we go - just happened again. I have both charts on a strategy but the one on the left is running without tick data underneath (which is what I thought may cause weird plots). Please see the enclosed chart.

[NinjaTrader_Matthew]

molecool,

Thanks for these screen shots.

Do the candles again appear normal after reloading?

Can you send me a screen shot of all your settings under Tools--> Options--> Data tab?

And to clarify, if Kinetick the only connection you're using? Do you connect directly through the Kinetick Adapter, or are you using the IQFeed client with your Kinetick account?

[molecool]

molecool,

Thanks for these screen shots.

Do the candles again appear normal after reloading?

Can you send me a screen shot of all your settings under Tools--> Options--> Data tab?

And to clarify, if Kinetick the only connection you're using? Do you connect directly through the Kinetick Adapter, or are you using the IQFeed client with your Kinetick account?

Here's my data settings:

http://screencast.com/t/RMmc88OnqtI

I connect directly via Kinetick only - I don't have any other data feed running.

[Harry]

NinjaTrader does not always correctly assemble the transition bar

I have had and still do have similar problems with other data feeds. This goes on for years, and in despite of numerous discussions, the problem has never been solved.

The main issue is the transition bar, which is the bar that NinjaTrader builds from both historical data and real-time data. NInjaTrader does not always build this bar properly.

all historical bars -> ok
transition bar (hybrid bar 1 historical tick + real-time ticks) -> sometimes produces spikes
real-time bars -> ok

Now, when I refresh via F5, the spike bar disappears, but the current bar on the chart may produce a new spike.

I have made the following observations:

(1) The direction of the artificial spike ALWAYS points in the direction of prior price action.
This is also confirmed by all the screen shots that have been posted in this thread.

(2) The problem does not occur if you connect to NinjaTrader and open a chart immediately. It occurs, if you connect and have a chart open for instrument 1, and then an hour or a few hours later

-> switch that chart to a new instrument 2
-> open a new chart for instrument 2

(3) The spike approximately represents the price action from the time when you have connected NinjaTrader or reloaded historical data and the time when you first open a chart for instrument 2

(4) I have also observed volume spikes for the hybrid bars.

(5) The problem does not always occur, but it will sometimes hit you and sometimes it won't. It is more frequent with some instruments than with others for unknown reasons.


How to deal with the problem:

Open a chart for an instrument. Wait until the bar close of the transition bar, which is built from a historical and the following real-time ticks. If you observe a price or volume spike switch the chart to a different instrument and then back again to the instrument that you wish to chart. The chart should now display correctly.

I have observed this bug on three different PCs with different data providers.

The problem is not linked to bad ticks from the data provider, but it is clearly a NinjaTrader problem, as NinjaTrader uses false real-time ticks (proof: direction of spike) when assembling the historical tick of the transition bar.

I am glad that somebody else comes up with the issue, as so far I could not persuade anybody that the bug exists and should be removed.

Just a note: NinjaTrader staff has been very helpful and effective to find solutions for all other problems that I have encountered. This is really the only significant nuisance, which I would like to have removed, and that is the reason that I have posted here.

[molecool]

I have had and still do have similar problems with other data feeds. This goes on for years, and in despite of numerous discussions, the problem has never been solved.

The main issue is the transition bar, which is the bar that NinjaTrader builds from both historical data and real-time data. NInjaTrader does not always build this bar properly.

all historical bars -> ok
transition bar (hybrid bar 1 historical tick + real-time ticks) -> sometimes produces spikes
real-time bars -> ok

Now, when I refresh via F5, the spike bar disappears, but the current bar on the chart may produce a new spike.

I have made the following observations:

(1) The direction of the artificial spike ALWAYS points in the direction of prior price action.
This is also confirmed by all the screen shots that have been posted in this thread.

(2) The problem does not occur if you connect to NinjaTrader and open a chart immediately. It occurs, if you connect and have a chart open for instrument 1, and then an hour or a few hours later

-> switch that chart to a new instrument 2
-> open a new chart for instrument 2

(3) The spike approximately represents the price action from the time when you have connected NinjaTrader or reloaded historical data and the time when you first open a chart for instrument 2

(4) I have also observed volume spikes for the hybrid bars.

(5) The problem does not always occur, but it will sometimes hit you and sometimes it won't. It is more frequent with some instruments than with others for unknown reasons.


How to deal with the problem:

Open a chart for an instrument. Wait until the bar close of the transition bar, which is built from a historical and the following real-time ticks. If you observe a price or volume spike switch the chart to a different instrument and then back again to the instrument that you wish to chart. The chart should now display correctly.

I have observed this bug on three different PCs with different data providers.

The problem is not linked to bad ticks from the data provider, but it is clearly a NinjaTrader problem, as NinjaTrader uses false real-time ticks (proof: direction of spike) when assembling the historical tick of the transition bar.

I am glad that somebody else comes up with the issue, as so far I could not persuade anybody that the bug exists and should be removed.

Just a note: NinjaTrader staff has been very helpful and effective to find solutions for all other problems that I have encountered. This is really the only significant nuisance, which I would like to have removed, and that is the reason that I have posted here.

Frankly speaking I wish I hadn't received your comment as this suggests that there may be an underlying data management problem. Truth be told however I have run NinjaTrader for about four years now and this is the first time I run into this particular issue. So let's see if they can come up with a solution.

Question: Have you ever tried to repair the database? I assume you have but just want to be sure. In order to fix some unrelated problem I have in the past resorted to manually removing all files in the db folder, basically giving NinjaTrader a fresh start in building its database. Of course you do lose all historical data this way.

You speak of a transition bar and I am not certain as what you are referring to. Are you talking about the open bar?

Also, are you running a strategy when the problem occurs? In my case I have one running on a 60-min chart which also loads a single tick series for executions. So I wonder if that may be what's causing the error.

The problem manifests itself on FX charts and of course reloading the chart resets the bar to its correct values.

[Harry]

Frankly speaking I wish I hadn't received your comment as this suggests that there may be an underlying data management problem. Truth be told however I have run NinjaTrader for about four years now and this is the first time I run into this particular issue. So let's see if they can come up with a solution.

Question: Have you ever tried to repair the database? I assume you have but just want to be sure. In order to fix some unrelated problem I have in the past resorted to manually removing all files in the db folder, basically giving NinjaTrader a fresh start in building its database. Of course you do lose all historical data this way.

You speak of a transition bar and I am not certain as what you are referring to. Are you talking about the open bar?

Also, are you running a strategy when the problem occurs? In my case I have one running on a 60-min chart which also loads a single tick series for executions. So I wonder if that may be what's causing the error.

The problem manifests itself on FX charts and of course reloading the chart resets the bar to its correct values.

Transition Bar: When you open your workspace and connect to your data feed, NinjaTrader will automatically backfill your chart with historical data. Each historical bar on a chart has a single tick.

Let us assume that you have a 5 minute chart and connect at 8:23 AM. In that case

- the bar with the time stamp 8:15 is a (single-tick) historical bar
- the bar with the time stamp 8:20 is a (single-tick) historical bar
- the bar with the time stamp 8:25 is a (multi-tick) transition bar
- the bar with the time stamp 8:30 is a (multi-tick) real-time bar

Let us have a look at the 8:25 AM bar. After connecting the backfill will cover the period from 8:20 to 8:23 (incomplete historical bar corresponding to a single tick) and NinjaTrader will then add all incoming ticks as supplied by your real-time data feed to that incomplete bar. That transition bar is therefore built from a single historical tick and real-time ticks which cover the remaining period.

There can be only one transition bar on your chart. All bars to the left of that bars are single-tick historical bars, all bars to the right of the transition bar are multi-tick real-time bars. If you open your chart exactly at 8:25 AM, there may be no transition bar, when the last historical bar is already complete.

If you refresh your chart via F5, then NinjaTrader will replace all multi-tick real-time bars on your chart with single-tick historical bars. At this stage, the last bar on your chart will become the new transition bar, as incoming real-time ticks are added to that bar. This explains that any spike that has been created on the previous transition bar will go away, when you refresh. However, you may observe a new spike for the last bar.

A spike is nothing else than a single false tick that has been added to the transition bar. You can observe that this tick is not always immediately added, when the incomplete historical bar is added to your chart, but the addition of this tick can be delayed a few seconds. We know that it is a historical tick that is being added as the spike always covers recent price action.

The spikes are related to the amount of real-time data handled by NinjaTrader, maybe it is a synchronisation issue or similar. What you can do

(a) disconnect and reconnect NinjaTrader -> all spikes should go away
(b) take the affected chart and toggle between two instruments -> the spike should go away

I would be interested to know, whether the behaviour on your side is similar as I have recently observed. The problem may depend on the architecture of the broker API or the datafeed adapter of NinjaTrader. It is not obvious that it will appear with all datafeeds.

[molecool]

Transition Bar: When you open your workspace and connect to your data feed, NinjaTrader will automatically backfill your chart with historical data. Each historical bar on a chart has a single tick.

Let us assume that you have a 5 minute chart and connect at 8:23 AM. In that case

- the bar with the time stamp 8:15 is a (single-tick) historical bar
- the bar with the time stamp 8:20 is a (single-tick) historical bar
- the bar with the time stamp 8:25 is a (multi-tick) transition bar
- the bar with the time stamp 8:30 is a (multi-tick) real-time bar

Let us have a look at the 8:25 AM bar. After connecting the backfill will cover the period from 8:20 to 8:23 (incomplete historical bar corresponding to a single tick) and NinjaTrader will then add all incoming ticks as supplied by your real-time data feed to that incomplete bar. That transition bar is therefore built from a single historical tick and real-time ticks which cover the remaining period.

By the way, I'm connecting via Kinetick - I'm going to try using the iQFeed adapter instead to see if I can replicate it like that.

There can be only one transition bar on your chart. All bars to the left of that bars are single-tick historical bars, all bars to the right of the transition bar are multi-tick real-time bars. If you open your chart exactly at 8:25 AM, there may be no transition bar, when the last historical bar is already complete.

If you refresh your chart via F5, then NinjaTrader will replace all multi-tick real-time bars on your chart with single-tick historical bars. At this stage, the last bar on your chart will become the new transition bar, as incoming real-time ticks are added to that bar. This explains that any spike that has been created on the previous transition bar will go away, when you refresh. However, you may observe a new spike for the last bar.

A spike is nothing else than a single false tick that has been added to the transition bar. You can observe that this tick is not always immediately added, when the incomplete historical bar is added to your chart, but the addition of this tick can be delayed a few seconds. We know that it is a historical tick that is being added as the spike always covers recent price action.

The spikes are related to the amount of real-time data handled by NinjaTrader, maybe it is a synchronisation issue or similar. What you can do

(a) disconnect and reconnect NinjaTrader -> all spikes should go away
(b) take the affected chart and toggle between two instruments -> the spike should go away

I would be interested to know, whether the behaviour on your side is similar as I have recently observed. The problem may depend on the architecture of the broker API or the datafeed adapter of NinjaTrader. It is not obvious that it will appear with all datafeeds.

I confess I'm a bit confused by your description. Historical bars don't just contain one single tick - they have an open, a low, a high, and a close. A tick by definition is a single value. I do roughly understand what are you trying to explain about the transition bar. If I open a chart then the open one is the transition bar as it'll have to load OHLC and then start adding ticks from there.

I suspect that it is a synchronization or latency error. In my case I'm connecting from Spain and although my connection is very fast the latency could introduce some type of problem. When I was trading out of CA I never experienced this particular issue.

[koganam]

I confess I'm a bit confused by your description. Historical bars don't just contain one single tick - they have an open, a low, a high, and a close. A tick by definition is a single value. I do roughly understand what are you trying to explain about the transition bar. If I open a chart then the open one is the transition bar as it'll have to load OHLC and then start adding ticks from there.

I suspect that it is a synchronization or latency error. In my case I'm connecting from Spain and although my connection is very fast the latency could introduce some type of problem. When I was trading out of CA I never experienced this particular issue.

Different meanings of the word "tick".

1. "Tick" as the event that causes a bar to be drawn or modified. If historical or realtime with COBC = false, there will be only one of these per bar. If realtime, then each incoming tick will cause a bar to be touched, hence the event OnBarUpdate(), triggered by each incoming tick that is recognized.
2. "Tick" as the minimum measure that a security can move. Most bars will usually cover more than one of these. Hence the range of the bar in ticks.

[Harry]

I confess I'm a bit confused by your description. Historical bars don't just contain one single tick - they have an open, a low, a high, and a close. A tick by definition is a single value. I do roughly understand what are you trying to explain about the transition bar. If I open a chart then the open one is the transition bar as it'll have to load OHLC and then start adding ticks from there.

I have simplified. Of course historical bars contain more than a single tick. I have more explained the behaviour of OnBarUpdate(), which

-> is called once for every historical bar (allowing you to access OHLC of that bar)
-> is called once with every incoming real-time tick in COBC = false (allowing you to access OHLC of that bar in progress)