Getting erroneous charts.

[Harry]

Here is another sample chart of UB 06-13:

Please notice the price and volume spike. After toggling to a different instrument and back, the spike goes away. Remember that I had download UB 06-13 data at around 15:40, and not touched UB 06-13 since this time. Again you can notice that NinjaTrader adds one false tick, which was produced around the connection time to the bar, which in turn produces the spike. There is also the volume of 7645 contracts shown. This value comes very close to the total volume observed between 15:40 and 16:35.

Conclusion:

When opening a chart after being connected for an hour, NinjaTrader sometimes (not always, but quite frequently)

-> adds a false tick to the current bar, the false tick being taken from the bar of the connection time
-> further shows the volume added up from the moment when connected to the datafeed (or update of historical data) and the moment when the chart was opened.

As a highly satisfied user of NinjaTrader - I am otherwise impressed with the overall stability, features and reliability of NinjaTrader 7 - I do not understand, why it should not be possible to solve this problem, which has annoyed me for years. Please fix it!

[Harry]

Sample Spikes

Sample spike: Here is a spike for a NQ 06-13 chart. This is what I did:

Connected NinjaTrader at 15:36 CET and updated historical data via the data manager. Then let NinjaTrader run, with a single chart open. Came back about half an hour later and opened a few new charts.

For NQ 06-13 NinjaTrader produced a false spike, as can be seen on the chart. To remove the spike, I had to toggle to a different instrument and then switch back to NQ 06-13. As you will notice this spike points into the direction of prior price action. NinjaTrader has again used a false tick to assemble that bar.

I am getting this every day. My datafeed here is Interactive Brokers, TWS version as specified in the NinjaTrader connection guide, NinjaTrader 7.0.1000.13.

All versions of NinjaTrader that I have used during the last 4 years have produced spikes nearly every day. The bug has never been removed. I believe that this is one of the single most important NinjaTrader bugs. You can never rely that NinjaTrader will display the current bar correctly, when you open a chart.

Tested and observed with 3 different machines under various operating systems. Discussed with NinjaTrader support at least 10 times. Problem never solved.

Will this behaviour go away with NinjaTrader 8?

[molecool]

Hello,

I'm not sure the yahoo connection should have an effect as that should just be equity and stock data. Do you have any forex mappings for yahoo setup?

However we do need to just simply this as much as possible to debug so please disconnect from any other providers until we can isolate or resolve this.

The log/traces will not provide any thing of use for this particular case.

As far as recreating, it is difficult but I appreciate the details you've given so far. If you can keep a mental log if what you're doing through out the day such as switching instruments, etc please let us know what you can provide from right before you saw this issue on a chart.

What would be very useful is if you could export your historical data the next time this occurs before you have reloaded it to resolve.

To set this up, please go to Tools--> Options--> Data tab--> check "Save chart data as historical"

The next time this occurs, before your reload the chart, please go to Tools--> Historical Data Manager--> Export tab and export the tick and minute data for the instrument that is displaying this bad bar.

Once exported, please send this support[at]ninjatrader[dot]com and we can look into this in detail.

Thanks Matthew - I have disabled the Yahoo connector as we don't need to throw any more monkey wrenches into this. I should have thought of this before - habits die hard.

I have now set the data setting to use the chart data as historical. Is this something one would usually keep off in a production system? Can you please elaborate as to why this exists in the first place?

The only other thought I have is that I am connecting out of Spain right now and it's possible that it may be caused by latency. My connection here is very fast (about 16Mb/sec) but latency may still be an issue, who knows. FWIW - the system I'm running this one is running under Parallels and the system clock is set to NYC.

Thanks for attempting to recreate this. I know it's a pain in the butt and I'll do my best to provide as much context as possible. Per your suggestion I will monitor things and then send you the chart data when I see it acting up again.

[NinjaTrader_Matthew]

Hello,

I'm not sure the yahoo connection should have an effect as that should just be equity and stock data. Do you have any forex mappings for yahoo setup?

However we do need to just simply this as much as possible to debug so please disconnect from any other providers until we can isolate or resolve this.

The log/traces will not provide any thing of use for this particular case.

As far as recreating, it is difficult but I appreciate the details you've given so far. If you can keep a mental log if what you're doing through out the day such as switching instruments, etc please let us know what you can provide from right before you saw this issue on a chart.

What would be very useful is if you could export your historical data the next time this occurs before you have reloaded it to resolve.

To set this up, please go to Tools--> Options--> Data tab--> check "Save chart data as historical"

The next time this occurs, before your reload the chart, please go to Tools--> Historical Data Manager--> Export tab and export the tick and minute data for the instrument that is displaying this bad bar.

Once exported, please send this support[at]ninjatrader[dot]com and we can look into this in detail.

[molecool]

Hello Molecool,

Thanks for the update.

I have not be able to reproduce this quite yet. Regarding your IQFeed client version, are you on the recent client version 5, or are you using the supported version 4.7.1.3?

At any rate, you should not experience this with the Kinetick adapter - I was simply curious if you were using that client instead.

Thanks for also giving me a screen shot of your data tab. In my tests, I use tenth pip for forex as this is the smallest increment that Kinetick/IQFeed will send. I'll start testing at half pip as per your settings to see if I can run into anything here.

In the mean time, is there any particular reason you're using half pip? Is this just a personal preference? As further tests for our development team to isolate this behavior, would you be able to test using the Tenth Pip settings and let us know if you see this behavior?

I appreciate the detailed and well thought out discussion on these items over the weekend. We'll be working with our development team regarding this behavior to better understand what is occurring. However we will need to be able to reproduce this behavior on our end in order to debug, so please feel free to share any specific steps or settings you may have to help us in this case.

Hi Matthew - I actually edited my comment a few times - can you please re-read it again?

In the meantime - my IQFeed client is 5.0.0.14 and I am running NT version 7.0.1000.12. The reason why I am running half pip is that I will eventually run this strategy on IB and AFAIK the smallest increment on their end is half pip.

Not sure what I can do to recreate the problem. It seems to be a rather complex problem and I'm not sure it's actually strategy related.

I will switch back to the Kinetick adapter now and report back. Are there any trace or log files that you would need to help identify this problem?

One thought I just had - is it possible this may be caused by having the Yahoo connector running at the same time? I just closed it and only have the Kinetick connector on.

[NinjaTrader_Matthew]

Hello Molecool,

Thanks for the update.

I have not be able to reproduce this quite yet. Regarding your IQFeed client version, are you on the recent client version 5, or are you using the supported version 4.7.1.3?

At any rate, you should not experience this with the Kinetick adapter - I was simply curious if you were using that client instead.

Thanks for also giving me a screen shot of your data tab. In my tests, I use tenth pip for forex as this is the smallest increment that Kinetick/IQFeed will send. I'll start testing at half pip as per your settings to see if I can run into anything here.

In the mean time, is there any particular reason you're using half pip? Is this just a personal preference? As further tests for our development team to isolate this behavior, would you be able to test using the Tenth Pip settings and let us know if you see this behavior?

I appreciate the detailed and well thought out discussion on these items over the weekend. We'll be working with our development team regarding this behavior to better understand what is occurring. However we will need to be able to reproduce this behavior on our end in order to debug, so please feel free to share any specific steps or settings you may have to help us in this case.

[molecool]

Continued strangeness

Over the weekend I have deleted all my tick and minute data in the db folder and also repaired the database. Today I'm connected live via iQFeed and am seeing strange values when right mouse clicking on the progress candle - please see the enclosed screen grab.

That 'high' value as reported in the right mouse candle info panel is obviously wrong and is another indication that there's something very strange going on.

To convey the magnitude of this - my strategy bought back at 1.3227 at 7:18am EDT despite the fact that according to the chart the EUR/USD hadn't climbed there.

Execution: Execution='NT-00026' Instrument='$EURUSD' Account='Sim101' Name='Short Position' Exchange=Default Price=1.32115 Quantity=1 Market position=Short Commission=0 Order='NT-00033' Time='6/10/2013 7:18:11 AM' Multiplier=1E-05
Short stop: 1.3227
Stop order: Order='NT-00034/Sim101' Name='Stop loss' State=PendingSubmit Instrument='$EURUSD' Action=BuyToCover Limit price=0 Stop price=1.3227 Quantity=1 Strategy='CrazyIvanPlus' Type=Stop Tif=Gtc Oco='' Filled=0 Fill price=0 Token='4094c8c1c1084afc93d32a7876452632' Gtd='12/1/2099 12:00:00 AM'
Stop order Action: BuyToCover
Stop order Price: 1.3227
Current Bid: 1.32115
*Current Ask*: 1.32115
B/A Spread: 0 ticks.

By the way, that is the same candle that supposedly has a high of 1.3201 according to the right mouse-click - which of course is way too low and also incorrect.

If you look at the second screen grab - this is on a different chart and after pressing F5 several times.

The third one shows what I'm seeing after reloading all historical data - and now the strategy's activities seem to be in sync with what apparently really happened.

Let's not forget the case reported at the beginning of this thread where Ninja is plotting long spikes that do not exist and my strategy executes based on that, which is in fact the inverse scenario.

All this explains some of the very strange behavior I have seen during live trading as of late and it has cost me literally weeks of detective work to pin this on it being a data problem. I definitely need some help with this as I'm out of options on how to fix this.

[molecool]

Alright...

Thanks for the input to both of you guys but I want to refocus this thread on how to fix the problem. I'm just deleted everything in my tick and minute folders, rebuilt the database, and will connect via the iQFeed connector instead of Kinetick. Once I see live data on Monday morning I'll chime in here once it happens again.

[Harry]

I confess I'm a bit confused by your description. Historical bars don't just contain one single tick - they have an open, a low, a high, and a close. A tick by definition is a single value. I do roughly understand what are you trying to explain about the transition bar. If I open a chart then the open one is the transition bar as it'll have to load OHLC and then start adding ticks from there.

I have simplified. Of course historical bars contain more than a single tick. I have more explained the behaviour of OnBarUpdate(), which

-> is called once for every historical bar (allowing you to access OHLC of that bar)
-> is called once with every incoming real-time tick in COBC = false (allowing you to access OHLC of that bar in progress)

[koganam]

I confess I'm a bit confused by your description. Historical bars don't just contain one single tick - they have an open, a low, a high, and a close. A tick by definition is a single value. I do roughly understand what are you trying to explain about the transition bar. If I open a chart then the open one is the transition bar as it'll have to load OHLC and then start adding ticks from there.

I suspect that it is a synchronization or latency error. In my case I'm connecting from Spain and although my connection is very fast the latency could introduce some type of problem. When I was trading out of CA I never experienced this particular issue.

Different meanings of the word "tick".

1. "Tick" as the event that causes a bar to be drawn or modified. If historical or realtime with COBC = false, there will be only one of these per bar. If realtime, then each incoming tick will cause a bar to be touched, hence the event OnBarUpdate(), triggered by each incoming tick that is recognized.
2. "Tick" as the minimum measure that a security can move. Most bars will usually cover more than one of these. Hence the range of the bar in ticks.

[molecool]

Transition Bar: When you open your workspace and connect to your data feed, NinjaTrader will automatically backfill your chart with historical data. Each historical bar on a chart has a single tick.

Let us assume that you have a 5 minute chart and connect at 8:23 AM. In that case

- the bar with the time stamp 8:15 is a (single-tick) historical bar
- the bar with the time stamp 8:20 is a (single-tick) historical bar
- the bar with the time stamp 8:25 is a (multi-tick) transition bar
- the bar with the time stamp 8:30 is a (multi-tick) real-time bar

Let us have a look at the 8:25 AM bar. After connecting the backfill will cover the period from 8:20 to 8:23 (incomplete historical bar corresponding to a single tick) and NinjaTrader will then add all incoming ticks as supplied by your real-time data feed to that incomplete bar. That transition bar is therefore built from a single historical tick and real-time ticks which cover the remaining period.

By the way, I'm connecting via Kinetick - I'm going to try using the iQFeed adapter instead to see if I can replicate it like that.

There can be only one transition bar on your chart. All bars to the left of that bars are single-tick historical bars, all bars to the right of the transition bar are multi-tick real-time bars. If you open your chart exactly at 8:25 AM, there may be no transition bar, when the last historical bar is already complete.

If you refresh your chart via F5, then NinjaTrader will replace all multi-tick real-time bars on your chart with single-tick historical bars. At this stage, the last bar on your chart will become the new transition bar, as incoming real-time ticks are added to that bar. This explains that any spike that has been created on the previous transition bar will go away, when you refresh. However, you may observe a new spike for the last bar.

A spike is nothing else than a single false tick that has been added to the transition bar. You can observe that this tick is not always immediately added, when the incomplete historical bar is added to your chart, but the addition of this tick can be delayed a few seconds. We know that it is a historical tick that is being added as the spike always covers recent price action.

The spikes are related to the amount of real-time data handled by NinjaTrader, maybe it is a synchronisation issue or similar. What you can do

(a) disconnect and reconnect NinjaTrader -> all spikes should go away
(b) take the affected chart and toggle between two instruments -> the spike should go away

I would be interested to know, whether the behaviour on your side is similar as I have recently observed. The problem may depend on the architecture of the broker API or the datafeed adapter of NinjaTrader. It is not obvious that it will appear with all datafeeds.

I confess I'm a bit confused by your description. Historical bars don't just contain one single tick - they have an open, a low, a high, and a close. A tick by definition is a single value. I do roughly understand what are you trying to explain about the transition bar. If I open a chart then the open one is the transition bar as it'll have to load OHLC and then start adding ticks from there.

I suspect that it is a synchronization or latency error. In my case I'm connecting from Spain and although my connection is very fast the latency could introduce some type of problem. When I was trading out of CA I never experienced this particular issue.

[Harry]

Frankly speaking I wish I hadn't received your comment as this suggests that there may be an underlying data management problem. Truth be told however I have run NinjaTrader for about four years now and this is the first time I run into this particular issue. So let's see if they can come up with a solution.

Question: Have you ever tried to repair the database? I assume you have but just want to be sure. In order to fix some unrelated problem I have in the past resorted to manually removing all files in the db folder, basically giving NinjaTrader a fresh start in building its database. Of course you do lose all historical data this way.

You speak of a transition bar and I am not certain as what you are referring to. Are you talking about the open bar?

Also, are you running a strategy when the problem occurs? In my case I have one running on a 60-min chart which also loads a single tick series for executions. So I wonder if that may be what's causing the error.

The problem manifests itself on FX charts and of course reloading the chart resets the bar to its correct values.

Transition Bar: When you open your workspace and connect to your data feed, NinjaTrader will automatically backfill your chart with historical data. Each historical bar on a chart has a single tick.

Let us assume that you have a 5 minute chart and connect at 8:23 AM. In that case

- the bar with the time stamp 8:15 is a (single-tick) historical bar
- the bar with the time stamp 8:20 is a (single-tick) historical bar
- the bar with the time stamp 8:25 is a (multi-tick) transition bar
- the bar with the time stamp 8:30 is a (multi-tick) real-time bar

Let us have a look at the 8:25 AM bar. After connecting the backfill will cover the period from 8:20 to 8:23 (incomplete historical bar corresponding to a single tick) and NinjaTrader will then add all incoming ticks as supplied by your real-time data feed to that incomplete bar. That transition bar is therefore built from a single historical tick and real-time ticks which cover the remaining period.

There can be only one transition bar on your chart. All bars to the left of that bars are single-tick historical bars, all bars to the right of the transition bar are multi-tick real-time bars. If you open your chart exactly at 8:25 AM, there may be no transition bar, when the last historical bar is already complete.

If you refresh your chart via F5, then NinjaTrader will replace all multi-tick real-time bars on your chart with single-tick historical bars. At this stage, the last bar on your chart will become the new transition bar, as incoming real-time ticks are added to that bar. This explains that any spike that has been created on the previous transition bar will go away, when you refresh. However, you may observe a new spike for the last bar.

A spike is nothing else than a single false tick that has been added to the transition bar. You can observe that this tick is not always immediately added, when the incomplete historical bar is added to your chart, but the addition of this tick can be delayed a few seconds. We know that it is a historical tick that is being added as the spike always covers recent price action.

The spikes are related to the amount of real-time data handled by NinjaTrader, maybe it is a synchronisation issue or similar. What you can do

(a) disconnect and reconnect NinjaTrader -> all spikes should go away
(b) take the affected chart and toggle between two instruments -> the spike should go away

I would be interested to know, whether the behaviour on your side is similar as I have recently observed. The problem may depend on the architecture of the broker API or the datafeed adapter of NinjaTrader. It is not obvious that it will appear with all datafeeds.

[molecool]

I have had and still do have similar problems with other data feeds. This goes on for years, and in despite of numerous discussions, the problem has never been solved.

The main issue is the transition bar, which is the bar that NinjaTrader builds from both historical data and real-time data. NInjaTrader does not always build this bar properly.

all historical bars -> ok
transition bar (hybrid bar 1 historical tick + real-time ticks) -> sometimes produces spikes
real-time bars -> ok

Now, when I refresh via F5, the spike bar disappears, but the current bar on the chart may produce a new spike.

I have made the following observations:

(1) The direction of the artificial spike ALWAYS points in the direction of prior price action.
This is also confirmed by all the screen shots that have been posted in this thread.

(2) The problem does not occur if you connect to NinjaTrader and open a chart immediately. It occurs, if you connect and have a chart open for instrument 1, and then an hour or a few hours later

-> switch that chart to a new instrument 2
-> open a new chart for instrument 2

(3) The spike approximately represents the price action from the time when you have connected NinjaTrader or reloaded historical data and the time when you first open a chart for instrument 2

(4) I have also observed volume spikes for the hybrid bars.

(5) The problem does not always occur, but it will sometimes hit you and sometimes it won't. It is more frequent with some instruments than with others for unknown reasons.


How to deal with the problem:

Open a chart for an instrument. Wait until the bar close of the transition bar, which is built from a historical and the following real-time ticks. If you observe a price or volume spike switch the chart to a different instrument and then back again to the instrument that you wish to chart. The chart should now display correctly.

I have observed this bug on three different PCs with different data providers.

The problem is not linked to bad ticks from the data provider, but it is clearly a NinjaTrader problem, as NinjaTrader uses false real-time ticks (proof: direction of spike) when assembling the historical tick of the transition bar.

I am glad that somebody else comes up with the issue, as so far I could not persuade anybody that the bug exists and should be removed.

Just a note: NinjaTrader staff has been very helpful and effective to find solutions for all other problems that I have encountered. This is really the only significant nuisance, which I would like to have removed, and that is the reason that I have posted here.

Frankly speaking I wish I hadn't received your comment as this suggests that there may be an underlying data management problem. Truth be told however I have run NinjaTrader for about four years now and this is the first time I run into this particular issue. So let's see if they can come up with a solution.

Question: Have you ever tried to repair the database? I assume you have but just want to be sure. In order to fix some unrelated problem I have in the past resorted to manually removing all files in the db folder, basically giving NinjaTrader a fresh start in building its database. Of course you do lose all historical data this way.

You speak of a transition bar and I am not certain as what you are referring to. Are you talking about the open bar?

Also, are you running a strategy when the problem occurs? In my case I have one running on a 60-min chart which also loads a single tick series for executions. So I wonder if that may be what's causing the error.

The problem manifests itself on FX charts and of course reloading the chart resets the bar to its correct values.

[Harry]

NinjaTrader does not always correctly assemble the transition bar

I have had and still do have similar problems with other data feeds. This goes on for years, and in despite of numerous discussions, the problem has never been solved.

The main issue is the transition bar, which is the bar that NinjaTrader builds from both historical data and real-time data. NInjaTrader does not always build this bar properly.

all historical bars -> ok
transition bar (hybrid bar 1 historical tick + real-time ticks) -> sometimes produces spikes
real-time bars -> ok

Now, when I refresh via F5, the spike bar disappears, but the current bar on the chart may produce a new spike.

I have made the following observations:

(1) The direction of the artificial spike ALWAYS points in the direction of prior price action.
This is also confirmed by all the screen shots that have been posted in this thread.

(2) The problem does not occur if you connect to NinjaTrader and open a chart immediately. It occurs, if you connect and have a chart open for instrument 1, and then an hour or a few hours later

-> switch that chart to a new instrument 2
-> open a new chart for instrument 2

(3) The spike approximately represents the price action from the time when you have connected NinjaTrader or reloaded historical data and the time when you first open a chart for instrument 2

(4) I have also observed volume spikes for the hybrid bars.

(5) The problem does not always occur, but it will sometimes hit you and sometimes it won't. It is more frequent with some instruments than with others for unknown reasons.


How to deal with the problem:

Open a chart for an instrument. Wait until the bar close of the transition bar, which is built from a historical and the following real-time ticks. If you observe a price or volume spike switch the chart to a different instrument and then back again to the instrument that you wish to chart. The chart should now display correctly.

I have observed this bug on three different PCs with different data providers.

The problem is not linked to bad ticks from the data provider, but it is clearly a NinjaTrader problem, as NinjaTrader uses false real-time ticks (proof: direction of spike) when assembling the historical tick of the transition bar.

I am glad that somebody else comes up with the issue, as so far I could not persuade anybody that the bug exists and should be removed.

Just a note: NinjaTrader staff has been very helpful and effective to find solutions for all other problems that I have encountered. This is really the only significant nuisance, which I would like to have removed, and that is the reason that I have posted here.

[molecool]

molecool,

Thanks for these screen shots.

Do the candles again appear normal after reloading?

Can you send me a screen shot of all your settings under Tools--> Options--> Data tab?

And to clarify, if Kinetick the only connection you're using? Do you connect directly through the Kinetick Adapter, or are you using the IQFeed client with your Kinetick account?

Here's my data settings:

http://screencast.com/t/RMmc88OnqtI

I connect directly via Kinetick only - I don't have any other data feed running.