Getting erroneous charts.

[molecool]

Alright...

Thanks for the input to both of you guys but I want to refocus this thread on how to fix the problem. I'm just deleted everything in my tick and minute folders, rebuilt the database, and will connect via the iQFeed connector instead of Kinetick. Once I see live data on Monday morning I'll chime in here once it happens again.

[molecool]

Continued strangeness

Over the weekend I have deleted all my tick and minute data in the db folder and also repaired the database. Today I'm connected live via iQFeed and am seeing strange values when right mouse clicking on the progress candle - please see the enclosed screen grab.

That 'high' value as reported in the right mouse candle info panel is obviously wrong and is another indication that there's something very strange going on.

To convey the magnitude of this - my strategy bought back at 1.3227 at 7:18am EDT despite the fact that according to the chart the EUR/USD hadn't climbed there.

Execution: Execution='NT-00026' Instrument='$EURUSD' Account='Sim101' Name='Short Position' Exchange=Default Price=1.32115 Quantity=1 Market position=Short Commission=0 Order='NT-00033' Time='6/10/2013 7:18:11 AM' Multiplier=1E-05
Short stop: 1.3227
Stop order: Order='NT-00034/Sim101' Name='Stop loss' State=PendingSubmit Instrument='$EURUSD' Action=BuyToCover Limit price=0 Stop price=1.3227 Quantity=1 Strategy='CrazyIvanPlus' Type=Stop Tif=Gtc Oco='' Filled=0 Fill price=0 Token='4094c8c1c1084afc93d32a7876452632' Gtd='12/1/2099 12:00:00 AM'
Stop order Action: BuyToCover
Stop order Price: 1.3227
Current Bid: 1.32115
*Current Ask*: 1.32115
B/A Spread: 0 ticks.

By the way, that is the same candle that supposedly has a high of 1.3201 according to the right mouse-click - which of course is way too low and also incorrect.

If you look at the second screen grab - this is on a different chart and after pressing F5 several times.

The third one shows what I'm seeing after reloading all historical data - and now the strategy's activities seem to be in sync with what apparently really happened.

Let's not forget the case reported at the beginning of this thread where Ninja is plotting long spikes that do not exist and my strategy executes based on that, which is in fact the inverse scenario.

All this explains some of the very strange behavior I have seen during live trading as of late and it has cost me literally weeks of detective work to pin this on it being a data problem. I definitely need some help with this as I'm out of options on how to fix this.

[NinjaTrader_Matthew]

Hello Molecool,

Thanks for the update.

I have not be able to reproduce this quite yet. Regarding your IQFeed client version, are you on the recent client version 5, or are you using the supported version 4.7.1.3?

At any rate, you should not experience this with the Kinetick adapter - I was simply curious if you were using that client instead.

Thanks for also giving me a screen shot of your data tab. In my tests, I use tenth pip for forex as this is the smallest increment that Kinetick/IQFeed will send. I'll start testing at half pip as per your settings to see if I can run into anything here.

In the mean time, is there any particular reason you're using half pip? Is this just a personal preference? As further tests for our development team to isolate this behavior, would you be able to test using the Tenth Pip settings and let us know if you see this behavior?

I appreciate the detailed and well thought out discussion on these items over the weekend. We'll be working with our development team regarding this behavior to better understand what is occurring. However we will need to be able to reproduce this behavior on our end in order to debug, so please feel free to share any specific steps or settings you may have to help us in this case.

[molecool]

Hello Molecool,

Thanks for the update.

I have not be able to reproduce this quite yet. Regarding your IQFeed client version, are you on the recent client version 5, or are you using the supported version 4.7.1.3?

At any rate, you should not experience this with the Kinetick adapter - I was simply curious if you were using that client instead.

Thanks for also giving me a screen shot of your data tab. In my tests, I use tenth pip for forex as this is the smallest increment that Kinetick/IQFeed will send. I'll start testing at half pip as per your settings to see if I can run into anything here.

In the mean time, is there any particular reason you're using half pip? Is this just a personal preference? As further tests for our development team to isolate this behavior, would you be able to test using the Tenth Pip settings and let us know if you see this behavior?

I appreciate the detailed and well thought out discussion on these items over the weekend. We'll be working with our development team regarding this behavior to better understand what is occurring. However we will need to be able to reproduce this behavior on our end in order to debug, so please feel free to share any specific steps or settings you may have to help us in this case.

Hi Matthew - I actually edited my comment a few times - can you please re-read it again?

In the meantime - my IQFeed client is 5.0.0.14 and I am running NT version 7.0.1000.12. The reason why I am running half pip is that I will eventually run this strategy on IB and AFAIK the smallest increment on their end is half pip.

Not sure what I can do to recreate the problem. It seems to be a rather complex problem and I'm not sure it's actually strategy related.

I will switch back to the Kinetick adapter now and report back. Are there any trace or log files that you would need to help identify this problem?

One thought I just had - is it possible this may be caused by having the Yahoo connector running at the same time? I just closed it and only have the Kinetick connector on.

[NinjaTrader_Matthew]

Hello,

I'm not sure the yahoo connection should have an effect as that should just be equity and stock data. Do you have any forex mappings for yahoo setup?

However we do need to just simply this as much as possible to debug so please disconnect from any other providers until we can isolate or resolve this.

The log/traces will not provide any thing of use for this particular case.

As far as recreating, it is difficult but I appreciate the details you've given so far. If you can keep a mental log if what you're doing through out the day such as switching instruments, etc please let us know what you can provide from right before you saw this issue on a chart.

What would be very useful is if you could export your historical data the next time this occurs before you have reloaded it to resolve.

To set this up, please go to Tools--> Options--> Data tab--> check "Save chart data as historical"

The next time this occurs, before your reload the chart, please go to Tools--> Historical Data Manager--> Export tab and export the tick and minute data for the instrument that is displaying this bad bar.

Once exported, please send this support[at]ninjatrader[dot]com and we can look into this in detail.

[molecool]

Hello,

I'm not sure the yahoo connection should have an effect as that should just be equity and stock data. Do you have any forex mappings for yahoo setup?

However we do need to just simply this as much as possible to debug so please disconnect from any other providers until we can isolate or resolve this.

The log/traces will not provide any thing of use for this particular case.

As far as recreating, it is difficult but I appreciate the details you've given so far. If you can keep a mental log if what you're doing through out the day such as switching instruments, etc please let us know what you can provide from right before you saw this issue on a chart.

What would be very useful is if you could export your historical data the next time this occurs before you have reloaded it to resolve.

To set this up, please go to Tools--> Options--> Data tab--> check "Save chart data as historical"

The next time this occurs, before your reload the chart, please go to Tools--> Historical Data Manager--> Export tab and export the tick and minute data for the instrument that is displaying this bad bar.

Once exported, please send this support[at]ninjatrader[dot]com and we can look into this in detail.

Thanks Matthew - I have disabled the Yahoo connector as we don't need to throw any more monkey wrenches into this. I should have thought of this before - habits die hard.

I have now set the data setting to use the chart data as historical. Is this something one would usually keep off in a production system? Can you please elaborate as to why this exists in the first place?

The only other thought I have is that I am connecting out of Spain right now and it's possible that it may be caused by latency. My connection here is very fast (about 16Mb/sec) but latency may still be an issue, who knows. FWIW - the system I'm running this one is running under Parallels and the system clock is set to NYC.

Thanks for attempting to recreate this. I know it's a pain in the butt and I'll do my best to provide as much context as possible. Per your suggestion I will monitor things and then send you the chart data when I see it acting up again.

[Harry]

Sample Spikes

Sample spike: Here is a spike for a NQ 06-13 chart. This is what I did:

Connected NinjaTrader at 15:36 CET and updated historical data via the data manager. Then let NinjaTrader run, with a single chart open. Came back about half an hour later and opened a few new charts.

For NQ 06-13 NinjaTrader produced a false spike, as can be seen on the chart. To remove the spike, I had to toggle to a different instrument and then switch back to NQ 06-13. As you will notice this spike points into the direction of prior price action. NinjaTrader has again used a false tick to assemble that bar.

I am getting this every day. My datafeed here is Interactive Brokers, TWS version as specified in the NinjaTrader connection guide, NinjaTrader 7.0.1000.13.

All versions of NinjaTrader that I have used during the last 4 years have produced spikes nearly every day. The bug has never been removed. I believe that this is one of the single most important NinjaTrader bugs. You can never rely that NinjaTrader will display the current bar correctly, when you open a chart.

Tested and observed with 3 different machines under various operating systems. Discussed with NinjaTrader support at least 10 times. Problem never solved.

Will this behaviour go away with NinjaTrader 8?

[Harry]

Here is another sample chart of UB 06-13:

Please notice the price and volume spike. After toggling to a different instrument and back, the spike goes away. Remember that I had download UB 06-13 data at around 15:40, and not touched UB 06-13 since this time. Again you can notice that NinjaTrader adds one false tick, which was produced around the connection time to the bar, which in turn produces the spike. There is also the volume of 7645 contracts shown. This value comes very close to the total volume observed between 15:40 and 16:35.

Conclusion:

When opening a chart after being connected for an hour, NinjaTrader sometimes (not always, but quite frequently)

-> adds a false tick to the current bar, the false tick being taken from the bar of the connection time
-> further shows the volume added up from the moment when connected to the datafeed (or update of historical data) and the moment when the chart was opened.

As a highly satisfied user of NinjaTrader - I am otherwise impressed with the overall stability, features and reliability of NinjaTrader 7 - I do not understand, why it should not be possible to solve this problem, which has annoyed me for years. Please fix it!

[molecool]

Will this behaviour go away with NinjaTrader 8?

That question is immaterial as NT8 is expected to go into beta perhaps later this year and most likely we'll have to go through a few months of bug fixing until anyone would want to use it in production. IF this is a systemic issue then it needs to be fixed and for version 7 which many of us are going to be using for several years. I have been a SWE in the past and I can tell you that new versions are not the answer to systemic problems (again assuming that we have one at hand here). Personally I find it puzzling that the two of us are the only ones reporting this however.

Questions:

• Where do you connect from?
• Is your connection stable?
• What is your data feed?
• Do you have several feeds open at the same time?
• Are you using full pips, half pips, or tenth pips in your FX settings?

I'm trying to establish if there are any commonalities between the two of us.

[molecool]

Here is another sample chart of UB 06-13:

As a highly satisfied user of NinjaTrader - I am otherwise impressed with the overall stability, features and reliability of NinjaTrader 7 - I do not understand, why it should not be possible to solve this problem, which has annoyed me for years. Please fix it!

Well, the good news is for us both is that it's not just one person reporting this, now there are two. We need to find a way for NT support to reproduce it first. Reporting what's happening on our end can only lead so far. So it's probably best if we help them recreate our environments as closely as possible.

[molecool]

Just happened again....

Enclosed are the EUR/USD charts showing what's happening. I'm seeing a delta between two identical charts. One showing the correct, the other the incorrect data. Also enclosed the tick and minute data for the EUR/USD.

[molecool]

Here's the tick data - I had to zip it to add it as an attachment - it was too big before it seems.

[NinjaTrader_Matthew]

Are these the minute and tick data combined in two files?

Can you export just the tick data and upload that?

[NinjaTrader_Matthew]

Thanks for the update - you can upload it via www.wetransfer.com and use support[at]ninjatrader[dot]com

[molecool]

Thanks for the update - you can upload it via www.wetransfer.com and use support[at]ninjatrader[dot]com

See below please - I just added it as a zip file.