Junk in the Zen Data feed -- serious bug

[Turtle Trader]

Hello,

Recently both my developer and myself have encountered junk data in the Zen Data feed.

The problem is the Double variable is a floating point and sometimes has junk in the sub-tick data that cause it to not compare correctly to another variable with the exact correct value.

Here is the note from my developer:

================================================== ===

Can you believe that the weird "sometimes it will, sometimes it won't"
behaviour of the indicator was ROUNDING ERROR!?!

When you compare Close[0] with Open[0] you don't expect to be measuring 1.2450 with 1.24500000000000000017. But that's what was happening. I could have punched the screen.

================================================== ===

It cost him two hours to find this problem, and I probably spent at least that when I had a similar problem with code I'm writing.

Not only does this create a problem for coder, but it also creates operational bugs in code that are written correctly.

Personally I consider it a serious bug in either NinjaTrader or the Zen Data feed.

Is it too much to expect the Ninja functions to return the exact tick price, nothing more or less, even if it is down to extremely insignificant digits?

As coders we should not have to worry about the precision of the data feed, especially if it is a Zen Fire feed.

And we should not have to massage the data on each tick with a cleanup function like Instrument.MasterInstrument.Round2TickSize(value).

Cannot Ninja or Zen clean the data before it is given to us? Or at lease provide a global run time option that does this automatically?

In the world of Financial applications, say like in the banking and any accounting systems, you would never find applications using a floating point (Double) to store or calculate prices or dollar (any currency) amounts. The precision is not good enough.

How about a run time option in the Control Panel/Options to clean up the ZenFire, or other feed?


Thanks....

[NinjaTrader_RyanM1]

Hello Turtle Trader,

Thank you for your post and suggestions.

We are aware of issues that relate to floating point arithmetic. When working with actual double values and floating point arithmetic, this is one of the things you should definitely be aware of.

The issue is not with the presentation of Zenfire data, but making equality comparisons in code. There are several approaches for dealing with this and are outlined in the link below:

http://www.ninjatrader.com/support/forum/showthread.php?t=3929

The link above also provides references to further Floating Point arithmetic analysis.

[Turtle Trader]

reply

God, sorry to say...but that is caveman type coding.

How about some higher level function embedded in Ninja that guarantees the values returned in the double don't have sub tick data?

It would save your developers such a headache.

I know you'll never do this..but it would be even better to have an option where all price data returned is in decimal format. The compiler could recognize the variable data type and match it.

[NinjaTrader_RyanM1]

Thanks for the contribution and suggestions. I will forward your concerns to our development team.

[NinjaTrader_Dierk]

First of all I suggest you reconsider your tone. Terms like
>> caveman type coding
... are inappropriate and offensive.

Next, the decision to use double and not decimal as .NET price types was well thought as the decimal type comes with a serious performance penalty which 99% of our users would not accept.

The downside is the potential of rounding issues which for the most part is already covered in NT. On your particular scenario I needed to know: do you experience the issue only for Open/High/Low/Close[0] or also for say Close[5] (meaning not the last bar)?

Thanks

[Turtle Trader]

reply

Yes, I understand that Decimal is implemented in software because the Intel chip has no native Decimal data type.

But that should not prevent Ninja from having an option in the data interface to round the Double sub-tick data to the tick level before the code gets it. It would not be hard for you to implement in your platform and it would solve these types of problems that require exact precision.

I'll ask my developer to respond to the specifics next week. I believe that Open [], Close [] comparison was his immediate problem.

------------------------------------------------------------------

Then, permit me then to clarify my comment in a more verbose and pleasing way.

When I first started coding it was in assembler. Each time there was a new language to learn it freed the us programmers from worrying about low level functions and allowed us to be more productive and focus more on the real algorithm to solve the business requirement.

By not having clean tick data in the Double variable it is forcing more work on the coder as then they now have to create, in their code, conditions that handle this problem. It means that they constantly have to be aware that the Double point variables may not test correctly, or may not produce expected results, and may cause unexpected run-time bugs in the code. It causes more code, which then makes reading the code more difficult for maintenance.

Being new to C# and Ninjascript, that to me, is a throwback to an earlier and less productive time. I was just very surprised to find this condition.

That's all.

By the way, I discovered this is not unique to Ninjascript. I queried Open-E-Cry which also has a C# based scripting language for their proprietary platform. They said they use Double for tick values, and sometimes the Double has sub-tick data in it.

I have no idea what Tradestation does, but can find out if you want.

Why not better your competition and raise the standard to a higher level?

It would make NinaScript more stable in this area and benefit everyone.

[saltminer]

price rounding issues on historical bars too

Hi,
just chipping in here as the developer.
I was testing on Simulated Data Feed of the 6E (EURUSD), with prices in the area of 1.3 with four decimal places.
I was quite surprised when coding to detect Doji bars, that sometimes the line
if Open[x] == Close[x] failed, even though the price was the same.
I discovered the issue was tiny discrepancies, such as
1.3341 versus 1.33410000000000000008

This was on current and historical bars.

I have had to go through every price test in the code and use
Instrument.MasterInstrument.Round2TickSize(), which is extremely cumbersome and I tend to agree with Turtle Trader that ANY price should ALWAYS be rounded to ticksize automatically for us.

saltminer

[NinjaTrader_Dierk]

There really should not be any surprise since as a developer you should be aware of the double.Epsilon logic related to comparing floating point values. Please consult Microsoft docs for details. The issue reported by original poster likely is different.

[Turtle Trader]

Value Added

For me the issue is not what is the basic functionality of C# and Microsoft compilers, and the problems with floating point..those are a given and not to be argued.

To me the issue is the value added that Ninja can offer it's developers by an optional rounding of the tick data before Ninjascript gets the data.

Any other problems with Doubles would still exist and would be handled inside the code as that then is a problem of C# and Intel. But at least the incoming data would not have sub-tick data in it.

When you consider the cost to add this feature to Ninja it seems to me it would be trivial. Just apply Round2TickSize() to each tick data in the background and allow the code to select the option.

I know you are all heads down with the new release...but this to me would be such a good thing to have in NT7.

I don't know how to make a formal request...but surely this is worth considering.

[NinjaTrader_Dierk]

Please see my post #5 on the info needed for proper analysis. Thanks

[Turtle Trader]

reply

Personally, I noticed it in the Close[0] data.

My strategy is extremely simple so I didn't go any further. (being a newbie c#). Was surprise though to see subtick data there.

I can write a script to dump some back bar data if that will help.

[NinjaTrader_Dierk]

Thanks, we needed to know if it only (!) occurs on OHLC[0] or on prior OHLC values as well. Thanks

[Turtle Trader]

reply

OK, I'll do some research and get back to you.

My apologies for the troglodyte comment earlier in the thread.

Will ask my developer to comment also, but he is down from an operation at the moment.

[Turtle Trader]

reply: don't laugh

OK, this is the best I can do as a newbie....see the attached file...perhaps someone can fix it and repost it here.

The idea is output a csv file when sub-tick data is encountered.

The idea is to wait for the first bar with sub-tick data. Then, for the next 10 bars, record the OHLC as it moves back into history to see if the sub-tick data is being propagated into the historical bars on the chart.

No guarantees there are not logic errors also...

Ha!

[NinjaTrader_Dierk]

Thanks. Would you mind providing some feedback on the question I posted in post #12.? This is critical for us to understand the nature of the issue.