Bug in ATR, TEMA, Lin Reg, and/or Indicator Initialization

[ChuckAllen]

See the attached file...NT-Bug16.Jpg.

Notice in this optimization...on the 2nd bar...
The 1st ATR is 187...it should be about 4.
The Lin Reg is 527...it should be about 1175.
The 2nd ATR is 187...it should be about 4.

It doesn't look like the values are close until about bar 100 or so.

[NinjaTrader_JoshP]

ChuckAllen,

There is a reason why nothing is plotted on those bars. You are looking at the indicator's unstable period and as such the values generated there are not going to be accurate as there is simply not enough bars to properly calculate the values yet.

[ChuckAllen]

Still A Big Major Problem

I'm terribly sorry, but that's not even close to the point. Of course there's not enough bars to do a proper calculation. But, the code logic should make up for it, not pick seemingly random values and keep them throughout the calculations. The way it is now, the entire invalid back calculation values take a long, long time to work themselves out. It looks like about 100 bars or so. The fact that you don't display the first several bars has nothing to do with the bad indicator values. Look at the included screen dump where you do start to display the values...they are still way, way off.
All it takes is some logic in the beginning of the calculations/indicator to basically say...if there's not enough bars to meet the min required number of bars to do the calcualtions properly, do some type of logic (depending on the indicator) to come close.

For example: if you need to do a 10 bar moving average, but you only have 5...do a 5 bar moving average. When you have 6...do a 6 bar moving average...etc.

I don't really care about the chart. Its the bad calculations in the indicators. That throws off all trades and charting. When I do long term charting...with something like weekly or monthly bars, the values will be off for a year or more.

[NinjaTrader_JoshP]

ChuckAllen,

It is not random values. I guarantee you if you actually ran through the logic of the indicator and did the math of what you were requesting by hand it would turn out to be exactly the same as to what is shown. You are running tons and tons of nested iterations. You are calling ATR of TEMA of TEMA of TEMA of LinReg. Please run through the calculations by hand and you will see what I am talking about.

If you want to run a simple test just throw up an SMA indicator onto the chart and see how it calculates without nesting.

[ChuckAllen]

I honestly can't believe it!

I honestly cannot believe that NT considers this a non-issue!

You are representing FINANCIAL software! Accuracy and Money do matter!

When an indicator is off...and the invalid calculated values are carried forward and invalidate the first several hundred bars...that is a Major Bug. All backtesting and optimization is bogus. It caused several bad trades and threw the results off.

[NinjaTrader_JoshP]

ChuckAllen,

Please look at the way the indicators are calculated. ATR means Average True Range. Average True Range is calculated from High versus Low prices of the input series. You are passing in an input series that does not have high or low prices. It in fact doesn't have prices at all. When you pass in a series that does not have prices ATR has no way of calculating it off of prices and will only be basing the calculation off of the single series you passed in which is LinReg(). That means all your calculations are just LinReg prices - LinReg prices. This for sure will result in the numbers you are seeing.

This is how ATR is calculated.

             if (CurrentBar == 0)
                Value.Set(High[0] - Low[0]);
            else
            {
                double trueRange = High[0] - Low[0];
                trueRange = Math.Max(Math.Abs(Low[0] - Close[1]), Math.Max(trueRange, Math.Abs(High[0] - Close[1])));
                Value.Set(((Math.Min(CurrentBar + 1, Period) - 1 ) * Value[1] + trueRange) / Math.Min(CurrentBar + 1, Period));
            }

You pass in LinReg as the input series. LinReg has no definition of High or Low or even Close. Result is OHLC all equal LinReg values. You are simply nesting indicators that do not make sense together based on what is required for the indicator. ATR indicator requires a Bar series as the input. This is not a bug. This is simply using an inappropriate input series for the indicator in question.

Please consider what you are trying to grab from using an ATR of LinReg. There is no "range" for linear regression plots that can be calculated into an average. There is only a single value per bar. ATR indicator is fundamentally by definition irrelevant when applied to something with only a single value.

[kdoren]

What you are seeing is the result of some poor design decisions. NT7 doesn't have a sensible way of handling "no output" from an indicator,

So instead of "no output", most of NT7's built-in indicators produce output on every bar, but the output is wrong at the beginning. An SMA(200) doesn't produce correct output until the 200th bar, but NT7's SMA indicator produces output on every bar - its just wrong for the first 199 bars. As a hack, they seem to have hard-coded a default value of "BarsRequired = 20" to suppress all indicator plots for the first 20 bars unless BarsRequired is overridden by the indicator code (I seem to recall that this was once a user-configurable parameter).

But this should really be determined by the indicator. If all you are doing is plotting an indicator, you could set "BarsRequired" as appropriate in the indicator code (i.e., in Initialize() for SMA you could put "BarsRequired = Period;", for TEMA you could put something like "BarsRequired = 3 * Period;").

Things get even worse when you feed an indicator output to another indicator or to a strategy. I'm not sure why you would want to take a TEMA of a TEMA of a TEMA of a LINREG, but if they all had periods of 7, the output wouldn't be valid until the 70th bar if my arithmetic is correct. But NT7 would display it starting on the 21st bar, and if you fed it to another indicator or strategy it would feed invalid data from the first bar.

Even goofier, if your indicator produces "no output" and you feed it to another indicator, NT7 will substitute the Closing price of the bar in place of the missing output. This can produce truly bizarre results. And there is no way for the receiving indicator to detect it.

Personally, in my own indicators, I suppress output until the indicator is valid (I just don't call Set()). But once an indicator starts producing output, it never stops, usually repeating the last valid output if a valid one can't be computed.

Bottom line is that many indicators produce bad output at the beginning, the invalid period will vary based on input parameters, the problem is worse if you feed indicators to other indicators, and it's up to you to figure out how long the invalid period is, and either ignore it or write code to suppress it. If you write your own indicators which suppress invalid output, they'll produce bad output if you apply another indicator to them.

[ChuckAllen]

I partially agree...I still think these are bugs...others agree

** Point #1. I agree that the ATR needs at least High & Low to calculate correctly. I just quickly picked it, because SLOPE() is not an option in your GUI. I am sorry...but...because of that...

I would like to report that ATR() does not return a compilation error when the series that is given to it is improper.

That being said, when there is only 1 price in the series...not a High and Low, the expected result, if no compilation error, would be 0. There is no ATR() when both the High and/or Low are the same. If that were the case, at least a line at 0 would show the error of my ways.

** I attached NT-Bug17.Jpg.
Please note that It contains a fairly simple LinReg and a TEMA of the LinReg...but no ATR. I agree that ATR was a poor choice to submit, so I took it out & resubmit this problem.

Note: I tried to optimize the week of 6/7. I used 10 minute bars. See where the Cross Hairs are displayed...The LinReg is 1154, the Tema is 1768. No price is even close to either of them...still way way off.

When I wrote my systems for Long Term Trading, Trade Station, and most others I have worked with...they try to give you something close when you Chart or Backtest. If you cannot fix the logic at the beginning of the charting/optimization period, Then, you should (behind the scenes, Without Charting Or Optimizing, use enough bars so the 1st bar's indicators are correct. If it contains values like the EMAs that carry forward the calculated data...you should allow enough bars to make up for the calculations and the invalid data carried forward. It usually needs about twice as many bars as the formula suggests to completely clear out the garbage.

It seems like all NT does is pull a number out of the hat...for example 20 seems to be it...and skips that number of bars when charting. However, the invalid data is used from Bar #1 for backtesting & optimization.

[NinjaTrader_JoshP]

1. For ATR, it is not as simple as just High - Low. There is more math in it. You can see my previous post with the other math involved. It uses previous bar's prices as well as current bar prices to help determine the value in the most efficient manner possible. The bottom line is still bogus in, bogus out.

Slope is available in the UI. It is in the Miscellaneous category of the Condition Builder.

2. LinReg of a period of 20 absolutely requires 20 bars before the values are relevant. Please consider the math of LinReg. It requires the calculation of the intercept. The intercept is the sum of 20 bars divided by the period. When you don't have 20 bars of data you are still adding that sum divided by the same period of 20. That means you have less bars divided by 20. Say your values were 1 on every single bar. 1+1+1... = 20. 20/20 = 1. Now let us say you only had 10 bars at the point of checking. 1+1+1...=10. 10/20 =0.5. Values are just fundamentally irrelevant till you meet the required amount of bars for calculations. There is no point even looking at it before then.

Then you are still doing massive nesting of TEMA of TEMA of TEMA of LinReg. As kdoren has pointed out, LinReg requires 20 bars based on your period setting. First TEMA requires another 20. 2nd TEMA requires another 20. 3rd TEMA requires another 20. That is 80 bars required before you will have correct values.

The statement about using 20 bars only to skip charting, but still having those bars processed in backtesting/optimization is inaccurate. To clarify, "Min. bars required" property is user settable with a default of 20. This property prevents all strategy processing of those bars. No strategy logic is processed till the bars required requirement is met. This means you will never trade till the bars required is met. Your strategy logic won't even change variable states when this requirement is not met. Those bars are all completely ignored except by the indicators. The indicators use them to try and establish enough data to get a properly calculated value. You can test this by printing the CurrentBar number on the first OnBarUpdate() event. The first event is only started when CurrentBar fulfills BarsRequired.

[ChuckAllen]

I Still Think This Is A MAJOR BUG

I attached this as a Text file and also a simple screen dump...NT-Bug18.Jpg.


I am a contract programmer. I own Allen Capital Management, Inc.
I have been incorporated for about 15 years. I have written between
40 and 50 complete systems. Several can still be seen on line at some
of my customer's sites. I have been programming in C, VB, etc. for
about 30 years.
I have been asked by one company in particular to consider porting all
of their code to NinjaTrader from Trade Station and C/VB. My extensive
testing and considerations will be passed on to them.
I attached the file NT-Bug18.Jpg with a very lengthy explanation.
Basically, notice that the I have the Data Box and Cross Hairs on the
exact same day in both charts. The bottom (bogus) chart is where I
asked for an optimization for the week of 6/7. The top chart used the
same NT code and I requested a chart from the 1st of the year.
The top (valid) chart shows the LinReg as 1062 and the TEMA as 1063.
The bottom (bogus) chart shows the LinReg as 1154 and the TEMA as 1768.
My main concern for my ACM customers, My contract customers, and my own
personal trading is accuracy of your NT product. I hope to get my point
across with this letter.
Josh, This is my wordy explanation.
Before I pass it onto my customers and my contract customers,
I would like to submit this to you and could you please give me the
email address of your owner, president, etc. I would like to offer
this open letter to everyone...from a very concerned NT customer.
I totally agree with everything that was included in your reply.
I'm glad that you finally agree that the bottom line is still...
garbage in, garbage out.
My point is that NT is supplying that garbage because NT is not able
programming wise or quality assurance wise to figure out how to fix it or
cares enough about their customers and their product to address the issue.
There is one Very Simple Glaring Issue.
If you use an indicator that takes X bars to be accurate, NOTHING should
be done with that data until the appropriate number of bars are available.
No Calculations, No Charting, No Backtesting, No Optimization, No Passing
Garbage to other nested functions...NOTHING!!!
The BEST way to work around this is to change the program to obtain enough
data to perform the entire SET of calculation with the proper number of bars
BEFORE doing anything!
That means, if my strategy, indicator, chart, etc. requires 100
bars of data, and I ask for an optimization, indicator, etc. to start on
Jan 1...NT should grab AT LEAST 100 to 125 bars of data before the end user
sees anything. So get at least data from about the previous Sept 1st.
NO CALCULATIONS SHOULD OCCUR ON THOSE first 100 - 125 BARS!!! If you do
use that data for calculations AND there is some indicator similar to EMA
that carries forward data...The calculations from Jan 1 will be closer,
but still potentially way off...due to the garbage being carried forward.
Whatever logic is used, WE THE CUSTOMER should be guaranteed to have valid
proper data from Jan1. The actual chart, optimization, calculations, etc.
should PROBABLY ONLY display or expose data from Jan 1!!! It would be nice
to have an option to chart the Lead Up data.

The Only exception to this would be if the programmer was intelligent
enough to calculate some values with less than adequate number of bars.
For example: If you need a 50 Day SMA, but only have 5 bars...Basically
sum those 5 bars and divide by 5 instead of 50. If I have 10 bars...sum
those 10 bars and divide by 10.
A SubCategory of this statement is slightly different when dealing with
any EMA type of indicator. What is meant by that is Any indicator that
will alter a previously calculated value by just adjusting that value
with Less Than total number of requested bars, instead of a complete
recalculation when enough bars are obtained. That means that this type
of indicator will need as much as 3 to 4 TIMES the number of bars that
one would initially think.
For Example: When a garbage EMA is calculated when less than the proper
number of bars exist, that garbage calculated value is KEPT and used to
further corrupt the indicator until such time as enough bars have been
present to completely remove the garbage from the calculation. With any
type of EMA, this will take at least 2 to 4 TIMES the number of bar
requested by the EMA...depending on the value, nesting, etc.

So far, here are some of what I have gathered from various replies
on this issue:
* Any optimization, backtest, indicator, etc. that I request will be
completely bogus until enough bars have been obtained to completely wash
out the garbage data calculations that has been accumulated.
* It is up to the Customer to write their own code to figure out how many
bars this will take & do special coding to accomplish this.
* None of the strategies that the Strategy Analyzer creates will be
accurate Until we the customer unlocks the code and fixes the Number Of Bars
Required issue.
* Basically, the strategies created by the Strategy Analyzer are only valid
to build the skeleton of the code. Once the code is unlocked, we either have
to start from scratch or do everything in NT code.
* We the Customer cannot request a proper starting date for any chart,
optimization, backtest, etc. and expect accurate results. We must write
special NT code to figure out how many Lead In bars are required...then
ignore all of that Lead In chart data.

Please let me know what NT thinks about these issues before I pass this
onto my customers.

Thanks for the time.
As always, we are all trying to work towards a better trading solution.

[aviat72]

I think you should treat the inbuilt NT system indicators as a starting point and not the end-all.

When you plan to do nested indicators, it is best to write your own code which explicitly takes care of the issues of the training/warm-up period and generates values which make sense.

I do agree that the NT infra could do a better job when it comes to putting default values for nested indicators. The quirks of putting the last value of the instrument, the last value for the indicator just come out of nowhere. Clearly this issue was not thought through completely.

In a lot of ways, NT indicators are "what you code is what you get". They put minimal layers and error checking above what is in the code. I personally like that since it gives me complete control. So if your indicators need 200 bars to stabilize, load the charts so that you have 200 extra bars before you start your strategy.

I have not written a lot of strategy, but my default template for strategies includes a start-date/start-time (or a wait period), before which it does not make any order related decisions. This way I do not have to worry about any instability issues.

However, you have to waste a lot of time and energy discovering these issues, which should have been properly documented with examples.

You have to treat NT as an Operating System. It allows you to do a lot of cool stuff but when it comes to the applications, the bundled stuff is not the most sophisticated and needs to be reviewed for more complex needs.

[aviat72]

Just to add to above:

I originally started writing my own event based multi-threaded trading system. However, to be really useful any system requires visualization tools and I do not have much of a background in that.

NT offers a good combination of both. You get a programing environment and a visualization environment.

All the other goodies like the strategy analyzer etc. are very useful gravy but not really that hard to develop once you have the basic infra in place.

However, every time NT adds a new feature (like the nested indicators) there is a long learning/stabilization curve, since they tend to code first and think through the architecture later. Their development model is too AGILE while it needs to be SPIRAL at least.

However, because the way they are structured, I am hopeful that better things will come.

One major enhancement would be a true-multi-threaded runtime model for the live mode. Right now all the charting etc. goes through a single thread; multi threading seems to be restricted to backtesting/optimization, and data downloads. Once NT can start harnessing multi-core CPUs for live trading it would truly have arrived. And it is not that difficult to program since there are very well defined boundaries between data management and computation.

[NinjaTrader_Ray]

ChuckAllen,

Thanks for taking the time to provide your insight. On the “glaring issue”, I fully understand what you are saying with respect to unstable period of indicator calculations. I further understand the implication in the example with the EMA calculation logic. We are in fact in the process of reviewing each of our system indicators to see where we can make improvements and adding a check in the EMA indicator code to wait until the required number of bars have been seen, run the calculation on the full set of data and continuing on from there makes sense. I will make it a point to see if there are other indicators that can be updated to ensure a better calculated value during the unstable period. Until then, there are already some ideas provided by other contributors in this thread.

On a related matter, you mentioned TradeStation. I am not sure if you have direct experience or not, but out of curiosity, I just checked their indicator logic for EMA and LinReg and their indicator logic is more or less identical to ours meaning, I believe you would have the same problems there…or did you have some way to work around that?

I am the CEO.

[ChuckAllen]

Trade Station Does Not Have This Problem

First off, sorry, I don't go into this forum unless I get an email or something that says there is a new post or reply to one of my posts. Is there a way to get an automatic email or other notification? That would be a really great improvement.

I converted about 10 systems for one of my customers from TradeStation to C/VB.
This same customer recommended NT to me & wants me to verify NT's accuracy and whether we can convert that same C/VB code to NT. When I ran my converted code and systems against TradeStation's trades, my first several trades were off. Its was because I did not load enough bars to generate an accurate indicator result. In my company's other ACM systems, I knew exactly how many bars I needed per trading system & basically hardcoded that number, since they were black box systems that could not be changed. When I converted the TradeStation systems, I did basically an offshoot of what you currently do...I loaded X bars (except I did it behind the scenes for calculation only). Whenever we did a short term optimization/backtest, OR when I looked at the results for the 1st several trades, some were off by a ton. To fix my code to match the trades that were output from TradeStation...all I did was look at the loops, values for the various indicators, and calculated the number of bars required. To make sure, I actually loaded 50% more than what I calculated would be required. My trades were 100% right on vs. TradeStation's. The reason for the extra logic was that the same system was used for various commodities, and the number of bars would vary accordingly. The Extra bars were Never charted or output in any way. In my debugging, I always compared the indicators on the 1st day the user requested against a much longer term indicator calculation, to make sure there was no garbage in my results..That way, they would get consistent trading results for any period requested. The Extra bars were only used to insure the accuracy of the indicators from the 1st day requested by the user.

I do not have the actual code that TradeStation uses. They May just use common sense and return a value that's close...without loading more historical data.
For example: If a 14 day EMA is requested & they only have 5 bars, return a 5 day EMA. The results from my 'Load the proper number of bars' before actual trading, probably would have come extremely close to 'Return A Value As Close As Possible...Given The Limited Number Of Bars' value. So I cannot say their code is actually done right.

The one thing all my customers and I want is accuracy. When we do an optimization or backtest from just May thru June and the same test from Jan thru June...the values on May 1st of both tests should have the Same Indicator Values. If the trades and/or indicators vary in any way, there is a problem and the results are questionable at best.

Thank you very much for your time.

[NinjaTrader_Ray]

This forum does generate notifications against threads you are subscribed to. I believe you can check what your settings are in the "User CP" section accessible from the main navigation bar of this forum.

Now to your response. I believe we are on the same page here. Our system is no different in that you should load X number of bars to ensure the indciators are initalized. This is inline with what you did for TradeStation and the other product you mentioned and also in line with several users who have contributed to this thread.

In the example you provided with TradeStation EMA, their EMA code is identical to ours in terms of its calculation logic. This means that if you load 5 days of data into a 14 day EMA, the ouput would be the same as NinjaTraders output.