Correlation Indicator - How to avoid comparison of different timestamps

[KirkHammett]

Hi,

currently, the Correlation Indicator is practically useless because the secondary bar series is processed after the primary bar. So the indicator compares the current price of the primary bar series with the price of secondary bar series of one bar ago since the current secondary bar series is not loaded. So the indicator is comparing different timestamps and therefore lagging.

I changed the code of the indicator to make it processes the data only in BarsInProgress == 1 to make sure that the comparison takes place only when the all data is loaded to compare the prices at the same timestamp. I tested it by adding the same instrument like the primary series to see if the value is always 1. However, I am still getting sometimes a value of 0 although it should always be 1 since I am comparing the the exact same instruments / prices.

Here is the code of the indicator (everything remained is unchanged).

protected override void OnBarUpdate()
        {


            if (BarsInProgress == 1)
            {
                if (SessionIterator == null || CurrentBars[0] < Period || CurrentBars[1] < Period)
                    return;
                if (Bars.BarsType.IsIntraday && !SessionIterator.IsInSession(Times[0][0], true, true))
                    return;

                avg1 = SMA(BarsArray[1], Period)[0];
                avg0 = SMA(BarsArray[0], Period)[0];

                //Print(String.Format("{0}; {1}; {2}", avg0, avg1, avg0 == avg1));

                double nominator = 0;
                double denominator1 = 0;
                double denominator2 = 0;

                for (int i = 0; i < Period; i++)
                {
                    nominator += (avg0 - Inputs[0][i]) * (avg1 - Inputs[1][i]);
                    denominator1 += (avg0 - Inputs[0][i]) * (avg0 - Inputs[0][i]);
                    denominator2 += (avg1 - Inputs[1][i]) * (avg1 - Inputs[1][i]);
                }

                double denominator = Math.Sqrt(denominator1) * Math.Sqrt(denominator2);

                Value[0] = denominator.ApproxCompare(0) == 0 ? 0 : nominator / denominator;
            }
        }​

I use 60minutes and the same trading hours template. Still I am getting sometimes a value of 0 instead of 1. Why is that so and how can I solve this problem?

Thanks in advance!

[NinjaTrader_ChrisL]

Hello, thanks for writing in. This is how multi-time frame script work, unfortunately, there is no changing this. This has been asked here in the past with an explanation of what is happening:

https://forum.ninjatrader.com/forum/ninjatrader-8/indicator-development/1108443-correlation-indicator-instrument-vs-itself

You can either compare the latest value of BarsArray[0] to the latest value of BarsArray[1] (wich is 1 bar behind) or you can produce a value of 1 by comparing BarsArray[0][1] (one bar ago) to the latest value of BarsArray[1][0] which, again, is already 1 bar behind, so it produced the "expected" value of 1.

[KirkHammett]

Hi Chris,

thanks for your response. I still don't really understand why the multi-timeframe script is a limitation here because I can execute my code in BarsInProgress == 1. I have changed to code to execute everything in if BarsInProgress == 1. This means that both the primary bar series and the secondary bar series are loaded. Still I am getting a 0 sometimes although it should always be 1. I thought that it might be some indicator processing logic because Jim was saying in your link that plots are synced with the primary series, but I have the same problem when trying in a strategy.

Here is what I tried in a strategy (not in the indicator):

if (BarsInProgress == 1)
            {
                double avg1 = SMA(BarsArray[1], Period)[0];
                double avg0 = SMA(BarsArray[0], Period)[0];

                double nominator = 0;
                double denominator1 = 0;
                double denominator2 = 0;

                for (int i = 0; i < Period; i++)
                {
                    nominator += (avg0 - Inputs[0][i]) * (avg1 - Inputs[1][i]);
                    denominator1 += (avg0 - Inputs[0][i]) * (avg0 - Inputs[0][i]);
                    denominator2 += (avg1 - Inputs[1][i]) * (avg1 - Inputs[1][i]);
                }

                double denominator = Math.Sqrt(denominator1) * Math.Sqrt(denominator2);

                double mycorrelation = denominator.ApproxCompare(0) == 0 ? 0 : nominator / denominator;

                Print(String.Format("{0}; {1}", Time[0], mycorrelation));
            }​

I added the exact same instrument like the primary series. In the prints mycorrelation is still 0 a couple of times which should not be possible. To me this looks like a bug. Are you able to reproduce?

Thanks for your help!


Edit: I did some further testing and the problem is the following code: "denominator.ApproxCompare(0) == 0 ? 0". When I only set mycorrelation = nominator / denominator the correlation is always 1 and therefore correct! I also don't really understand why you need that in the correlation logic but this is definitely causing the problem! Are you abel to reproduce?

[NinjaTrader_ChrisL]

Hi, this function is to prevent division by zero. It is used because of the quirks that computers have with double precision/float values. It is not a bug.

[KirkHammett]

Thanks for the explanation. But still, could you please test the code I provided in my post above? This function causes the correlation to be 0 although it should definitely be 1. If there is no problem with this function, why is the correlation sometimes 0 although it should definitely be 1?

[NinjaTrader_ChrisL]

Hello, thanks for the follow up. You can print out the value of the denominator to see what is is. This should be a sufficient explanation of the output you are seeing.

double denominator = Math.Sqrt(denominator1) * Math.Sqrt(denominator2);

Print(denominator);

[KirkHammett]

I did the following in the indicator with using the approxcompare functino

if (Value[0] == 0)
                    Print(String.Format("{0}; {1}; {2};  {3}", nominator, denominator, nominator / denominator, Value[0]));​

Here are the results:
4.48228494427172E-23; 4.48228494427172E-23; 1; 0
1.58818677610181E-22; 1.58818677610181E-22; 1; 0
7.50666405891873E-23; 7.50666405891873E-23; 1; 0
1.39586728368323E-22; 1.39586728368323E-22; 1; 0
1.58818677610181E-22; 1.58818677610181E-22; 1; 0
6.15577472104306E-22; 6.15577472104306E-22; 1; 0
9.19566347217544E-22; 9.19566347217544E-22; 1; 0
2.95365517227372E-21; 2.95365517227372E-21; 1; 0
3.26090107104889E-21; 3.26090107104889E-21; 1; 0
3.77442513507947E-21; 3.77442513507947E-21; 1; 0
4.22249853314178E-21; 4.22249853314178E-21; 1; 0
3.97046694025453E-21; 3.97046694025453E-21; 1; 0
4.56173305716802E-21; 4.56173305716802E-21; 1; 0
4.56173305716802E-21; 4.56173305716802E-21; 1; 0
4.9140732240244E-21; 4.9140732240244E-21; 1; 0
4.4559573363378E-21; 4.4559573363378E-21; 1; 0
5.02512222125964E-21; 5.02512222125964E-21; 1; 0
5.02512222125964E-21; 5.02512222125964E-21; 1; 0
5.65807048622756E-21; 5.65807048622756E-21; 1; 0
7.54082403327775E-21; 7.54082403327775E-21; 1; 0
5.89753927356166E-21; 5.89753927356166E-21; 1; 0
5.65807048622756E-21; 5.65807048622756E-21; 1; 0
5.62848585463172E-21; 5.62848585463172E-21; 1; 0
5.08111200897183E-21; 5.08111200897183E-21; 1; 0

So both the nominator and dominator are close to 0 but not 0 and the Value[0] is 0 although it should be 1. To me this looks like a problem in the logic of the code but maybe there is a good explanation for it. Still, I think it is very obvious that the Value[0] being 0 is not correct.

I understand that the logic intends to solve the problems when dealing with very little numbers but the Value[0] being 0 is simply wrong. Maybe there is a better way of solving this problem. So I would rather say that the logic makes sure that the correlation value is 99% correct but in some cases if the numbers are very small, the indicator may give wrong results.

[NinjaTrader_ChrisL]

Hi, the epsilon value for ApproxCompare() is 1E19, so these values are close enough to zero for ApproxCompare() to return 0.

[KirkHammett]

Understood! Still, if both the nominator and the denominator exceed E19 like in my case it makes no sense to return 0. We are having a division (nominator / denominator) to calculate the correlation. A very small number divided by another very small number is not 0 unless one of the small numbers is indeed 0. The result of the division is not even close to 0. This is why to me this logic doesn't seem quite right. I don't know if this is something you want to address with your development team, since it gives wrong results in certain cases.

Still, thanks for helping me out in understanding the issue!