Announcement

Collapse
No announcement yet.

Partner 728x90

Collapse

StdDev implementation blowing up

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    StdDev implementation blowing up

    I am running into an odd issue since I had moved an indicator to processing off OnMarketData.Its easiest described in a picture.

    The Cyan line is an expected mean value, with which I am calculating standard deviation bands from, and at some point in the data the std deviation calculation "blows up". Now they aren't going to NAN or near double.epsilon, but they become incredibly large. Now when I had this indicator running as a simple lookback version in OBU, I never had any issues with this.

    Click image for larger version  Name:	stdDevBlowUp.png Views:	0 Size:	99.2 KB ID:	1134621

    Code wise it looks like this. Now this is trimmed down code. Is there some catch that I am missing?

    Code:
    predictedMeanSD = StdDev(predictionOutput, 25);
    
    protected override void OnMarketData(MarketDataEventArgs marketDataUpdate)
    {[INDENT]double prediction = Filter.getPrediction()[0];[/INDENT][INDENT]if (double.IsNaN(prediction) == true)
    {[/INDENT][INDENT=2]//Print("Test");
    predictionOutput[0] = lastOutput;[/INDENT][INDENT]}
    else
    {[/INDENT][INDENT=2]// cyan predictedMean
    Values[0][0] = prediction;
    // secondary series that is the input to the stdDev();
    predictionOutput[0] = prediction;
    double meanSD = predictedMeanSD[0];
    
    Values[1][0] = prediction + meanSD;
    Values[2][0] = prediction - meanSD;[/INDENT][INDENT]}[/INDENT]
     
    
    
     }
    Last edited by BraisedInBlue; 12-31-2020, 01:56 PM.

    #2
    Hello BraisedInBlue,

    By 'they' you are referring to Values[1] having a larger value than you expect?

    It looks like you are adding the prediction with predictedMeanSD[0] and using redundant variables. Are prediction and predictedMeanSD[0] values not what you expect?

    Below is a link to a forum post that demonstrates how to use prints to understand behavior.


    Print these values in each market update.
    Chelsea B.NinjaTrader Customer Service

    Comment


      #3
      So for a little context ill add this context.

      predictedMeanSD = StdDev(predictionOutput, 25);

      My predicted value is fine, (cyan line in the picture), its the result of the SD that blows up, snaps back for a few bars then goes large again. And judging by the oddity of this behavior, I assumed I am abusing some construct of the data synchronization of Series objects. But since I have no access to that source code I cannot debug it.

      But yes, I am referring to Values[1] / Values[2]. They are essentially bollinger bands on the predicted mean value. So they are 1 Std dev +/-.
      Last edited by BraisedInBlue; 12-31-2020, 01:59 PM.

      Comment


        #4
        Hello BraisedInBlue,

        These are being set to your calculated values.

        Print these values.

        Please include the output saved in a text file with your reply.

        For those prints, what values are you expecting?
        Chelsea B.NinjaTrader Customer Service

        Comment


          #5
          Threw in a quick print for you. And I would expect the standard deviation calculation to be a continuous value, not randomly jumping up to 700 every 5-10 iterations. I would note that I am simply using the bundled StdDev implementation from NT.

          This is the very first outputs that occur, so the erroneous value exists on iteration 0.

          Predicted Mean: 3659.75070701344 predicted mean StdDev: 720.261601738847
          Predicted Mean: 3659.85274342656 predicted mean StdDev: 12.5183759476877
          Predicted Mean: 3659.90509559076 predicted mean StdDev: 12.5136035719735
          Predicted Mean: 3659.94606147413 predicted mean StdDev: 12.511166690333
          Predicted Mean: 3659.97970570458 predicted mean StdDev: 719.915055689682
          Predicted Mean: 3660.06907097844 predicted mean StdDev: 11.6027516389269
          Predicted Mean: 3660.13178451524 predicted mean StdDev: 11.5988102958967
          Predicted Mean: 3660.16434863597 predicted mean StdDev: 11.596059385723
          Predicted Mean: 3660.2333303023 predicted mean StdDev: 719.584068888935
          Predicted Mean: 3660.31412533831 predicted mean StdDev: 10.547303707801
          Predicted Mean: 3660.37073297924 predicted mean StdDev: 10.5439316134489
          Predicted Mean: 3660.10775443957 predicted mean StdDev: 719.284607161023
          Predicted Mean: 3660.19881397459 predicted mean StdDev: 9.53101669062211
          Predicted Mean: 3660.38120165264 predicted mean StdDev: 9.52730230462807
          Predicted Mean: 3660.51693490619 predicted mean StdDev: 9.51995880345425
          Predicted Mean: 3660.65720303036 predicted mean StdDev: 719.028643028541
          Predicted Mean: 3660.85134648068 predicted mean StdDev: 8.67453234090441
          Predicted Mean: 3660.96844567712 predicted mean StdDev: 8.66742550870946
          Predicted Mean: 3661.05312724812 predicted mean StdDev: 8.66321692152365
          Predicted Mean: 3661.02520867491 predicted mean StdDev: 718.812352739385
          Predicted Mean: 3660.8709438097 predicted mean StdDev: 7.96708102311503
          Predicted Mean: 3660.69543089913 predicted mean StdDev: 7.97228929879988
          Predicted Mean: 3660.52369212124 predicted mean StdDev: 718.615324395858
          Predicted Mean: 3660.40890598664 predicted mean StdDev: 7.35343538694591
          Predicted Mean: 3660.26804992934 predicted mean StdDev: 7.3573400794323
          Predicted Mean: 3660.09467552978 predicted mean StdDev: 7.36222267355169
          Predicted Mean: 3659.91058565243 predicted mean StdDev: 718.438063126345
          Predicted Mean: 3659.71193623492 predicted mean StdDev: 6.8553777927657
          Predicted Mean: 3659.43493454851 predicted mean StdDev: 6.86237430146111
          Predicted Mean: 3659.22078769942 predicted mean StdDev: 6.87248661902968
          Predicted Mean: 3658.95654230675 predicted mean StdDev: 718.268069406164
          Predicted Mean: 3658.66703545982 predicted mean StdDev: 6.40647097325306
          Predicted Mean: 3658.40272130674 predicted mean StdDev: 6.41764379056032
          Predicted Mean: 3658.1365719219 predicted mean StdDev: 6.42826462976662
          Last edited by BraisedInBlue; 12-31-2020, 02:23 PM.

          Comment


            #6
            And after enough iterations it breaks down (StdDev).

            Predicted Mean: 3688.09821876968 predicted mean StdDev: 8990.16341063635
            Predicted Mean: 3688.11429922075 predicted mean StdDev: 9019.16185405334
            Predicted Mean: 3688.12984483542 predicted mean StdDev: 8990.16402712728
            Predicted Mean: 3688.14265074922 predicted mean StdDev: 9019.16289447769
            Predicted Mean: 3688.15522532203 predicted mean StdDev: 9019.16289447769
            Predicted Mean: 3688.17017340123 predicted mean StdDev: 8990.1646489946
            Predicted Mean: 3688.18510779647 predicted mean StdDev: 9019.16333233601
            Predicted Mean: 3688.196669421 predicted mean StdDev: 8990.16464902041
            Predicted Mean: 3688.20520980456 predicted mean StdDev: 9019.16375476014
            Predicted Mean: 3688.21655780363 predicted mean StdDev: 8990.16464906401
            Predicted Mean: 3688.22900900225 predicted mean StdDev: 8990.16464906401
            Predicted Mean: 3688.23919223022 predicted mean StdDev: 9019.16512712993
            Predicted Mean: 3688.2520159158 predicted mean StdDev: 8990.16560110902
            Predicted Mean: 3688.26533098215 predicted mean StdDev: 8990.16560112819
            Predicted Mean: 3688.27949394217 predicted mean StdDev: 9019.16555682737
            Predicted Mean: 3688.2914582784 predicted mean StdDev: 9019.16555682737
            Predicted Mean: 3688.30360930212 predicted mean StdDev: 9019.16555682737
            Predicted Mean: 3688.31600251061 predicted mean StdDev: 9166.23137866925
            Predicted Mean: 3688.32728518052 predicted mean StdDev: 9137.69774566811
            Predicted Mean: 3688.33830152055 predicted mean StdDev: 9166.23225085589
            Predicted Mean: 3688.35176617417 predicted mean StdDev: 9137.69819706821
            Predicted Mean: 3688.36352110054 predicted mean StdDev: 9137.69819708898

            Comment


              #7
              Hello BraisedInBlue,

              predictionOutput is the input series?

              This is being set to lastOutput on each market update only in real-time?

              Are all of the predictionOutput and lastOuput values expected values on every bar in real-time and every market update?
              Chelsea B.NinjaTrader Customer Service

              Comment


                #8
                That is correct, predictionOuput (Series<double>) is the input to the StdDev. And the print I posted shows that they are reasonable continuous price values.

                Last output is passed as an input only on the error case that the predictor returns a NaN. LastOutput is the input to the predictor and is just a filtered version of the price data.

                These values are only updated after so much time has elapsed. Its not shown in the code, but this prevents the updates from iterating too many times per bar. Basically, these values (prediction and prediction SD) should get updated / processed every 1/4 delta time (1 min bar, so 15 seconds).

                The printed values I posted are predictionOutput[0] and predictionMeanSD[0]. So by my understanding you should be seeing the latest input and latest output to the StdDev indicator.

                Comment


                  #9
                  Hello BraisedInBlue,

                  So something in how the input series predictionOutput is being is set is causing the issue?
                  This is where your focus should be.

                  Specifically on a specific bar, what value is set for predictionOutput and what value are you expecting?
                  Chelsea B.NinjaTrader Customer Service

                  Comment


                    #10
                    Actually, I believe I am causing something to break down in the Series bar synchronization. Probably because I am using something in a way that was not intended or permitted. I have gotten things to mostly work when connected to the real data source and using tick replay, but using market replay runs into problems.

                    I am not saying its a bug, I believe its far more likely I am mis-using something.

                    It looks something like this:

                    InputSeries (Close[]) -> Filter1()

                    Filter1Series -> StdDev1()

                    Filter1Series & StdDev1Series -> Predictor()

                    PredictorSeries -> StdDev2()

                    Plot 1 <- PredictorSeries
                    Plot 2 <- PredictorSeries + StdDev2Series
                    Plot 3 <- PredictorSeries - StdDev2Series

                    Now because of the input flow, I have to put delays in between steps to allow the stages of the pipeline to prime before feeding data to the next piece. But all of this is occuring within OnMarketData, and thus is being executed multiple times for each Bar. So I am repeatedly updating the [0] index of each series.
                    Last edited by BraisedInBlue; 01-01-2021, 10:02 AM.

                    Comment


                      #11
                      I think I finally may have figured out what was going on, a combination of me misunderstanding how data is getting loaded compounded by an iterative algorithm. When you pull up a chart of a date range, the chart will load data before the range specified, (not sure on the data source). When you do this with market replay, you need to have data loaded (downloaded) in the days prior to your chart's range. It seems that if you dont, NT is loading some data set ( Historical data?) that isn't compatible with tick replay / market replay. It looks like this data was causing some inconsistent data to enter my algorithms and was also causing the inconsistencies with StdDev.

                      As soon as I loaded more data (the data in front of the date range I was testing under) the issues/inconsistencies disappeared.

                      Comment


                        #12
                        Hello BraisedInBlue,

                        Yes, historical data is downloaded for the Days to load automatically from NinjaTrader's Historical Data Servers when using a live license key on the Playback connection.
                        The historical data downloaded will be for the bar type of the chart, either tick, minute, or day.

                        Historical data is loaded and processed before real-time or playback data begins streaming.


                        Use if (State == State.Historical) return; in any scripts you do not want to process historical data.
                        Chelsea B.NinjaTrader Customer Service

                        Comment

                        Latest Posts

                        Collapse

                        Topics Statistics Last Post
                        Started by Geovanny Suaza, 02-11-2026, 06:32 PM
                        0 responses
                        596 views
                        0 likes
                        Last Post Geovanny Suaza  
                        Started by Geovanny Suaza, 02-11-2026, 05:51 PM
                        0 responses
                        343 views
                        1 like
                        Last Post Geovanny Suaza  
                        Started by Mindset, 02-09-2026, 11:44 AM
                        0 responses
                        103 views
                        0 likes
                        Last Post Mindset
                        by Mindset
                         
                        Started by Geovanny Suaza, 02-02-2026, 12:30 PM
                        0 responses
                        556 views
                        1 like
                        Last Post Geovanny Suaza  
                        Started by RFrosty, 01-28-2026, 06:49 PM
                        0 responses
                        554 views
                        1 like
                        Last Post RFrosty
                        by RFrosty
                         
                        Working...
                        X