Strategy Analyser... Profit per Month inconsistency

[2inthebush]

Hi Jim,

So far, I've been unable to reproduce this issue with the Sample MA Strategy.

Now, for what it's worth, I'll discuss my screen capture attached. This is one example of many while testing my own strategies. What I've found so far is that the PPM results for each individual WF test are mostly correct as per: cumulative profit * (30.5 / # days). Also, the sum of the entire individual results in the net profit column equals the net profit of the overall WF optimization: $2908.

I'd presume that the overall PPM of the WF optimization should be found with the following calculation:
overall WF optimization net profit * (30.5 / # days) = overall WF optimization PPM

So in this case of my screenshot, the calculation would be:
$2908 * (30.5 / 207) = $428.47 PPM

However, NT calculated overall WF PPM as ($320.54).

[tgn55]

Thanks NinjaTrader_Jim, now it starts to become clear.

So, as I guessed in my last post, the reported overall PPM is indeed a weighted average, and it is this that is responsible for the seamingly nonsensical results I reported in post#1 of this thread. Now we know how that summary number is calculated (is this documented anywhere ??), that mystery is solved.

The other part of the "weird math" is I believe due to timing... it's been years since I dug into this, but as I recall, if a trade is made on say the last bar of a period, this very short duration trade's profit or loss effectively gets amplified, and can be reported as a fairly extreme value. Hopefully, the ticket raised might address that issue.

2inthebush ... the math you show is not really evident in your screenshot... but if we assume your total profit of $15,095 is correct, then what you are calculating is a simple average... not a weighted average. The two will (almost) always be different... the exception being if each period of the WF test all had exactly the same number of trades... which is extremely unlikely!

The bottom line is this: The way NT calculates the overall PPM is, well... novel I would contend that it does NOT give a natural interpretation of profit PER MONTH... ie, a metric that is named by reference to a period of TIME. The way the weighted average is calculated means the reported figure simply gives more weight to profit (or loss) in months where more trades are closed in the month. Whatever pros or cons you attach to this way of looking at it...you just need to remind yourself... "this is NOT an average or expected monthly income". It is something else... something about trade density.

Your presumption about how PPM should be calculated is not unreasonable... it just isn't the way Ninja calculates it, Thanks to Jim for clarifying this.

Hope that helps

Cheers,
T.

[tgn55]

PS... NinjaTrader_Jim ... the ref to negative numbers... just to be very clear, it is now obvious to me how this can occur... it is simply an artifact of the weighting process. I show a simple but graphic example using the formula you revealed:



There you have it... a strategy that was overall profitable, had 2 out of 4 winning months, but gets reported as quite bad... negative $38/month. I'd say this is misleading!

I could also equally easily show a strategy that lost money overall, but would be reported as having a positive PPM (If you want to see how, enter the above in Excel, and just change the sign of all the Profit values in column B...) What you will get is we LOST $350 overall... but reports as making $38 profit per month !!

That is nothing if not misleading, counter-intuitive etc... It should be fixed.

Cheers,
T.

PPS... Above, I refer a few places to "month"... but of course in a Walk-Forward run, each test period can be an arbitrary time period. Might even be a year! Or 5 days... So, again, referring to the result as Profit Per Month is just wrong...unless Jim's formula has a missing term.

[2inthebush]

tgn55


Yes, I agree that a some sort of weighted average calculations are likely the cause of the confusion here. I've also seen those WF results in which the overall net profit was negative, but the PPM was positive. Completely useless. Thank you so much again for taking the time to dig into this some more.

After some sleep, I realized that my above attempt at some maths (not a strong skill for me) was a complete mess. So I've edited that previous post to a much simpler description of my screenshot in case somebody else reads this thread in the future.

I suppose there's nothing further we can do about this issue, unless NT could be convinced to change the WF PPM calculations in order to provide more intuitive and useful results for the user. Going forward, I'll simply ignore NT's WF PPM figures.

[NinjaTrader_Jim]

2inthebush, as tgn55 mentions, the numbers add up if you perform the calculation for combined Profit Per Month that is weighted by trades.

tgn55, I understand the sentiment that trying to decipher the calculation for combined Profit Per Month is not what one would expect. I am not sure if this would have any action taken without hearing further impact, but I have submitted a change request for it, and I have also brought it up with Product Management should they decide to take action. I'm also seeing if documentation can be improved to cover this item.

Change Request ID: SFT-5338

[2inthebush]

Thank you for your help, Jim. Maybe a Profit Per Month calculation that is weighted by trades is preferable to some users. However, I'd guess that a simple PPM calculation would be more intuitive and useful for most users. Ideally, users could choose between a weighted or simple calculation. The PPM can be especially useful when comparing WF optimizations based on different optimization/test periods.

[tgn55]

Thanks NinjaTrader_Jim .

So... what about the issue of nomenclature? I think the current formula for calculating the overall PPM in fact has NOTHING to do with "MONTH"... so why is there an "M" in PPM?

What we know:

• it is NOT based on MONTHS
• it is NOT an average profit per trade, in spite of #trades appearing in the formula. Why? Well, you just can't have an average being negative and the total being positive
• it IS based on WF Test Period, but if the user sets up the WF parameters to represent a month - that is entirely by chance

Hopefully, if someone actually thinks about this when writing the documentation, they will come to the realisation the PPM is a total misnomer that can only cause confusion.

As I see it, the only decent solution to this is either a) change the name, or b) change the formula (or perhaps both ;-)

Hope that helps,
T.

[NinjaTrader_Jim]

Hello tgn55,

We submitted a change request, and I brought the matter up with Product Management. At this time there are higher priority development tasks, so we are going to continue tracking impact.

As for documentation, I am seeing if we can add a note to clarify on what combined Profit Per Month shows.

[tgn55]

Thanks Jim. I also have higher priorities ;-) Would be nice to sort this... but I won't be adding anything more.

Cheers,
T.