Strategy Analyser... Profit per Month inconsistency

[tgn55]

So... if a walk-forward run shows total net profit of $18,000... how can the profit per month be negative ??

Something weird is going on, and I appreciate that a simple averaging of the individual period PPMs is possibly NOT quite right. But see the attached pics... in one case the difference is huge.

Note the discrepancy is replicated in both the results pane and the log history pane...




If not yet convinced... try these numbers...


This is just nuts.... and yet it seems this is the number ultimately reported in the Summary view

I did some more research... the above weirdness seems to have only happened on a few groups of walk-forwards I ran over the last couple weeks.
Other results look more sensible...it it quite inconsistent.

Finally, here's an example that does NOT include the above nonsense... yet the calculation of overall PPM over the walk-forward period is still unfathomable (to me) and I have a degree in mathematics ;-)



This is really worrying... as I routinely look at the Profit per Month to identify strategies/parameter settings worthy of closer attention.
Right now, my confidence level in the reported statistic is pretty low !

Cheers,
Trevor... the SA quality control dude ;-)

[tgn55]

Mystery partially solved...

After digging deeper, I've found the apparent nonsense profit per month values are an artifact of the walk-forward process. If the strategy puts on a trade on the last bar before the walk-forward test period ends... the time in trade is calculated as effectively zero... or perhaps only a few seconds, which then results in the huge multiplier that impacts profit per month enormously!

Furthermore, it seems in each period, the PPM is probably calculated per trade... not on aggregate profit for the period.

OK... that perhaps explains some more apparently incorrect values, but I still don't get how the numbers work in the last pic above. Remains a mystery, but in view of the fundamental problem with PPM calcs... I probably don't really care any more!

The real takeaway from this intriguing deep-dive is this (I think ;-) : Beware of believing the profit per month numbers on walk-forwards!! They can be seriously skewed by the artifact of chopping up sessions, which triggers the above (FYI, the most extreme case I've seen is 1 trade, profit of $955, got translated to over $29,000 per month. That's not a typo...)

So... lesson is perhaps be guided by Total Net Profit... DO NOT trust Profit per Month on walk-forwards!

Well, that's my conclusion... happy to be corrected.

[NinjaTrader_Jim]

Hello tgn55,

Thanks for your post.

We will be following up after we have been able to review.

[tgn55]

Thanks Jim. The only "fix" that comes to my mind is to calculate PPM differently on walk-forward tests, something more like (profit for the test period) * 30.5 / (length of test period). That would then be a more meaningful metric of walk-forward performance.

Cheers,

[NinjaTrader_Jim]

Hello tgn55,

Thanks for your patience.

I have set up a test on my end using a very simple strategy that enters, and then exits when the Strategy Analyzer finishes backtesting the Test Period. I have a variable added to the strategy to allow it to be optimized, although the variable is completely unused.

When I calculate Profit Per Month, I am getting the results I would expect based off of our help guide: cumulative profit * (30.5 / # days)

I have attached a test script and a demo below.

https://drive.google.com/file/d/1kpbhO_x-tsCcJucLCi4Ht5NBW_0Wlihj/view?usp=drivesdk

I don't see what you are using for your test period, but what is perplexing to me is that in the screenshot linked below, your start date is listed at a date after your end date. I believe this is what is skewing your results which could be an issue with your installation.

https://ninjatrader.com/support/forum/filedata/fetch?id=1069252&d=1567239644

Could you run the same test I am and could you report back if you are able to see the same results I am? Please also provide screenshots of the results so we can verify we are seeing start and end dates like we would expect.

I look forward to hearing back.

[tgn55]

Thanks Jim... I will run your tests and report back in a while... been up most of the night. (It's now 7:00 AM where I am !!)

As for the dates, I don't believe that is the issue... I'm in Australia, so my screenshots are showing d/m/yy format... I suspect that is why you are reading them as whacko!.

When I look at the actual trades, it is clear what is going on... a trade is entered right on the last bar, then walk forward logic closes it out on the same bar. This results in a REALLY short time in trade value... not a month, but literally a few seconds!! And that is what sends PPM through the roof.

Hope that helps...

Cheers,

[tgn55]

Hey Jim, I updated my reply to make it a little more accurate... if you read it real soon after I posted, see above re date formats.

I have another request... can you please look at the last image in my OP #1...
Putting aside how each PPM is calculated (yeah I know about the 30.5 day stuff...not the issue), I'm just assuming the overall number is calculated from the individual test period data.
I think the screengrab pretty much sums up my query...

Maybe my assumption is wrong...but it's does seem odd.

Cheers,

[NinjaTrader_Jim]

Hello tgn55,

I've been able to reproduce the same results you are and confirm the same. "# days" is relative to the time in trade as opposed to the test period. This creates a multiplier like 30.5 for trades made in a single day. I have sent in a request to change this behavior. When I receive a tracking number, I will update this post.

Thanks for bringing this area of improvement to light.

[tgn55]

Yes Jim... the timing messes up the per-test period PPM values.

So let's put that aside for a moment... and just accept the numbers a walk-forward reports for each test period.
The question still remains... How the heck do you get the number for the overall walk-forward... please, look again at the last pic in my original post... it lays it out pretty clearly.

Maybe, if your developers can "fix" the oddball per-period PPMs, this issue will go away... but I'm not sure there isn't something else weird going on.

Cheers,

[NinjaTrader_Jim]

Hello tgn55,

Yes, my apologies for not addressing this question in my last post. This will be more difficult for me to create a test script to reproduce. Could you email me an exported script and a screenshot showing the parameters I can use to reproduce a WFO backtest that displays these results? Please email me at platformsupport [at] ninjatrader [dot] com with the text "Attn Jim." Please also include the thread URL and the ticket ID "2261722."

Thanks in advance, I look forward to investigating further.

[tgn55]

Not too keen to share my script Jim ... maybe this will get sorted if/when the other issue is resolved, and maybe it's just a case of the thing NOT being calculated as I figure it might be.

Attached is the output from running Sample MA Crossover. Still shows similarly puzzling number for the overall PPM ... $697, which is way different than either of the two other possibilities that seem kind of reasonable

PPM Sample SMA X.zip
Cheers,

[2inthebush]

Was this issue ever resolved? I'm experiencing the same issues as the OP, with wild discrepancies in profit per month calculations in walk-forward test results.

[tgn55]

I have not used Walk Forward much since starting this thread...and have also changed my trading style a lot, being more a swing trader now than I was then. Trades that remain open for weeks or even months really are not going to perform "properly" when the time period is sliced up by the WF process, and trades get prematurely terminated mid-trade, so I have even less interest in this now!

AFAIK, nothing has changed - but NinjaTrader_Jim did mention referring this to up and (maybe ?) getting a tracking number... I never saw it.

As for the last pic in my original post... if anyone can explain how the overall PPM figure is derived...given the individual results... they deserve a prize. I did wonder if it is a weighted average, rather than a simple average (which it patently is NOT)... but no explanation was ever given that I know of... and it is scary how misleading the number might be.

Buyer beware...

T.

[NinjaTrader_Jim]

Hello tgn55 and 2inthebush,

I believe I was looking for a specific case showing negative numbers, and was not able reproduce that case.

However, digging back into this, I do have some detail to answer the combined Profit Per Month question. It is calculated as follows:

((Individual Profit Per Month A * Individual Total Trades A) + (Individual Profit Per Month B * Individual Total Trades B)) / (Combined Total Trades) = Combined Profit Per Month

Example:

((3502.24 * 1771) + (25756.72 * 1190)) / 2961 = 12446.12

This example can be referenced against my screenshot which is a bit simplified, but the math for combined Profit Per Month is the same.

The change request was for "Profit per month based on trading days vs calendar month" with the internal ticket ID SFT-3227. This is staying as is until we get enough feedback to change the behavior.

2inthebush, if you have a specific question on individual profit for month calculations, please share an example demonstrating a specific result in question, preferably with a small test using the SampleMACrossover strategy so I can reproduce and do some math.

[2inthebush]

Thank you for the update and the effort you've put into this previously, tgn55. Not sure how much more time I'm willing to devote to trying to solve this myself, but here goes...