Bug in NinjaTrader: SA shows incorrect MFE in certain cases

[bltdavid]

I've attached a screenshot that shows 6 trades highlighted on the Trades tab in Strategy Analyzer. It is my assertion that the MFE values for these 6 trades are incorrect.

When a trade is exited via limit order (aka, the trade hits its target) it seems to me that the MFE value should be the same as the profit achieved by hitting the target. And, indeed, in most cases, this is exactly true.

I have found a situation where 1) NinjaTrader shows an MFE for a trade that was achieved only after that trade had exited. And, 2) the MFE value that is shown is higher than the profit achieved by hitting the profit target, ie, it shows a value that was not possible to achieve.

So, the MFE is too high, and is calculated from future bars after the trade had already exited.

With my understanding of MFE, that makes no sense. An MFE represents the most favorable profit ever achievable during the life of that trade -- "life of the trade" being defined by entry and exit times. Because, obviously, after a trade exits, the tracking of the MFE for that trade should cease as well, right? What use is there to show an MFE that represents a profit only achievable after a trade has already exited?

My attached screenshot is from a backtest of a strategy that enters a position with two contracts, using managed mode. The strategy exits the first contract via a limit order (aka, a profit target) and the second contract continues as a runner until the strategy reverses or the session closes.

It is this managed mode scenario of a two contract entry with multiple exits that produces flawed values for the MFE of the first exited trade.

I submit that NinjaTrader's MFE calculation is flawed in this case. For a trade that hits its profit target (aka, exited via a limit order with a limit price, regardless of managed/unmanaged, scale entry or scale exits, etc), an MFE value higher than the profit made by hitting the target makes no sense, and, in fact, should be impossible.

And, on top of that, we have, in this case, a scenario where the MFE shown for a trade is actually coming from calculations on future bars, after the trade has exited.

Can NinjaTrader support confirm this bug?

[sledge]

So you are SURE your runner didn't contribute to your MFE? Like go a few points beyond the first one profit target of 200 ticks?

[bltdavid]

I'm not sure what you're asking. What do you mean by "contributed"?

If the 2nd contract "contributed" (as seen by the values in the MFE column) then I would say this contribution is incorrect.

I can't think of any reasonable explanation why anyone would want a trade's MFE to include calculations from future bars that occur after the trade has exited.

Yes, these were bars only seen by the runner, and naturally they are included in the calculation of the MFE for the runner.

But the first contract hit its target, and exits. That trade is over, it is done.

Each trade is a separate line in the Trades tab. I understand that the 1st target and the runner are connected, but why is the target's MFE the same as the runner's MFE? They clearly did not exit at the same time, so why does the target's MFE become the same as runner's MFE?

Referring to the annotations in my screenshot,

The target "L3073tgt" MFE should be capped by its target profit. It makes no sense to me that *this* trade's MFE be derived from bars *after* it has exited.

[bltdavid]

Hmm, I think the question of "Is this a bug?" will center around the precise technical definition of a "trade".

Reading here, and elsewhere,

http://www.mypivots.com/dictionary/definition/473/maximum-favorable-excursion-mfe

If MFE is the "maximum amount of profit that could be extracted from a trade" then I think the key to understanding the bug question is the definition of "trade".

If a "trade" is allowed to be defined as the position taken by 2 contracts (target+runner) and the "trade" is complete only after the runner exits, then this *does* explain why the target's MFE is the same as the runner. In effect, the MFE for the target is the same as the runner is meant to tell us "if this target had not exited early, it would have made the same profit as the runner".

I submit the above is 2 trades. It is a single position, sure, but it consisted of 2 trades.

Within NinjaTrader, I think the definition of "trade" is more along the lines of the individual lines seen in the Trades tab. Indeed, each line in the Trades tab represents an entry in the Performance.AllTrades collection -- a collection of 100 trades in that object means 100 lines for the Trades tab.

In effect, the definition of "position" vs "trade" are being intermingled a bit.

A) We see target+runner in the Trades tab as 2 trades if the target hits it profit before the runner exits. This is 2 trades to NinjaTrader.

B) We see target+runner as 1 trade in the Trades tab if they exit together.

In both cases, it is 1 position, but with A we have 2 trades, and with B we have 1 trade.

Now, if NinjaTrader wants to argue that scenario A with 2 trades is still the same "position" and thus the runner's MFE is allowed to be retroactively applied to the target (because, hey, if that target had just stayed in the trade, oops, I mean position, with that runner and had not exited early by hitting its darn profit target price), then I think that is an overly generous interpretation that negates the usefulness of the MFE, and I presume it skews the average MFE reported on the Summary tab.

For all intents that I see from the NinjaTrader program and its documentation, a "trade" is a precise definition, a very technical thing -- there is actually a class named "Trade" that is stored in a "TradeCollection". Every Trade object in the "Performance.AllTrades" trade collection results in a single line on the Trades tab in Account Performance.

[In NinjaScript strategy programming, the "Performance" object holds 4 trade collections, the Trades tab represents information coming from the "Performance.AllTrades" collection.]

So, if the argument is, that the 2 lines on the Trades tab showing 2 trades (target+runner) make up a "position" and that is why both trades get the same MFE --- well, that doesn't sound right. That muddy rationale goes against the relatively precise definitions available in the API's trade collection.

The trades in the TradeCollecton have no support for the concept of a "position" consisting of "1 or more trades" which would justify the MFE for all trades on all lines of that position to be made the same.

I think, in the scenario I have outlined, the MFE calculation is incorrect. It is a bug. Now, it's possible that this MFE calculation may be by design, but I think the engineering design choice (if this calculation is intentional) was a poor one.

The MFE (and MAE, too) reported on a single line in the Trades tab should be able to stand on its own and not be affected by future bars of trades reported on following lines. Especially since these futures bars are occurring **after** the trade has exited. That is, it seems like a bug for future bars to impact the MFE of an already exited trade.

Of course, I may be missing something and I just don't understand the subtleties of MFE calculations (which, is certainly possibly), but, after studying this in depth, my general impression is: this seems like a bug, not a feature.

[sledge]

Hmm, I think the question of "Is this a bug?" will center around the precise technical definition of a "trade".

Reading here, and elsewhere,

http://www.mypivots.com/dictionary/definition/473/maximum-favorable-excursion-mfe

If MFE is the "maximum amount of profit that could be extracted from a trade" then I think the key to understanding the bug question is the definition of "trade".

If a "trade" is allowed to be defined as the position taken by 2 contracts (target+runner) and the "trade" is complete only after the runner exits, then this *does* explain why the target's MFE is the same as the runner. In effect, the MFE for the target is the same as the runner is meant to tell us "if this target had not exited early, it would have made the same profit as the runner".

I submit the above is 2 trades. It is single position, sure, but it consisted of 2 trades.

In Market Replay - things can get really funky.

If you have 2 contracts - and enter - and you EXIT @ market (or even a limit) with 1 fill at A and 1 fill at A + .25 that's considered 2 trades and not together..

Talk about destroying your win/loss ratio charts vs win percentage... totally incompatible. You have NO idea where you are in that space.

This has an example of the chart I'm talking about:

http://www.traderrach.com/the-simple-truth-about-profitable-forex-trading-systems/

Anyways, one must scale back to 1 contract and extrapolate manually from there. If there are different exits - I just run another strategy with 1 contract to total that in.

I'm not sure if you used separate named entries for each target.. (that might be a way around it?), or entered as 2 contracts.

[bltdavid]

As seen in the "Entry name" column of the screenshot in my first post, the target and the runner were both entered with the same entry signal "L3073". This was done with a single entry, using a market order,

EnterLong(2, "L3073");

When this entry order fills, inside OnExecution() the code sets the profit target for 1 contract using this code,

ExitLongLimit(1, true, 1, TargetPrice, "L3073tgt", "L3073");

The TargetPrice (calculated elsewhere) is 200 ticks above the entry price. The 3rd argument with the value of "1" means a quantity of 1 contract.

The whole point of this thread is to say that: using a single entry in managed mode with multiple exit orders results in MFE and MAE values that are, arguably, incorrect.

I'm focusing on the simple case: single entry with 2 contracts with 1 target and 1 runner, which only requires the 2 function calls shown in the above code.

This gets complicated very quickly (as you stated) with a single entry order of N contracts where N > 2 and some number of multiple target orders.

I believe the issue is with managed mode: a single entry with multiple exits results in MFE (and MAE) values on the Trades tab which I contend are incorrect.

NinjaTrader will probably tell me to use 2 entry orders, one for each contract. That is, in managed mode, one entry order for each target order that I wish to make.

So, sure, I could do that, but I'm going for the full monty confession from support. I think it's a bug, and I want support to admit that this is a bug. Or I want a very good explanation why NinjaTrader calculations are correct and why my calculations are wrong.

This may sound arrogant, but the incorrect MFE numbers from NinjaTrader for these managed mode trades with single order/multiple exits are misleading and not as useful as correctly calculated MFE values.

A correctly calculated MAE/MFE value should only take into account the bars of the trade for which the trade was "alive". After a trade exits, it seems awfully indefensible to retroactively update the MAE/MFE value of a closed trade because of calculations involving the future bars that *follow* the closed trade. That is exactly what is happening here. Ergo, I strongly contend these MAE/MFE values are 1) incorrect and 2) a bug in NinjaTrader.

[NinjaTrader_CalH]

Hello bltdavid,

I would be happy to look into this for you and test it on my end.

I would need a toy/sample script from you that re-produces this scenario. This will give me a base underlying of what is being submitted and then I could add in debugging prints to get a better idea of what is occurring in the background.

[bltdavid]

Done, see attached.

Compile and run strategy "bltMACrossover" in Strategy Analyzer where,

EntryQuantity=2
TargetQuantity=1
TargetTicks=<something reasonable>

which represents a 2 contract entry order with 1 target and 1 runner.

Most any instrument and bar series should illustrate the problem. For example, try using ES on a 1-Minute bar series for last couple of months. The desire is to generate trades where the target order hits TargetTicks and exits for a profit, and the runner order continues on until reversal or session close.

If the target order hits its price target, there will be 2 lines in the Trades tab, one line for the target order and another line for the runner. But, in this situation, the MAE & MFE values for the target will be the same as the runner; this reproduces and illustrates the problem.

[When looking at the Trades tab, focus your attention on the "Exit name" column. Any exit name with a suffix "tgt" represents a target order which hit its target price.]

I contend that the problem is: for the performance output generated by the attached strategy, looking at the MAE/MFE columns in the Trades Tab, for trades with an exit name ending in "tgt", the MAE/MFE values shown are A) incorrect and B) represent a bug in NinjaTrader.

[bltdavid]

Hi Cal, do you see the same behavior?

Can you confirm the reported values are incorrect?

[NinjaTrader_CalH]

bltdavid,

I am currently having NinjaTrader_Bertrand take a look into this as you have an open ticket with him on the matter. When he returns he will follow up with any information.

[swetrader]

Noticed the same bug

I'm new with NinjaTrader and when I digged into the sim trade stats for the first time today I noticed the same error (to me there is no doubt that this calculation method is flawed), so I totally agree with bltdavid on this and think he's outlined the case clearly in this thread.

Glad to see that you're looking into it Cal!

BR

[NinjaTrader_Bertrand]

bltdavid,

to clarify I'm quoting an important part from your post how to reproduce -

'If the target order hits its price target, there will be 2 lines in the Trades tab, one line for the target order and another line for the runner. But, in this situation, the MAE & MFE values for the target will be the same as the runner; this reproduces and illustrates the problem.'

The reason you see this expected behavior in managed mode is that your script currently does not scale in to scale out later (as our official tip suggests here - http://www.ninjatrader.com/support/f...ead.php?t=3751).

You will need to use this structure to have two separate trades for the reporting of correct values (NT views a trade as completed buy / sell transaction).

[NinjaTrader_Bertrand]

I'm new with NinjaTrader and when I digged into the sim trade stats for the first time today I noticed the same error (to me there is no doubt that this calculation method is flawed), so I totally agree with bltdavid on this and think he's outlined the case clearly in this thread.

Glad to see that you're looking into it Cal!

BR

BR, can you please shed some more light on your report - where do you see this exactly, in backtesting a NinjaScript strategy or in review of your manual account performance figures?

[bltdavid]

But the MAE/MFE values being reported are flat wrong.

Are you saying they are being calculated correctly?

I understand that my 2 function example seems to abuse managed mode (m/m), but the m/m dance between entry/exit still happens correctly in the strategy.

My point is: why is it acceptable for *this* use of m/m to report incorrect MAE/MFE numbers?

Why are these incorrect MAE/MFE values allowed to stand and be passed off as "expected behavior"?

Are you saying NinjaTrader deliberately designed the product to produce incorrect results?

There are industry standard definitions for these terms.

Expected behavior is not the same as correct behavior.

I submit that the behavior of NinjaTrader, in this scenario, is incorrect.

Can you admit that the MFE values, as reported, are wrong?

[swetrader]

BR, can you please shed some more light on your report - where do you see this exactly, in backtesting a NinjaScript strategy or in review of your manual account performance figures?

Hi Betrand,

Under Account Performance -> Trades -> MFE column.

I sold two contracts in the ES at 2099.50 and exited contract 1 at my first PT at 2097.25 and got stopped out of my runner at 2096.25. On both contracts, however, the MFE is 6.75 in the MFE column. So apparently, the MFE for the first contract is based on the movements of my second contract after the first contract is closed, which obviously makes no sense, leading to flawed statistics and conclusions.

Thank you for your help with this.