Strategy Analyzer is terrible

[mbrandau]

Can someone please explain why the Strategy Analyzer is so bad? Does NT understand that we pay between $1000 - $1600 for this platform. In my case I also have the multi-broker lifetime license. Please explain why the Strategy Analyzer is so bad.

Today, on a current auto trader I had built, I used the Optimization option to target which setup I should use. I posted a pic of the results after 5 hours of it running. It says with the top option, if I ran those configurations, I would make $36k between 10/1/21 - current. It says 53% winning %. So I ran that exact configuration on Replay and the reality is 36% winning % and a loss of $14k for the same exact period. Tell me how this happens. Why is your platform so unreliable for the money we spend on it.

This is just 1 example. A while back I tested something else and it claimed I would have made $1.7 million since Jan 1 2021. Reality is several thousand in losses per week. WHY??

I'm more than just a little annoyed at this point.

[DTSSTS]

the Backtest is always much better than the Playback, Dependent on what type bar you are back testing you need to consider what the slippage should be in your testing as it has a horrific effect on results. Also if strategy is running OnPriceChange the results are still OnBarClose and your entries and exits are on the Open of the Next Bar. Some of the weird bars the Open is actually NOT the real open, so again slippage with that type bar would be even worst. On you Optimization, 2 things do not try to optimize so many different settings at one time and ALSO do not optimize a year of data, try something like 45-60 days, Good Luck

[mbrandau]

Not even close, the strategy analyzer is not even close to reality. I was using the 2 min HA bars, on bar close and it was only 3 months of test. Absolutely ridiculous. We should be refunded our money for this platform. I have used the analyzer dozens of times and none of them, on any data set, in any example, were anywhere close to reality

[DTSSTS]

How much slippage are you testing with the HA Bar, That would be likely the hardest bar to Backtest with the variable bars size the slippage would not be consistent either. I would think you should test on Live data and might also want a secondary data series for you exits or maybe a trailing stop

Live Data is likely the only way to test HA

[mbrandau]

Whatever the slippage default is. Didn't change it

[mbrandau]

But let's be clear here. Doesn't matter what bar, time or renko I am using. All terrible.

[NinjaTrader_ChelseaB]

Hello mbrandau,

It is possible to increase backtest fill accuracy with 1-tick intra-bar granularity and TickReplay.

Below is a link to a forum post that details.
https://ninjatrader.com/support/foru...377#post773377

As well as a link to a forum post that demonstrates comparing real-time, playback, and backtest by writing the values used to file.
https://ninjatrader.com/support/foru...nce#post100192

Renko bars specifically cannot use TickReplay and do not backtest well. This is because the open price of the bar changes during the lifetime of the bar.

https://ninjatrader.com/support/forum/forum/ninjatrader-8/platform-technical-support-aa/109640-isfirsttickofbar-bug?p=839636#post839636

https://ninjatrader.com/support/forum/forum/ninjatrader-8/platform-technical-support-aa/1100676-big-difference-in-strategy-performance-using-unirenko-bars?p=1100852#post1100852

[timmbbo]

I agree, it is bad, which is why I do not exclusively rely upon it. Instead, I use the Strategy Analyzer to identify strategies of interest. After that, then I back test them.

[dtaylor]

I can’t find a use for it either. Seams market replay is the only way to get accurate results. You see this from the better vendors as well. They will not show backtest, strategy analyzer results, only market replay. It’s not accurate in any useful sense in my experience.

[NinjaTrader_BrandonH]

Hello dtaylor,

Thanks for your notes.

Testing a strategy using Market Replay data on the Playback connection would be the most accurate way to test how a strategy might behave.

When it comes to backtesting in the Strategy Analyzer, you may use Tick Replay along with an added 1-tick series to have logic processed intra-bar and have orders filled intrabar as noted by Chelsea in post # 7.

Tick Replay would be used to have the logic process OnEachTick or OnPriceChange with historical data, but this does not allow for intra-bar order fills. You would need to add a single tick data series and submit orders to that single tick data series for a strategy that uses Tick Replay. See the links Chelsea shared on post # 7 for more information.