Backtests forget summary info and recalculate differently when requeried

[markdshark]

(Title should read "...forget Analysis info..."

I've been living with this problem for a while and thought I'd post about it in case it's a known and well understood issue or better yet can be fixed. Currently using the latest and greatest version of Ninjatrader but AFAIK this problem has always existed.

Essentially once I complete a backtest (I use tick replay in my tests due to how I want the indicators to calculate), I can review the Summary, Analysis etc tabs as I wish. However even though I save my workplace it seems the Analysis reports are discarded after a while (and almost certainly on restart) and if I wish to go back and look at an Analysis tab for a previous backtest the strategy goes into a "Generating report" mode which completes much faster than the original backtest and in which it appears as though the "tick replay" instruction is forgotten as the new results differ wildly from the actual original strategy test results. This is always the case and there's no way of knowing if when selecting the Analysis tab I will see the old chart or if the strategy will start regenerating. And if it's regenerating I know the result will be useless so I have no option but to re-run the backtest once the report generation is complete. Often/usually I have no choice but to shut down the platform and restart it so I can abort the useless report generation and run the strategy again without wasting more time.

Just wondering why this situation exists.

[NinjaTrader_ChelseaB]

Hello markdshark,

There are Strategy Analyzer logs that save a snapshot of the script code and properties so that you can re-retrieve the results.

https://ninjatrader.com/support/helpGuides/nt8/backtest_logs.htm

However, this does re-run the backtest. As long as the data hasn't changed the backtest results should be the same.
If this results are different from loading the strategy analyzer log, this might imply the historical data has changed.

Note, when using Calculate.OnEachTick/.OnPriceChange, you will also want to implement a 1-tick series for order fill accuracy.
Below is a link to a forum post that details.

https://ninjatrader.com/support/forum/forum/ninjatrader-8/strategy-development/94098-isfirsttickofbar-vs-onbarclose-for-backtest-live?p=773377#post773377

[markdshark]

Thanks Chelsea. I love the strategy analyzer logs feature and I've occasionally used it to go back in time to a point in my code where it performed better than after I made a bunch of "improvements". Very useful to have both the code and the test results from that point in time.

That said I do use a 1-tick series and there's no reason that I can think of to conclude that the historical data has changed.

Mark

[NinjaTrader_ChelseaB]

Hello Mark,

With the same code, exact same data, and same parameters, I would expect the same results.

It may be necessary to write out the data and condition values to text file to see where the differences are coming from.

Below is a link to a forum post with an example script that has sample code.

https://ninjatrader.com/support/forum/forum/ninjatrader-8/strategy-development/100192-comparing-real-time-historical-and-replay-performance#post100192

May I have you add this debugging code and provide the text files from both tests so I may assist in analyzing?

[markdshark]

Thanks for the link. I ran it again and it this time it generated the same result so there's definitely some confusion on my end. I'll investigate further and follow up on your suggestion if and when I find it occurring again. Thanks!

One quick question regarding the Strategy Analyzer logs. Is it not possible to select multiple logs to remove at one time? None of the expected key modifiers (<ctrl>, <cmd>, <Shift> etc) seem to allow for that.