Strategy Analyzer saved backtests losing info with NT 7 beta 17

[jdfagan]

In my attempts to try to reproduce a different error (see here -> http://www.ninjatrader.com/support/f...ad.php?t=15119), I ran into a different issue with Strategy Analyzer that I wanted to report since I was meticulously documenting all my steps anyways for above attempt at reproducing..

Its very likely these set of steps could be reduced to smaller subset, but NT support, please follow these exactly (at least initially to reproduce the undesirable behavior of losing chart/graph/executions/trades/periods info from saved backtests on your end). Please see Austin for copy of the "NG ##-##.txt" I mention below.

Steps to reproduce this error:

• Re-install partially fresh version of NT7 (cleared out the following directories before re-installing NT7 - cache, db, export, import, incoming, log, outgoing, tmp, trace). This step may be optional but for me I had to do this as Strategy Analyzer was very unstable from at least this error before - possibly due to other errors I've received today.
• Disconnect from Zen-Fire if already connected
• Instrument Manager -> add NG ##-## (Nymex version) to Default instrument list
• Historical Data Manager -> UTC 08:00 + Generate Minute -> Choose "NG ##-##.txt" file -> Wait for import to finish -> Ok to close dialog popup
• File -> New -> Strategy Analyzer -> Select NG ##-## in Default List
• Select Backtest (far right) and pin it so it stays open in Strategy Analyzer
• Choose SampleMACrossOver strategy
• Change following params like so: Type = Renko, Brick size = 5, From = 1/1/2000, To = 12/31/2008, Slippage = 1
• Run Backtest - backtest appears to be working properly with Control Center appropriately showing "Running Backtest...". Takes about 1 min for this backtest to work for 2 years worth of tick data feeding the Renko 5 tick chart.
• Click Save toolbar button to save this backtest result to within the Strategies -> SampleMACrossOver node. Use default name presented by NT to save this backtest.
• Select SampleMACrossOver from saved backtests within the Strategies->SampleMACrossOver tree node on left side of StrategyAnalyzer. Oddity note 1: Once you do this step, note the Backtest pane in far right side of Strategy Analyzer now adds a "Instrument(s)" param within the Data series category and assigns it "NQ 09-10". Not sure why its doing this (ok, I do have NQ 09-10 in my default list cause NT comes installed with this in there but as I have yet to connect to Zen-Fire, this will be blank data in this symbol - only one symbol has data at this point - its NG ##-##).
• Run Backtest. Note, it still has NQ 09-10 selected in Instrument. Screen flashes ever so briefly and same backtest results displayed as from the selected saved backtest strategies. I guess the backtest didn't really run at this point because I have selected a "saved backtest" in far left pane. This is bad UI experience IMO.
• Select NQ 09-10 in Default list on left pane
• Leave settings as existed for NG ##-## and run backtest.
• Note the backtest results come back with 0 trades as expected since no data yet exists for NQ.
• Select saved backtest Strategies->SampleMACrossover->SampleMACrossOver
• Click Save toolbar button -> Save backtest as SampleMACrossOver (Slow-25)
• Select saved backtest Strategies->SampleMACrossover->SampleMACrossOver (Slow-25)
• Select NG ##-## in Default list on left pane
• Change Slow param from 25 to 50 -> Run Backtest -> takes about 3 seconds this time to run backtest (nice caching job NT team! very speedy second time around for similar backtest on NG ##-##).
• Click Save toolbar button -> Save backtest as SampleMACrossOver (Slow-50)
• Select saved backtest Strategies->SampleMACrossover->SampleMACrossOver (Slow-25) -> Note the Summary tab trade info changes for this different backtest result - 18044 Total # of Trades
• Select saved backtest Strategies->SampleMACrossover->SampleMACrossOver (Slow-50) -> Note the Summary tab trade info changes for this different backtest result - 11033 Total # of Trades
• Select saved backtest Strategies->SampleMACrossover->SampleMACrossOver -> Should be same exact summary as Slow-25 version.
• Right-click this saved backtest -> Delete -> Yes
• Select saved backtest Strategies->SampleMACrossover->SampleMACrossOver (Slow-50) -> Note the Summary tab trade info changes for this different backtest result - 11033 Total # of Trades
• Select saved backtest Strategies->SampleMACrossover->SampleMACrossOver (Slow-25) -> Note the Summary tab trade info changes for this different backtest result - 18044 Total # of Trades
• Select NG ##-## in Default list on left pane
• Change Slow param from 50 to 20 -> Run Backtest -> takes about 3 seconds this time to run backtest (nice caching job NT team! very speedy second time around for similar backtest on NG ##-##) -> 22303 Total # of Trades
• Select saved backtest Strategies->SampleMACrossover->SampleMACrossOver (Slow-25) -> Note the Summary tab trade info changes for this different backtest result - 18044 Total # of Trades
• Select Chart tab - notice no trades showing up on this saved backtest
• Select Graphs tab - notice no graph; rest of tabs are all blank as well; ditto for other saved backtest - no graphs, trades, etc. in the other tabs
• Select NG ##-## in Default list on left pane
• Leave Slow param at 20 and re-run Backtest -> notice the Chart, Graphs, Executions, Trades, etc. tabs look correct and filled in with info from this backtest run.
• Click Save toolbar button -> Save backtest as SampleMACrossOver (Slow-20)
• Select saved backtest Strategies->SampleMACrossover->SampleMACrossOver (Slow-20) -> Note the Summary tab trade info changes for this different backtest result - 22303 Total # of Trades
• Again, with Slow-20 saved backtest, Graphs, Executions, Trades, Periods all are blank and not showing data like was just shown after doing a backtest but before saving the backtest.

Note: If interested, I can forward you bunch of images I took during the above test path. Too many pics to upload though via this forum - about 14 snapshots I took in my attempt to catch bad behavior. Let me know if that would help.

Whew. Ok, back to trying reproducing original error from different thread..meh.

JD

[NinjaTrader_Jason]

Hello jdfagan,

I have not been able to reproduce these steps - Austin is not in the office yet. I have send him an email to forward me the NG ##-##.txt file.

"I guess the backtest didn't really run at this point because I have selected a "saved backtest" in far left pane."

Unfortunately this is as expected. You will need to select the instrument in the tree node on the left side of the Strategy Analyzer. If no instrument is selected and you re-run the backtest, it will display prior results.

"Note the backtest results come back with 0 trades as expected since no data yet exists for NQ.

Select saved backtest"

When I select 'Save backtest results' while all results display zero, I receive an error that states 'There are no results to save'.

[NinjaTrader_Jason]

Hello jdfagan,

I received the NG ##-## text file and I will reproduce the steps you describe.

Once finished, I will let you know the outcome. Please expect my response tomorrow.

[jdfagan]

simpler set of steps to reproduce

Here's simpler set of steps that I used to reproduce this problem:

1. Re-install partially fresh version of NT7 (cleared out the following directories before re-installing NT7 - cache, db, export, import, incoming, log, outgoing, tmp, trace). This step may be optional but for me I had to do this as Strategy Analyzer was very unstable from at least this error before - possibly due to other errors I've received today
2. Disconnect from Zen-Fire if already connected
3. Instrument Manager -> add NG ##-## (Nymex version) to Default instrument list
4. Historical Data Manager -> UTC 08:00 + Generate Minute -> Choose "NG ##-##.txt" file -> Wait for import to finish -> Ok to close dialog popup
5. File -> New -> Strategy Analyzer -> Select NG ##-## in Default List
6. Select Backtest (far right) and pin it so it stays open in Strategy Analyzer
7. Choose SampleMACrossOver strategy
8. Change following params like so: From = 1/1/2000, To = 12/31/2008, Slippage = 1
9. Run Backtest - backtest appears to be working properly with Control Center
10. Note Summary, Chart, Graphs, Executions, Trades, Periods, Orders, Settings tabs all seem correct immediately after backtest (see attachment 1 for view of graphs tab for example)
11. Click Graphs tab to view Cumaltive Profit graph
12. Click Save toolbar button to save this backtest result to within the Strategies -> SampleMACrossOver node. Use default name presented by NT to save this backtest.
13. Navigate to just saved backtest: Strategies -> SampleMACrossOver -> SampleMACrossOver
14. Note the Graph immediately disappears. Also, Chart (missing trade visuals), Executions, Trades, Periods tabs are also blank/incorrect. Only Summary, Orders, Settings tabs remain in tact for saved backtest. (see attachment 2 for view of blank graphs tab within saved backtest)

JD

[NinjaTrader_Jason]

Hello JD,

Thank you for providing these steps. I am able to reproduce this behavior on my end. However, this as expected behavior - when you save backtest results, it will only save the Summary and Settings-tabs.

You can test it yourself. Perform a backtest and save it using the same steps. Close the Strategy Analyzer and open up a new one. Open the saved results and you will see only data in the Summary and Settings-tabs. However, you steps do display a different discrepancy - data in the Orders-tab will remain to be displayed.

I will forward your findings to Development. Thank you for reporting.

[jdfagan]

but why?

I am able to reproduce this behavior on my end. However, this as expected behavior - when you save backtest results, it will only save the Summary and Settings-tabs.

At the very least the saved backtests (or saved Optimizations or saved Walkforwards) should keep the Graphs tabs at the very least. Not being able to visualize the Equity Curves for past backtest runs kind of defeats the usefulness of saved backtests as one would then need to re-run that backtest to re-visualize these graphs. A typical manual use case for an analyzer is to run a series of many backtests (saving each one that looks promising), tinker with params, run some more, etc. Then go thru the saved backtests in the end to share with a colleague sitting next to him (or via sharing screen via screencast) the set of saved backtests. Having the Graphs tabs be saved is highly useful. Ideally the trades tab too (if not all tabs) would also be saved - i.e., let us worry about the disk space cost of saving backtests - don't limit the usefulness of saved backtests by not saving even half of the tabs of a backtest

JD

[NinjaTrader_Jason]

Hello JD,

I see, I will forward your suggestion to save all other tabs to our Development team and ask them if they can add this to the list of future considerations of the software.

Thank you for your feedback.

[dnoff]

So I am a little confused on the saved results. I can see the summary and also the executions, but there is no trades or graphs??

With Beta 21, what is the expected behavior when I save backtest results??

If trades are not saved, is that a limitation that will be removed or some design/system limitation??

Thanks

[NinjaTrader_Jason]

Hello dnoff,

Unfortunately only the Summary and Settings tabs will be saved.

I am not sure about the specifics of this limitation, however it is on our list of future considerations to save the other tabs as well.