exit on session close inconsistent in backtests

[NinjaTrader_Kate]

Hello 2inthebush,

Thank you for your reply.

Yes, but before you do so I would advise running a test with the Sample MA Crossover on the same time frame and instrument you were testing with previously to ensure you're seeing results that match between the top and bottom of the Strategy Analyzer.

After you've confirmed whether you are seeing results as expected, you can shut down NinjaTrader and open your old and new NinjaTrader 8 folders. You'll want to pull the strategyanalyzerlogs folder over from the old to the new folder. Once complete, restart NinjaTrader and you should be able to access those prior runs.

Please let us know if we may be of further assistance to you.

[2inthebush]

I've completed running several Analyzer tests on the updated NT platform, using Sample MA Crossover as well as my own strategies. I've tested with time-based bars and range bars. The same issues persist with inconsistent data between the upper and lower results windows. Specifically, the issue arises when I return to previous test results after running a subsequent test. Immediately after completing a test, the results data are the same. However, after running a subsequent test and then returning to the previous test results, the previous test results will now display inconsistent data.

Fortunately, the "exit at session close" issue appears to have been resolved.

[NinjaTrader_Kate]

Hello 2inthebush,

Thank you for your reply.

Would you be able to supply a screenshot showing inconsistent results on an optimization using the Sample MA Crossover? Also, please supply screenshots of all the parameters set for the strategy that produced those results so I may try to reproduce on my end.

Thanks in advance; I look forward to assisting you further.

[2inthebush]

I've uploaded screenshots per your request. In this instance, I ran this test twice with the exact same settings. Same issue with inconsistent data when I returned to the previous test results.

[2inthebush]

I tried another experiment, running two identical standard backtests on Sample MA Crossover. At first, there were no issues when I switched back to the previous standard backtest results. Then I opened previous test results from an optimization test. Then I returned to the Sample MA Crossover standard backtest results. Now, these results displayed the inconsistent data issue (see screenshot). So it seems that the issue arises when switching from a set of optimizer test results to another set of test results.

[2inthebush]

After running approximately 40 walk-forward optimization tests, I have not seen any issues with inconsistent data in those test results.

[NinjaTrader_Kate]

Hello 2inthebush,

Thank you for your reply.
:
This is very helpful. Could you clarify with the exact steps necessary to reproduce? You're running a backtest or an optimization? Do you run it, switch to a prior result, and then run it again?

Thanks in advance; I look forward to assisting further.

[2inthebush]

So far, I've reproduced these issues with optimization and standard backtests. I haven't seen these issues arise in walk-forward optimizations, however.

These are the steps to reproduce the issue:

• run an optimization test. When the test completes, the results in the upper and lower window will match. The most recent test always has matching results between the upper and lower windows
• run a few more optimizations, either with the same settings or different settings. The settings appear to be irrelevant. Run a few standard backtests as well if you want to reproduce the issue with standard backtests
• open the prior optimization test results and view each test result summary. Many of these results (not all) will now show the discrepancy between the upper and lower results windows. After viewing the optimization test results, open the results of the standard backtests. Now some of these will likely show discrepancies in the summary results as well.
• restarting NT doesn't fix any of the discrepancies, once they have appeared

[NinjaTrader_Kate]

Hello 2inthebush,

Thank you for your reply.

That's also very helpful, thank you for the steps. I will do some testing and get back to you with whether I am able to reproduce the behavior.

Thanks in advance; I look forward to assisting you further.

[2inthebush]

Any update on reproducing these issues?

[NinjaTrader_Kate]

Hello 2inthebush,

Thank you for your patience.

Thus far I have not had luck reproducing this using the directions provided. When you are opening prior optimization/backtest results, are you opening them from the log pane of the Strategy Analyzer or in a new window or tab?

Do you have any backup software running that could be interfering with saving files?

Thanks in advance; I look forward to assisting you further.

[2inthebush]

Months ago, I'd excluded the NT8 folders from my backup software, specifically to avoid potential conflicts.

Usually, I open a new Analyzer tab then open a test result from the log in the bottom window of Analyzer to view those results in that new tab. Sometimes I simply open the test results within the same Analyzer tab in which a ran my last test.

Since you'd asked about that, I decided to test some other methods. I experienced some more odd behavior when right-clicking on a test result in the bottom log window. If I right-click and choose "open in strategy analyzer tab" or "open in new strategy analyzer", then those results will open in a new tab or new analyzer window as expected. However, once that new tab or window opens, then NT automatically runs a standard backtest on those results. This is especially odd considering that the results were initially an optimization or walk-forward optimization test.

With that said, I'll specifically reiterate my experience with the discrepancies within the test results:

• If I open the previous test results in the current tab, new tab, or new Analyzer window the discrepancies will remain consistent within all of them.
• Closing and reopening Analyzer does not change the discrepancies at all. They will remain consistent.
• Closing and restarting NT8 does not change the discrepancies at all. They will remain consistent.
• restarting my computer does not change the discrepancies at all. They will remain consistent.

[NinjaTrader_Kate]

Hello 2inthebush,

Thank you for your reply.

I have been wholly unable to replicate this behavior. If you can provide a strategy that this behavior is routinely replicable on, I'd be happy to try that, but I have tried testing on multiple instruments and time frames and am not seeing prior results changing at all.

Thanks in advance; I look forward to assisting you further.