Backtest Results Very Inconsistent - Retesting Same Variables, Different Results

[Daveman0]

Hello,

I'm using the strategy analyzer to test out some strategies, and I've been surprised to find that the results for testing the same input variables over the same historical timeframes can give dramatically different results when testing on different days - I'm wondering if there's some explanation for this? I'm quite sure I'm testing the same instrument, same timeframe, and same strategy input variables, and the results are sometimes 50-200% different when I re-test them a few days later - this doesn't give me a great deal of confidence in the results the backtester is presenting - any suggestions for how to make sense of this?

Thank you very much -

[NinjaTrader_Kate]

Hello Daveman0,

Thank you for your post.

When you are using the Strategy Analyzer, are you connected to your normal live data feed or are you connected to the Playback Connection? Being connected to the playback connection while backtesting will cause incorrect results.

Thanks in advance; I look forward to assisting you further.

[Daveman0]

I'm connected to live data feed. Here's an example. I ran this backtest with these variable inputs on May 27th:



Then I ran the same variables today on June 1st for the same time historical period, using the same variables (I changed the order of the variables in the script, but the underlying script is the same), and got completely different results -


This seems extremely strange to me, and makes it essentially impossible to make meaningful decisions around these backtest results - I am also getting consistently different results from what I got a few days ago across a wide range a variable inputs and date ranges...

Any clarification as to the potential source of these discrepancies would be greatly appreciated -

[NinjaTrader_Kate]

Hello Daveman0,

Thank you for your reply.

Does the discrepancy remain after clearing your cache?

To delete the cache, follow the steps below.

• Shut down NinjaTrader.
• Open the Documents > NinjaTrader 8 > db folder.
• Delete the sub-folder named 'cache'.
• Restart NinjaTrader and test.

Please let us know if we may be of further assistance.

[jeronymite]

Daveman0 In addition to the actions recommended by NinjaTrader_Kate above, you may also want to ensure that your date range does not include "today", since it will (most likely) be an incomplete data series. If you test with a range that ends today, and then test that same range tomorrow, you may see tomorrow will (most likely) have more data. This may not be the case if the test is run after the close of the market for the instrument ... but keep it in mind, to try to be sure of data consistency.

If you are already taking this into account, bear in mind that there are inevitably some elements of random timing involved in any simulation. How much or even whether that would contribute to the significant difference you see is debatable ... but again, just keep it in mind.

And also remember that certain bar types (e.g. Renko) do not work well in the Strategy Analyzer if you are making any reference to those types of bars in your strategy.

Thanks.

[Daveman0]

Hi jeronymite thanks for your input - I'm using standard OHLC bars, but I was wondering if using Tick data as a basis would throw results off over a longer period of time? The strategy uses moving averages based on Tick bars, so I wonder if the bars are compiling the same way each time? I have no sense of how to check this, especially when dealing with millions of bars' worth of historical data - any thoughts or suggestions would be appreciated!

[NinjaTrader_Kate]

Hello Daveman0,

Thank you for your reply.

jeronymite is likely accurate here. If you ran the backtest on May 27th that included data for that day, we would expect that only data up to the time of the backtest would be available. Running the backtest on a later date again would include all data for the 27th and would show different results.

Please let us know if we may be of further assistance to you.

[Daveman0]

Hi Kate, I was running these tests after market closing hours on 5/27, I'm quite sure that is not the reason for the discrepancy - in examples posted, the interval for backtesting is 1.5 years and the difference between the two results is 10% of total trade volume and 35% net profit during this period - this was surely not caused by incomplete trading data from a single day in a 500+ day sample range.

Do you have any thoughts on whether tick bars may be compiling differently at different times using the Strategy Analyzer? That's the main thing I can make sense of at the moment, but don't know how to verify it. Thank you -

[Daveman0]

Hello Daveman0,

Thank you for your reply.

Does the discrepancy remain after clearing your cache?

To delete the cache, follow the steps below.

• Shut down NinjaTrader.
• Open the Documents > NinjaTrader 8 > db folder.
• Delete the sub-folder named 'cache'.
• Restart NinjaTrader and test.

Please let us know if we may be of further assistance.

After clearing the cache, I am getting more or less consistent results as the 2nd batch of tests (from June 1st) - so that's a good step. But I'm still bewildered as to why all my results from May 27th/28th are so completely different? Any insight would be greatly appreciated -

Is it recommended to frequently delete the cache folder to ensure consistent results? Thank you -

[NinjaTrader_Kate]

Hello Daveman0,

Thank you for your reply.

Deleting the cache would definitely be a first step if you're seeing inconsistent results. As far as results go, are you seeing consistent results for the dates in question after clearing the cache?

Thanks in advance; I look forward to assisting you further.

[Daveman0]

Hello Daveman0,

Thank you for your reply.

Deleting the cache would definitely be a first step if you're seeing inconsistent results. As far as results go, are you seeing consistent results for the dates in question after clearing the cache?

Thanks in advance; I look forward to assisting you further.

There are some minor inconsistencies, but overall it looks pretty consistent yes -

[jayds1019]

This is an older thread but I noticed a 2000 tick timeline. I'm wondering if that is too large of a timeline to have "accurate" entries and exits. Iv been testing with 1 tick, back 6 months. Now that being said I am seeing discrepancies myself. Running the same strategy in the analyzer and running the strategy on a historical chart will yield 2 different results. This is why I found this thread because I'm looking to sure up these tests.

[NinjaTrader_ChelseaB]

Hello jayds1019,

Welcome to the NinjaTrader forums!

By timelime are you meaning you have set 'Load data based on' set to 'Bars' and the 'Bars to load' is set to 2000?

Add debugging prints (or write to file) the data and condition values to see what is different.

Below is a link to a guide with an example script that has sample code.

https://support.ninjatrader.com/s/article/Developer-Guide-Comparing-Strategy-Results