Strategies give different results with .21

[lookOutBelow]

Why would I get different results when going from version .20 to .21? I've got multiple strategies, that with the same exact settings give me different results after I installed the new version. The differences are not minor. As an example. One optimization in .20 gave me $11,000 return, and .21 gives me $2,800.

Is this a known issue?


This is really frustrating if I'm going to have to go through all my strats and re-optimize them. And then probably again when a new version comes out?

[NinjaTrader_Jason]

Hello lookOutBelow,

This is not a known issue. Could it be historical data has changed once you upgraded? Perhaps a different session template is selected?

If you use the exact same settings and data, the results should be the same.

[lookOutBelow]

I'm using the same session templates and same strat parameters.

When I first installed the new version, I didn't delete the database, I just optimized the strat using the same settings. The results were very far off. I then went to delete, then repair the database.

I did notice that when deleting the database, there was still data left over. Without connecting to a provider, I was still able to get results when running my strats. As if not everything was removed. I went through "db" folders and saw lots of leftover files in the various folders. Clearing the database still left stuff lying around.

So, I deleted all of that by hand and re-ran the strats. This time, it didn't have any data to work with.

I re-connected to Zenfire and re-ran the strats. I got results this time. Not the same as .20 (and pretty far off), but I got new/different results.

[lookOutBelow]

I have an idea I'd like to test out, to see whether this is the problem. Can you please let me know if this will screw things up more??

If I go to:

My Documents\NinjaTrader 7\bin\Custom\Indicator

And copy all the .20 files from this same folder (I backed it up prior to installing the new .21 version) to the current folder above-- Will this screw anything up? Is there some special vodoo going on that won't allow this to go over smoothly?

I'm just not sure if doing this will cause more problems some where else.


Basically, I'm wondering if there is a Custom/Indicator that is throwing things off.


Thanks,
Rick

[NinjaTrader_Jason]

Hello lookOutBelow,

Please note that historical data is not saved in the database in NinjaTrader 7. It is saved in separate folders at (My) Documents-->NinjaTrader 7-->db-->day/minute/tick.

Do you still have the results available from version 7.0.0.20? If so, please compare these results to the new results and check where the discrepancy lies.

There has nothing changed how strategies are backtested between versions.

[lookOutBelow]

I didn't save the results. I took very careful notes of the settings and duplicated them in the new version. Here is an example of the differences between using the same settings, time periods, sessions, etc:

version .20
========
$11,560 Total Net Prof
2.75 profit factor
102 trades
65.69% percent profitable

Version .21
========
$1,860 Net
1.24 PF
83 trades
51.81%


As you can see, it is quite different. I did a Diff compare between the indicators in my backup folder and the current Custom/Indicators folder and they are exactly the same. So, that isn't the issue.

[NinjaTrader_Jason]

Unfortunately I am not sure why there is a difference and do not see a way to determine what caused it.

You can save backtest results in the Strategy Analyzer, see the link below.

http://www.ninjatrader.com/support/helpGuides/nt7/saving_performance_results.htm

I suggest to save your results this way. If you come across a discrepancy, please compare the results to the prior results you saved. This will allow you to check where the discrepancy lies.

[lookOutBelow]

I tried saving backtest results once. But it took SOOOO long to do it, I decided not to do it again. However, if I knew this would happen, I would of done it before upgrading.

BTW, I went into the Historical Manager and deleted all the historical data for FESX (the instrument I'm running the strat on). After re-running with the same settings I've been using all along, I got yet another set of results. Completely different from the other two results.

[NinjaTrader_Jason]

That is strange - you make use of only one data feed in NinjaTrader, correct? Please tell me which one.

Could you please send me the strategy you used, so I can test it on my end. You can send it to support[at]ninjatrader[dot]com. Put 'Attn Jason' in the subject header and reference this forum thread.

In the mean time, can you please perform the same for the SampleMACrossover strategy. Run a backtest. Subsequently delete the historical data and perform a second identical backtest on new historical data from the same data feed (using the same session template).

[lookOutBelow]

This is only getting stranger.


I'm doing a Walk Forward. My strategy parameters are 6 different numbers. As I go through each one, I widen the range and find a more optimal setting for each value.

I do this for a while, coming up with better returns.

I go back and return all my numbers to the original default position and the returns are different from where they were when I started. Using the same exact settings and parameters!

Then, I go and clear out my historical data, and I get another completely different set of returns.

I don't know what is going on, but something is seriously screwy if the same exact parameters are giving different values when I switch parameters around and then go back to where I originally started. And then, the results are off again after I clear the historical data.

None of this makes sense at all. I'm using the same historical provider, the same input parameters, sessions, times, etc, etc. But get different results.



I'll have to try this against a default strategy to see if I get the same problem. I'm not comfortable sending my strategies.

[lookOutBelow]

To add even more craziness to this whole mess.

After I've cleared the historical data, and re-run the original parameters (I get that third set of values after the historical data is reloaded from Zenfire). If I immediately re-run the Walk-Forward, it returns to the original set of returns!!!

I can't seem to get any consistency..


Here is how I'm working...

1) I take my strategy and optimize it over a two month period
2) I write down the best parameter values
3) I switch over to Walk-Forward and use those same exact parameter values as my starting point.
4) I tweak each parameter to see if I can squeeze better returns out of it...

If a parameter value is "32". I run another Walk forward with "32-33" then, "32-34", then "31-32", etc, etc. Until I get the best results. So there is a lot of going back and forth, re-running the walkforward over and over again.

After doing this for the 6 parameters, I went back and reset them to their initial "optimized" default values and that is when I get different results than I did to begin with. And then tried resetting historical data, etc, etc...

[NinjaTrader_JoshP]

lookOutBelow,

Just jumping in here, does the particular strategy you are using happen to be a multi-time frame or multi-instrument strategy?

[lookOutBelow]

Hi Josh.

No, it is working off of FESX only. And only on 1-minute timeframe.



I tried to reproduce it with the Macd crossover example. But with limited time playing around with it, I couldn't get the same problem. I guess it has something to do with my strategy. But it seems very odd that the same settings can give different results. Even if I'm playing around with parameters in between those same-setting-runs.


I'm pushing through it and paying more attention to when it is happening to see if I can come up with a more specific pattern.

[NinjaTrader_Jason]

I do not experience the issue so far when testing the SampleMACrossOver strategy on my end - it seems to be related to your strategy as you mention.

Please let me know when you find a pattern as per the issue you experience.

[dave1992]

I see wildly different results too. The two attached charts show a multi time frame strategy with entry on the tick time frame using market price entry. In Beta 20, the entry is as expected (when passing the yellow dot) In Beta 21, ALL entries appear to be at the start of the 5 min bar, so giving you a massive gain in profits. In fact I thought I'd found the holy grail looking at the results initially.

I can guarantee the strategy and price data was the same.

I'm going to put some prints in to see what's going on. But in the meantime, it's nothing to do with this is it? (I realise the OP was not multi series)

Fixed
4044
NinjaScript strategies
Entry signal handling could get out of sync on multi-series strategies with more than one series on the same instrument.