Has anyone run some backtests/optimizations in the Strategy Analyzer

[NinjaTrader_Ray]

Would love to hear some feedback on anyone that has been testing the performance/memory of backtesting/optimization of strategies.

[PrTester]

Would love to hear some feedback on anyone that has been testing the performance/memory of backtesting/optimization of strategies.

I try, if i use the default strategies, great, if i use one of mines lots of chums, and lot of message, need to strip them down and start adding code to see if they work, but never end an optimization :-(

[daqu40]

First test aren't working

My first tests aren't working. It seems the OnBarUpdate is not doing anything.

I tried the SampleMACrossOver and nothing.

I'm digging through the forum to see if there are any configuration issues I can look at to fix this.

Well 10 minutes later I figured it out. I didn't have all the data loaded for my strategy. Seems to be basically working.

Execution speed is initially slower. However my strategy is not producing the same results either so I'll need to get the same results before offering a fair estimate.

I am very happy to see that all 8 CPUs (Quad Core with HyperThreading) are about equally active during the backtest. This is a very welcome change and I expect I'll get much faster tests once I get the code changes ironed out.

Memory impact seems to be fairly minimal. I run test on ES futures going back to 09-07; 9 full sets of data on 1, 30, 60, 1440, and daily SP500 data series. I'm just using Windows Task Manager, but I only got a bout 6% increase in physical memory usage when running these tests.

I'll post new entry when I get this working properly.

[laparker]

A rough test optimizing on 2 params gave me a speed increase by a factor of 2.7 using default optimizer

[laparker]

some issues

These are some of the issues I have come across:

On back test tabbed grid, after running test and selecting a tab other than summary, on returning to summary tab, Long Trade and Short Trade detail is blank

Genetic optimizer (GA) crashes with bool params

GA doesn't find anywhere near optimum params. I'm using 20 generations, population of 256. On 6.5 am using Swig's MOGO optimiser and piersh's. Both say that they have started with "Perter's" work and taken it further. Both seem to converge on decent params using 20 gens and 256 pop. Perhaps you could look at their code and see how it differs from yours.

On Trades tab, the Trade# is all 1. The numeric Bars column does an alpha sort

[oclop]

Hello,

I have also done some tests with NT7 Beta2. The issue I have found is that the genetic optimiser only performs the first generation and it stops after it. This may explain why the optimiser fails to find close to optimum results.

I was also glad to see that both CPUs were used when I tried the optimiser. No problems with the memory for the time being. I have not quantified yet the gains in execution speed.

I would also like to make some suggestions for improvement:

• It would be nice if when selecting one optimisation iteration to have the option to reuse those parameters directly on the same underlying for a backtest with a range of dates defined by the user. This would reduce significantly the time needed to check the results of the parameters proposed by the optimiser on a data sample that has not been used during the optimisation.
• For the genetic optimiser it would be good to have a column with the generation number and the iteration for each optimisation. This would help to identify clearly and keep track record of the optimisations that work.
• It would also be good to find a way to better present the parameters that were used for each iteration, especially the ones that have been optimised. I have seen no improvement from NT6.5. The way it is presented now with all parameters in alphabetical order separated by comas is not user friendly at all.

Regards, Oscar

[NinjaTrader_Dierk]

...
On back test tabbed grid, after running test and selecting a tab other than summary, on returning to summary tab, Long Trade and Short Trade detail is blank
...
On Trades tab, the Trade# is all 1
...
The numeric Bars column does an alpha sort

Just reproduced these ones. Will look into. Thanks

[NinjaTrader_Ray]

Thanks for all the feedback, suggestions and issue reports.

[NinjaTrader_Dierk]

Guys, please hold off trying the GeneticOptimizer. There are a couple of issues which I just addressed. Thanks for reporting you findings.

[paschall]

backtest is invaluable

I use the Backtest feature each programming session. It is invaluable to my work. Although, right now, I have to optimize parameters manually, it does give me a very nice objective performance evaluation for my strategies. One of the reasons I chose NT for development work.

cheers,
D

[zweistein]

I did try some backtesting, but usually I do some automation because I feed the backtest results into a Neural Net.


So my code is something like:

ninjaserver.Instance.BackTestDialog("SampleMACrossOver");
ninjaserver.Instance.SetBackTestParameter("Instrument","FDAX 12-09");
string dt = DateTime.Now.AddDays(-30).Date.ToString();
ninjaserver.Instance.SetBackTestParameter("BackTestFrom", dt);
ninjaserver.Instance.SetBackTestParameter("BackTestTo", dt);
ninjaserver.Instance.RunBackTest(60); // 60 s timeout
object results = ninjaserver.Instance.BackTestResult("Performance.AllTrades.TradesPerformance.Currency. CumProfit");


and due to automation I one cycle is at least one second.

Therefore I cannot talk about speed...

Best regards
Andreas

[dnoff]

I have tried running the optimizer several times tonight and it constantly fails to run beyond the a few seconds and throws the following errors:

11/1/2009 8:13:48 PM Default Error on running optimizer: Value cannot be null.
Parameter name: value
11/1/2009 8:14:00 PM Strategy Error in setting strategy parameters: Input string was not in a correct format.

The settings I am trying to optimize can be run successfully in 6.5 (just takes a while), and I enter the values (checked several times) by expanding the parameters and typing the values into the boxes.

Thoughts

[NinjaTrader_Dierk]

Please make sure you are on latest NT7B3. If issue persists, then please post your strategy and any info required to reproduce the problem. Thanks

[paschall]

back test results not accurate

I just hand-checked some backtest results against a strategy. I only ran 2 days worth so I could scroll back and hand-check things. Over the past 2 days, my strategy generated 3 long trades. Backtest results? It says there was only 1 long trade (albeit a successful one). Same erroneous result with the short trade data. Hand-checking shows 8 short trades. Back-test output Summary window indicates 5 short trades. I double-checked the date range, time frame of the backtest compared to the chart window, etc. Used default values in both instances.

Any ideas why the discrepency?

cheers,
D

[NinjaTrader_JoshP]

Please provide the strategy and relevant backtest settings used including instrument, dates, time frame, etc. Thank you.