Optimizer simply needs more work

[cgeorgan]

Guys-

I have to admit, after several days with this product I can only conclude that the optimizer needs more work.

Using identical settings, I can often come up with totally different performance (note, I'm not talking about the results of the optimizer - I'm referring to the optimizer itself).

Sometimes, I can plow through 10-20 complex optimizations quickly, with times as low as 5 minutes per run.

Other times, *using the same settings and same instruments*, I can get results that vary as bad as 2 hours, to never getting done at all (that is, the dreaded "saving" lock followed by the equally dreaded "aborting" lock).

There is nothing in the log files or trace files that would give me an indication as to what I'm doing wrong.

It's not a memory issue, nor is it a performance issue - I monitor all these things in process via the Task Manager.

Guys, this thing needs more work...I definitely wouldn't move it to production from beta in its current form.

PS - as a note, here's what I'm trying to optimize:

• 2 parameters which vary from 10 to 50 in 5 step increments
• 3 parameters which each vary from 0 to 50 in 5 step increments
• Only one time series, from 6/1 to today.
• Using the genetic algorithm, I do 10 generations and 50 samples/generation.

Like I said - using these settings I can sometimes get a lot of work done. At other times, I get nothing done.

It's not limited to other background processes either. I ran probably 20 optimization runs this morning on an NT process that had been running 15 trading strategies since the prior night without a restart. Right now, I'm not running *anything* and I can't get a single optimization to run, many restarts later.

[NinjaTrader_JoshP]

cgeorgan,

Why would you expect the results to be the same? You are using Genetic Algorithm. Genetic Algorithm is not an exhaustive optimization algorithm, meaning, it does not test every combination possible and will definitely yield different results through different runs if you are not running it with enough permutations. The Genetic Algorithm uses random factors to speed things up. Even with a lot of permutations in the testing, results can be different depending on the results, data, and the random factor I mentioned earlier.

The Genetic Algorithm's benefit is to get you within the ballpark of the results. It is never guaranteed to give you absolute maximums. It can indeed find maximums if given enough generations to run and enough samples per generation, but that defeats the purpose of a genetic algorithm.

If you want absolute results you should be using an exhaustive optimizer like the default optimizer. The genetic is used to find acceptable results in a much quicker amount of time without testing every combination possible. This is the reality of computer optimization that everyone deals with. In a perfect world we would all be able to run exhaustive tests and get perfect maximum results every time, but with the complexity of things people want to test computers just aren't fast enough to be able to test billions of combinations in a reasonable amount of time. Hence the creation of alternative algorithms that can do decently well in a much more reasonable time frame.

There is simply a trade off between accuracy and speed of optimization. If we had infinite time to let our computers run an exhaustive test we would definitely do that, but since most people do not have infinite time, trading marginal accuracy is a trade many are willing to take for the increased speed of optimization.

[cgeorgan]

Josh -

I found it important to note early in my post that it was not the *results* of the optimizer that are bothering me. It's the performance of the optimizer itself.

[NinjaTrader_JoshP]

As such, the random factors of the Genetic Algorithm do not guarantee you every generation between different runs are identical. Thus multiple runs can result in different behavior. You may very well simply end up using combinations that are problematic based on your code while other times these problem combinations are never hit.

I suggest you add Print()s in your strategy and monitor the Output Window for whether or not the optimizer is actually stuck and where in the strategy code it gets stuck at.

[cgeorgan]

As such, the random factors of the Genetic Algorithm do not guarantee you every generation between different runs are identical. Thus multiple runs can result in different behavior. You may very well simply end up using combinations that are problematic based on your code while other times these problem combinations are never hit.

I suggest you add Print()s in your strategy and monitor the Output Window for whether or not the optimizer is actually stuck and where in the strategy code it gets stuck at.

I can see where I'm getting with this.

Again, I'm talking about the optimizer's performance during different sessions, using identical parameters.

I'm not referring to the output, but to the absolute performance.

Randomness would explain 10 or 20 minute differences. It would not explain 5 minute vs. Did-Not-Finish runs.

This morning, I had extremely consistent performance optimizing a single strategy over 8 separate instruments. I ran the optimization for each instrument using at least 4 different timeframes. Each time I ran the optimizer, it was finished in a reasonably predictable time. Lots of productive work was accomplished (all of it while 18 live strategies were running in the background).

This afternoon, after restarting the system, I attempted to do *the same thing*. Same instruments, same settings on the optimizer. This time, My optimizer indicates it could take over 2 hours - for *any* instrument. I tried shutting down all the strategies I was running but this didn't really help. A couple of the times, the optimizer just refused to cooperate and hung on "saving". Then it hung on "aborting".

I'm telling you, I understand what you're talking about with the results of the optimizer. What I'm also telling you is that the performance of the product (not the output of the product) is clearly variable even when using identical starting parameters. When I say variable, I do not mean normally distributed variance as you've suggested. I mean discretely different performance that cannot be explained by normal statistical description.

[cgeorgan]

Here's what my code keeps trying to repeat...this is all I can get, and it's out of the trace file:

2010-07-16 15:29:35:171 (DTN-IQ) Data.Bars.GetBarsNow: instrument='6A 09-10' from='2010-05-31' to='2010-07-16 23:59:59' period=1 Tick splitAdjusted=False dividendAdjusted=False bars=0 session='CME F/X Globex'
2010-07-16 15:29:40:546 (DTN-IQ) Data.Bars.GetBarsNow: instrument='6A 09-10' from='2010-05-31' to='2010-07-16 23:59:59' period=1 Tick splitAdjusted=False dividendAdjusted=False bars=0 session='CME F/X Globex'
2010-07-16 15:29:47:609 (DTN-IQ) Data.Bars.GetBarsNow: instrument='6A 09-10' from='2010-05-31' to='2010-07-16 23:59:59' period=1 Tick splitAdjusted=False dividendAdjusted=False bars=0 session='CME F/X Globex'
2010-07-16 15:29:54:859 (DTN-IQ) Data.Bars.GetBarsNow: instrument='6A 09-10' from='2010-05-31' to='2010-07-16 23:59:59' period=1 Tick splitAdjusted=False dividendAdjusted=False bars=0 session='CME F/X Globex'
2010-07-16 15:30:02:140 (DTN-IQ) Data.Bars.GetBarsNow: instrument='6A 09-10' from='2010-05-31' to='2010-07-16 23:59:59' period=1 Tick splitAdjusted=False dividendAdjusted=False bars=0 session='CME F/X Globex'
2010-07-16 15:30:09:390 (DTN-IQ) Data.Bars.GetBarsNow: instrument='6A 09-10' from='2010-05-31' to='2010-07-16 23:59:59' period=1 Tick splitAdjusted=False dividendAdjusted=False bars=0 session='CME F/X Globex'
2010-07-16 15:30:14:546 (DTN-IQ) Data.Bars.GetBarsNow: instrument='6A 09-10' from='2010-05-31' to='2010-07-16 23:59:59' period=1 Tick splitAdjusted=False dividendAdjusted=False bars=0 session='CME F/X Globex'
2010-07-16 15:30:19:578 (DTN-IQ) Data.Bars.GetBarsNow: instrument='6A 09-10' from='2010-05-31' to='2010-07-16 23:59:59' period=1 Tick splitAdjusted=False dividendAdjusted=False bars=0 session='CME F/X Globex'
2010-07-16 15:30:26:531 (DTN-IQ) Data.Bars.GetBarsNow: instrument='6A 09-10' from='2010-05-31' to='2010-07-16 23:59:59' period=1 Tick splitAdjusted=False dividendAdjusted=False bars=0 session='CME F/X Globex'
2010-07-16 15:30:31:578 (DTN-IQ) Data.Bars.GetBarsNow: instrument='6A 09-10' from='2010-05-31' to='2010-07-16 23:59:59' period=1 Tick splitAdjusted=False dividendAdjusted=False bars=0 session='CME F/X Globex'
2010-07-16 15:30:36:734 (DTN-IQ) Data.Bars.GetBarsNow: instrument='6A 09-10' from='2010-05-31' to='2010-07-16 23:59:59' period=1 Tick splitAdjusted=False dividendAdjusted=False bars=0 session='CME F/X Globex'
2010-07-16 15:30:42:375 (DTN-IQ) Data.Bars.GetBarsNow: instrument='6A 09-10' from='2010-05-31' to='2010-07-16 23:59:59' period=1 Tick splitAdjusted=False dividendAdjusted=False bars=0 session='CME F/X Globex'
2010-07-16 15:30:50:125 (DTN-IQ) Data.Bars.GetBarsNow: instrument='6A 09-10' from='2010-05-31' to='2010-07-16 23:59:59' period=1 Tick splitAdjusted=False dividendAdjusted=False bars=0 session='CME F/X Globex'
2010-07-16 15:30:55:468 (DTN-IQ) Data.Bars.GetBarsNow: instrument='6A 09-10' from='2010-05-31' to='2010-07-16 23:59:59' period=1 Tick splitAdjusted=False dividendAdjusted=False bars=0 session='CME F/X Globex'
2010-07-16 15:31:00:921 (DTN-IQ) Data.Bars.GetBarsNow: instrument='6A 09-10' from='2010-05-31' to='2010-07-16 23:59:59' period=1 Tick splitAdjusted=False dividendAdjusted=False bars=0 session='CME F/X Globex'
2010-07-16 15:31:30:140 (DTN-IQ) Data.Bars.GetBarsNow: instrument='6A 09-10' from='2010-05-31' to='2010-07-16 23:59:59' period=1 Tick splitAdjusted=False dividendAdjusted=False bars=0 session='CME F/X Globex'
2010-07-16 15:31:49:562 (DTN-IQ) Data.Bars.GetBarsNow: instrument='6A 09-10' from='2010-05-31' to='2010-07-16 23:59:59' period=1 Tick splitAdjusted=False dividendAdjusted=False bars=0 session='CME F/X Globex'
2010-07-16 15:32:14:359 (DTN-IQ) Data.Bars.GetBarsNow: instrument='6A 09-10' from='2010-05-31' to='2010-07-16 23:59:59' period=1 Tick splitAdjusted=False dividendAdjusted=False bars=0 session='CME F/X Globex'
2010-07-16 15:32:32:953 (DTN-IQ) Data.Bars.GetBarsNow: instrument='6A 09-10' from='2010-05-31' to='2010-07-16 23:59:59' period=1 Tick splitAdjusted=False dividendAdjusted=False bars=0 session='CME F/X Globex'
2010-07-16 15:32:53:875 (DTN-IQ) Data.Bars.GetBarsNow: instrument='6A 09-10' from='2010-05-31' to='2010-07-16 23:59:59' period=1 Tick splitAdjusted=False dividendAdjusted=False bars=0 session='CME F/X Globex'
2010-07-16 15:33:22:390 (DTN-IQ) Data.Bars.GetBarsNow: instrument='6A 09-10' from='2010-05-31' to='2010-07-16 23:59:59' period=1 Tick splitAdjusted=False dividendAdjusted=False bars=0 session='CME F/X Globex'
2010-07-16 15:33:45:953 (DTN-IQ) Data.Bars.GetBarsNow: instrument='6A 09-10' from='2010-05-31' to='2010-07-16 23:59:59' period=1 Tick splitAdjusted=False dividendAdjusted=False bars=0 session='CME F/X Globex'
2010-07-16 15:34:11:125 (DTN-IQ) Data.Bars.GetBarsNow: instrument='6A 09-10' from='2010-05-31' to='2010-07-16 23:59:59' period=1 Tick splitAdjusted=False dividendAdjusted=False bars=0 session='CME F/X Globex'
2010-07-16 15:34:32:390 (DTN-IQ) Data.Bars.GetBarsNow: instrument='6A 09-10' from='2010-05-31' to='2010-07-16 23:59:59' period=1 Tick splitAdjusted=False dividendAdjusted=False bars=0 session='CME F/X Globex'
2010-07-16 15:34:55:703 (DTN-IQ) Data.Bars.GetBarsNow: instrument='6A 09-10' from='2010-05-31' to='2010-07-16 23:59:59' period=1 Tick splitAdjusted=False dividendAdjusted=False bars=0 session='CME F/X Globex'
2010-07-16 15:35:15:453 (DTN-IQ) Data.Bars.GetBarsNow: instrument='6A 09-10' from='2010-05-31' to='2010-07-16 23:59:59' period=1 Tick splitAdjusted=False dividendAdjusted=False bars=0 session='CME F/X Globex'
2010-07-16 15:35:37:406 (DTN-IQ) Data.Bars.GetBarsNow: instrument='6A 09-10' from='2010-05-31' to='2010-07-16 23:59:59' period=1 Tick splitAdjusted=False dividendAdjusted=False bars=0 session='CME F/X Globex'
2010-07-16 15:36:00:843 (DTN-IQ) Data.Bars.GetBarsNow: instrument='6A 09-10' from='2010-05-31' to='2010-07-16 23:59:59' period=1 Tick splitAdjusted=False dividendAdjusted=False bars=0 session='CME F/X Globex'
2010-07-16 15:36:28:765 (DTN-IQ) Data.Bars.GetBarsNow: instrument='6A 09-10' from='2010-05-31' to='2010-07-16 23:59:59' period=1 Tick splitAdjusted=False dividendAdjusted=False bars=0 session='CME F/X Globex'
2010-07-16 15:36:51:750 (DTN-IQ) Data.Bars.GetBarsNow: instrument='6A 09-10' from='2010-05-31' to='2010-07-16 23:59:59' period=1 Tick splitAdjusted=False dividendAdjusted=False bars=0 session='CME F/X Globex'
2010-07-16 15:37:12:796 (DTN-IQ) Data.Bars.GetBarsNow: instrument='6A 09-10' from='2010-05-31' to='2010-07-16 23:59:59' period=1 Tick splitAdjusted=False dividendAdjusted=False bars=0 session='CME F/X Globex'
2010-07-16 15:37:36:046 (DTN-IQ) Data.Bars.GetBarsNow: instrument='6A 09-10' from='2010-05-31' to='2010-07-16 23:59:59' period=1 Tick splitAdjusted=False dividendAdjusted=False bars=0 session='CME F/X Globex'
2010-07-16 15:38:01:234 (DTN-IQ) Data.Bars.GetBarsNow: instrument='6A 09-10' from='2010-05-31' to='2010-07-16 23:59:59' period=1 Tick splitAdjusted=False dividendAdjusted=False bars=0 session='CME F/X Globex'
2010-07-16 15:38:20:484 (DTN-IQ) Data.Bars.GetBarsNow: instrument='6A 09-10' from='2010-05-31' to='2010-07-16 23:59:59' period=1 Tick splitAdjusted=False dividendAdjusted=False bars=0 session='CME F/X Globex'
2010-07-16 15:38:37:093 (DTN-IQ) Data.Bars.GetBarsNow: instrument='6A 09-10' from='2010-05-31' to='2010-07-16 23:59:59' period=1 Tick splitAdjusted=False dividendAdjusted=False bars=0 session='CME F/X Globex'
2010-07-16 15:38:54:234 (DTN-IQ) Data.Bars.GetBarsNow: instrument='6A 09-10' from='2010-05-31' to='2010-07-16 23:59:59' period=1 Tick splitAdjusted=False dividendAdjusted=False bars=0 session='CME F/X Globex'
2010-07-16 15:39:13:250 (DTN-IQ) Data.Bars.GetBarsNow: instrument='6A 09-10' from='2010-05-31' to='2010-07-16 23:59:59' period=1 Tick splitAdjusted=False dividendAdjusted=False bars=0 session='CME F/X Globex'
2010-07-16 15:39:36:171 (DTN-IQ) Data.Bars.GetBarsNow: instrument='6A 09-10' from='2010-05-31' to='2010-07-16 23:59:59' period=1 Tick splitAdjusted=False dividendAdjusted=False bars=0 session='CME F/X Globex'
2010-07-16 15:39:55:562 (DTN-IQ) Data.Bars.GetBarsNow: instrument='6A 09-10' from='2010-05-31' to='2010-07-16 23:59:59' period=1 Tick splitAdjusted=False dividendAdjusted=False bars=0 session='CME F/X Globex'
2010-07-16 15:40:14:515 (DTN-IQ) Data.Bars.GetBarsNow: instrument='6A 09-10' from='2010-05-31' to='2010-07-16 23:59:59' period=1 Tick splitAdjusted=False dividendAdjusted=False bars=0 session='CME F/X Globex'
2010-07-16 15:40:34:250 (DTN-IQ) Data.Bars.GetBarsNow: instrument='6A 09-10' from='2010-05-31' to='2010-07-16 23:59:59' period=1 Tick splitAdjusted=False dividendAdjusted=False bars=0 session='CME F/X Globex'
2010-07-16 15:40:53:953 (DTN-IQ) Data.Bars.GetBarsNow: instrument='6A 09-10' from='2010-05-31' to='2010-07-16 23:59:59' period=1 Tick splitAdjusted=False dividendAdjusted=False bars=0 session='CME F/X Globex'

Just keeps trying to get this info.

Ps, I'm not actually trying to get 1 tick bars...

[NinjaTrader_JoshP]

cgeorgan,

I am not talking about the results. The purpose of using Print()s in the strategy is to determine truly what is happening under the hood. Only through having those Print() statements would it become evident if it was truly a dead lock in your app somewhere or if the optimizer is still processing through the strategy or not.

A dead lock would be seen when nothing gets printed to the Output Window anymore. A non deadlock scenario would be one that continues to receive printouts as it processes different parts of the code. As the code reaches different parts different Print() lines will trigger. If no lines are being triggered all of a sudden then that would signify an issue and that would definitely be something to look into.

As such I have just ran the GA on SampleMACrossOver over 15 times with no problematic behavior. Please confirm such is the case on your end.

[cgeorgan]

The optimizer doesn't get into a dead lock (often). What it does instead is that it will take over 2 hours for one run, vs. 5 minutes for another, using identical inputs. That performance is not random; when the optimizer is working it works consistently well. When it's not working, it is consistently bad. If it were random, I'd expect good runs interspersed with bad runs.

I mean, look - I'm no idiot and I'll take your advice and run a thorough debug through the code (note: I run this one strategy 24 hours over 18 instruments and I don't have problems that cause a lock up).

However, I get the feeling that the debug results will show that it's not the indicator that's the problem. I get the distinct feelign that this has more to do with my data, or some sort of database corruption, than anything else -

[NinjaTrader_JoshP]

I suggest sending us your data and database as you achieve the optimizing in such a state without changing anything.

When you reach such a state, please send us the chart data, database, strategy in use, name of instrument being used, all optimization settings being used. That way we will be able to try it on our end with your exact setup.

[NinjaTrader_JoshP]

Development is interested in the time zone your PC is in and the session template you are using. It looks to be a custom one so we would want to know how this session template was defined. Thanks.

Also, for the repetitive bars request, can you confirm that this indeed occurs every time your optimizer "hangs"?

[cgeorgan]

Development is interested in the time zone your PC is in and the session template you are using. It looks to be a custom one so we would want to know how this session template was defined. Thanks.

Also, for the repetitive bars request, can you confirm that this indeed occurs every time your optimizer "hangs"?

Sorry, just saw this.

Alrighty, I'm using GMT -6 on my development box, which corresponds to Chicago's Central Time.

The session I'm using (I've labeled it "CME F/X Globex") is also on the same time zone, and is essentially:
Sunday 5PM - Monday 4PM
Monday 5PM - Tuesday 4PM
..
..
Thursday 5PM - Friday 4PM.

I'll look into your request re: Repetitive bar requests. Offhand I would say yes, but I'd like to confirm it to make sure.

CG

[NinjaTrader_JoshP]

Okay. Unfortunately we were unable to reproducible on our end yet with those settings after multiple attempts. My earlier suggestion about sending us data+database, etc. would still be relevant going forward. Thanks.

[cgeorgan]

Okay. Unfortunately we were unable to reproducible on our end yet with those settings after multiple attempts. My earlier suggestion about sending us data+database, etc. would still be relevant going forward. Thanks.

Josh -

I'm running tests all day (need to have full days of strategy testing). In the meantime I'm working on test code I can send you guys to try and reproduce.

In terms of helping you guys, one thing I do in my code is create multiple time frames (more than 2, sometimes 4). Part of the optimization deals with parameters on these timeframes -

[cgeorgan]

Development is interested in the time zone your PC is in and the session template you are using. It looks to be a custom one so we would want to know how this session template was defined. Thanks.

Also, for the repetitive bars request, can you confirm that this indeed occurs every time your optimizer "hangs"?

Josh -

How many GetBars requests would you consider "normal"? I was trying to do optimization runs on the FDAX today, and I was consistently getting many lines of GetBarsNow requests for the exact same time periods, over the course of several minutes.

[NinjaTrader_JoshP]

cgeorgan,

Every optimization has several iterations within that single optimization run. Each optimization run will pull data if the load data rules are hit. http://www.ninjatrader-support.com/H...HistoricalData

So if it takes a few minutes for one iteration to complete then you would see loading data again on the next iteration.

It seems to me you are testing with an end date of today?