Optimizer and threads

[NinjaTrader_Bertrand]

Thanks for reporting back, looking forward to your files then.

[JustCallMeDan]

Files just sent with a clean NinjaTrader setup. Here are some screenshots showing the core loads during optimisation - note the graph in the top right showing core usage history; there's a marked difference between default and genetic optimisations.

[NinjaTrader_Bertrand]

Thanks for the screenshots and I got your files and will look into it.

[Trader.Jon]

Dan,

Where did you get that graph in your screenshot?

TY

[JustCallMeDan]

The screenshots show a core usage graph from the sidebar gadget All CPU Meter v 3.3 available from http://www.addgadget.com/ or through the Windows gadgets online at http://gallery.live.com/

I believe it works on Vista and Windows 7.

If you want temperatures showing you need to get CoreTemp as well (http://www.alcpu.com/CoreTemp/) and set it to load at Windows startup. I now notice they also have a competing gadget on the Add Ons page!

For XP (no gadgets) I had to use CoreTemp with the Add On: Core Temp Grapher Beta

[JustCallMeDan]

For public posterity, I've also recreated this issue on my Intel Core 2 Duo laptop and submitted relevant files to NinjaTrader support, along with a simple strategy based on SampleMACrossover which additionally performs instance counting in the log. On both my systems the number of strategy instances never climbs above one for genetic optimisations, but there are always as many instances as there are cores for exhaustive optimisations.

Fingers crossed Bertrand et al can get to the bottom of it!


P.S. The strategy file is attached in case anyone else wishes to reproduce these results on a different multi-core system. The more examples the better, I suspect.

[NinjaTrader_Bertrand]

Thanks for posting - I will need to clarify with development if that's a valid way to prove / disprove core usage, I'll keep you in the loop on our findings.

Thanks for the support,

[Trader.Jon]

Works for me!

Dan;

I started up the sample and all 4 cores on my antique 9250 are @ 50%+ using PHGenetic NT7R1 Win7Pro 64bit, and then the same with the supplied Genetic, and the Default.. With so little code to optimize its really not needing any GO, but the selection was available .

I was concerned that it wouldnt use all the cores when I compared the declarations in the strategy versus mine:
#region Using declarations
using System;
using System.IO;
using System.Collections;
using System.ComponentModel;
using System.Diagnostics;
using System.Drawing;
using System.Drawing.Drawing2D;
using System.Xml.Serialization;
using NinjaTrader.Cbi;
using NinjaTrader.Data;
using NinjaTrader.Indicator;
using NinjaTrader.Gui.Chart;
using NinjaTrader.Strategy;
using System.Net.Mail;
using System.Security.Principal;
using System.Windows.Forms;
using System.Threading;

as all mine all have the 'Threading'. I vaguely remember someone else posting that there was an issue with NT7 R2 having something not right with the genetic optimizations, but I cant put my eyeballs on that posting: I do remember enough that I thought I would wait before upgrading.

If I can remember to do so I will check on my dual processor opteron and try out your sample there, but it is NT7 R1 also, and has consistently been using 8 cores during PHGenetic runs.

Jon

[eleven]

I must weigh in as well. I have an Intel i7-2600k second generation "sandy bridge" with 8 GB of ram. It has 4 physical cores and 4 HT cores. When running GO it will utilize the 4 physical cores but they will never add up beyond 12-13%. (which is 1/8=12.5%) I am not using any bools or enums as well.

Therefore, GO uses multiple cores but it does not utilize the full potential of using 4 cores. Basically, the 4 physical cores are no faster than 1 physical core.

I upgraded my machine from a dual core Intel E6600 2.4 GHz to hopefully get a boost in optimization times. So far I don't think I have realized that benefit from cores, only CPU architecture and increased clock.

[neb1998]

I have been working on this all evening. Reinstalling all .net software, modifying the genetic optmizer code to remove the possibility of turning off multithreading even if bool exist. Checking configuration files, i do not see the problem.

[eleven]

Neb,

What processor specs are you running? It makes a difference.

My laptop uses an athlon2 2.0 GHz - both cores pegged 100%
My old desktop (Intel dual core E6600) - both cores near 100%
My wifes desktop (AMD Phenon2 3.0 GHz) - all 3 cores <100% (70-80% if I recall, but didn't think much of it at the time)
My brand new desktop (Intel i7 2nd gen 4.2 GHz) - all 4 physical cores add up to 12.5% (1/8 = 12.5%, 4 physical and 4 HT)

I don't get it either. It seems the faster the proc it hits a limit on what NT will utilize. This is all tested on GO. I haven't analyzed the exhaustive (default) optimization yet.

[neb1998]

i have 2 i7920's both OC'ed to 4ghz, and they only use 25%, but i have hyperthreading turned off, when it on it will use about 12.5% or so.

I havent tried on my laptop but its a dualcore e series intel, i could try it. Maybe the processor and dot net are not cooperating - although i saw someone with a 1055t having the same problem.

[neb1998]

I forgot, i have an old E8600 also oc'ed to 4ghz, maybe i will try it and see if it works. Dualcore at 4ghz is a hell of a lot faster than single at 4.

The E series cpu's mhz per mhz are not much slower (1-5%) or so per core than the i7's.

meaning 2 cores on an i7 versus 2 cores on an e8600 there is rarely much of a difference using MC or Ninja that i have seen.

[NinjaTrader_Bertrand]

Guys, thanks for the efforts, patience and research put into this topic - please hold off further testing for now, we were able to reproduce an inconsistency and are looking into matters.

Thanks for the support,

[NinjaTrader_Bertrand]

Guys, here's an update : thanks to your patient support we were able to reproduce the issue and fixed it for the next NT7 update. Please restest then and report back your findings in this thread.

Thanks,