Strategy Analyzer - Optimization - Unexecpted multi-thread activity

[rgoudie]

Greetings.

(My questions are towards the end of this posting.)

I have been doing tests on the Strategy Analyzer's optimization feature. I notice that when it runs, it effectively launches multiple threads that each process a combination of properties.

Example: I created a basic strategy that does nothing but output its current configuration in a separate text file based on the thread's managed identifier. Here is the important portion of the settings:




I am simply going through the cross-product of the values 1 and 2 for both test properties A and B.

After running the optimization, I notice that I have 8 files, which implies that 8 threads were launched during the optimization process. (The filenames start with the letter T, followed by the thread identifier.)




The text files contain a timestamp followed by the current state each time that OnStateChange() is called, followed by the current values of A and B.
I also log each call to OnBarUpdate() to indicate when it is called during the chain of events.

I do see files for each pairing of values.

---

File: T0075_DEBUG.txt
Pair:(1, 1)

18:19:00.7966 | State: SetDefaults | A: 0 | B: 0
18:19:00.7986 | State: Configure | A: 1 | B: 1
18:19:00.8006 | State: DataLoaded | A: 1 | B: 1
18:19:00.8216 | State: Historical | A: 1 | B: 1
18:19:00.8226 | 2023-03-30 12:00
18:19:00.8226 | 2023-03-30 12:15
18:19:00.8236 | 2023-03-30 12:30
18:19:00.8236 | 2023-03-30 12:45
18:19:00.8246 | 2023-03-30 13:00
18:19:00.8246 | 2023-03-30 13:15
18:19:00.8256 | 2023-03-30 13:30
18:19:00.8256 | 2023-03-30 13:45
18:19:00.8266 | 2023-03-30 14:00
18:19:00.8266 | 2023-03-30 14:15
18:19:00.8276 | 2023-03-30 14:30
18:19:00.8276 | 2023-03-30 14:45
18:19:00.8286 | 2023-03-30 15:00
18:19:00.8365 | State: Terminated | A: 1 | B: 1

---

File: T0033_DEBUG.txt
Pair:(1, 2)

18:19:00.7966 | State: SetDefaults | A: 0 | B: 0
18:19:00.7986 | State: Configure   | A: 1 | B: 2
18:19:00.8006 | State: DataLoaded  | A: 1 | B: 2
18:19:00.8216 | State: Historical  | A: 1 | B: 2
18:19:00.8226 |     2023-03-30 12:00
18:19:00.8226 |     2023-03-30 12:15
18:19:00.8236 |     2023-03-30 12:30
18:19:00.8236 |     2023-03-30 12:45
18:19:00.8246 |     2023-03-30 13:00
18:19:00.8246 |     2023-03-30 13:15
18:19:00.8256 |     2023-03-30 13:30
18:19:00.8256 |     2023-03-30 13:45
18:19:00.8266 |     2023-03-30 14:00
18:19:00.8266 |     2023-03-30 14:15
18:19:00.8276 |     2023-03-30 14:30
18:19:00.8276 |     2023-03-30 14:45
18:19:00.8286 |     2023-03-30 15:00
18:19:00.8375 | State: Terminated  | A: 1 | B: 2

---

File: T0036_DEBUG.txt
Pair:(2, 1)

18:19:00.7966 | State: SetDefaults | A: 0 | B: 0
18:19:00.7986 | State: Configure   | A: 2 | B: 1
18:19:00.8006 | State: DataLoaded  | A: 2 | B: 1
18:19:00.8216 | State: Historical  | A: 2 | B: 1
18:19:00.8226 |     2023-03-30 12:00
18:19:00.8226 |     2023-03-30 12:15
18:19:00.8236 |     2023-03-30 12:30
18:19:00.8236 |     2023-03-30 12:45
18:19:00.8246 |     2023-03-30 13:00
18:19:00.8246 |     2023-03-30 13:15
18:19:00.8256 |     2023-03-30 13:30
18:19:00.8256 |     2023-03-30 13:45
18:19:00.8266 |     2023-03-30 14:00
18:19:00.8266 |     2023-03-30 14:15
18:19:00.8276 |     2023-03-30 14:30
18:19:00.8276 |     2023-03-30 14:45
18:19:00.8286 |     2023-03-30 15:00
18:19:00.8385 | State: Terminated  | A: 2 | B: 1

---

File: T0087_DEBUG.txt
Pair:(2, 2)

18:19:00.7966 | State: SetDefaults | A: 0 | B: 0
18:19:00.7986 | State: Configure   | A: 2 | B: 2
18:19:00.8006 | State: DataLoaded  | A: 2 | B: 2
18:19:00.8216 | State: Historical  | A: 2 | B: 2
18:19:00.8226 |     2023-03-30 12:00
18:19:00.8226 |     2023-03-30 12:15
18:19:00.8226 |     2023-03-30 12:30
18:19:00.8236 |     2023-03-30 12:45
18:19:00.8236 |     2023-03-30 13:00
18:19:00.8246 |     2023-03-30 13:15
18:19:00.8246 |     2023-03-30 13:30
18:19:00.8256 |     2023-03-30 13:45
18:19:00.8256 |     2023-03-30 14:00
18:19:00.8266 |     2023-03-30 14:15
18:19:00.8266 |     2023-03-30 14:30
18:19:00.8276 |     2023-03-30 14:45
18:19:00.8276 |     2023-03-30 15:00
18:19:00.8385 | State: Terminated  | A: 2 | B: 2

---

The preceding files are all as expected! They pass through all states up until the Terminated state.
However, the remaining 4 files contain unusual information.

---

File: T0012_DEBUG.txt

This file seems to repeat the pair 2,1, but doesn't make it to the Terminated state. This seems like an anomaly. Why repeat a pair that was already processed on a different thread?

[18:19:01.0290 | State: Configure | A: 2 | B: 1
18:19:01.0300 | State: DataLoaded | A: 2 | B: 1
18:19:01.0320 | State: Historical | A: 2 | B: 1
18:19:01.0320 | 2023-03-30 12:00
18:19:01.0320 | 2023-03-30 12:15
18:19:01.0330 | 2023-03-30 12:30
18:19:01.0330 | 2023-03-30 12:45
18:19:01.0340 | 2023-03-30 13:00
18:19:01.0340 | 2023-03-30 13:15
18:19:01.0340 | 2023-03-30 13:30
18:19:01.0350 | 2023-03-30 13:45
18:19:01.0350 | 2023-03-30 14:00
18:19:01.0360 | 2023-03-30 14:15
18:19:01.0360 | 2023-03-30 14:30
18:19:01.0360 | 2023-03-30 14:45
18:19:01.0370 | 2023-03-30 15:00

---

File: T0021_DEBUG.txt

I don't know what is going on in this thread. I see values that are completely outside the range of the pairs.

18:19:00.0920 | State: SetDefaults | A: 0 | B: 0
18:19:00.8415 | State: SetDefaults | A: 0 | B: 0
18:19:00.8934 | State: Terminated | A: 20 | B: 14
18:19:01.0111 | State: Terminated | A: 2 | B: 1
18:19:01.0121 | State: Terminated | A: 20 | B: 14
18:19:01.0121 | State: SetDefaults | A: 0 | B: 0
18:19:01.0280 | State: SetDefaults | A: 0 | B: 0
18:19:01.0500 | State: Terminated | A: 2 | B: 1
18:19:01.5537 | State: SetDefaults | A: 0 | B: 0
18:19:01.5547 | State: Terminated | A: 20 | B: 14

---

File: T0002_DEBUG.txt

Another unusual thread with invalid values.

I have no idea how this thread managed to output a line without the prefix timestamp.

18:19:00.357 | State: Terminated | A: 20 | B: 14
State: Terminated | A: 20 | B: 14
18:19:00.364 | State: Terminated | A: 20 | B: 14
18:19:00.797 | State: Terminated | A: 20 | B: 14

---

File: T0004_DEBUG.txt

Only one line in this thread, but with invalid values.

18:19:00.0970 | State: Configure | A: 20 | B: 14

---

---

Question 1: Aren't these last 4 threads unusual?
Question 2: What are they trying to do?

I can see how they might confuse coding logic and potentially cause unexpected results.

Attached are relevant files. I could not attach more than 5 files, so the actual debug files are not included..

Thanks for your reply.

Ray

[P.S. I am not able to edit my typo in the posting title.]

[waltFX]

I use the Optimizer ALOT and I found inconsistencies and submitted bug-reports that were verified to be issues, BUT "the development team is looking at it" with no ETA.
>> normal optimization results are pretty good but I would not trust the "walk-forward" feature which definitely has issues that make the results un-useable...

[QuantKey_Bruce]

I use the Optimizer ALOT and I found inconsistencies and submitted bug-reports that were verified to be issues, BUT "the development team is looking at it" with no ETA.
>> normal optimization results are pretty good but I would not trust the "walk-forward" feature which definitely has issues that make the results un-useable...

waltFX Can you elaborate any on specific things about walk-forward that make it unusable?

[waltFX]

Bruce,
As I trade Forex, which sometimes has over 50 trades in a day, I set-up a WF for an opt period of 2 days and a test of one day (yes that crazy)...
First, any Friday trades are propagated to Saturday (and sometimes Sunday) so the results are inflated.
Second, running the WF with the exact same parameters, returns different results most times.
Repeatedly running with the same parameters returns the same GROUP of results, meaning the same result would pop-up at least once in five runs...
Also, Checking/unchecking the "Break at EOD" screwed things up even more.

>> I demonstrated this using the sample SMA strat so NT did see the issues to log the bug...

As mentioned, a normal optimization seems to work OK so I built an Auto-it (of course) to set the dates, run a normal optimization and save the result grid into a CSV.
This takes MUCH more time to run over 5 months (about 2hours) but I import the result files into SQL to get useable results.
>> as "auto-it" takes over the PC, it is un-useable while processing (and sorry, I won't share the script as it is very complex)

[QuantKey_Bruce]

Bruce,
As I trade Forex, which sometimes has over 50 trades in a day, I set-up a WF for an opt period of 2 days and a test of one day (yes that crazy)...
First, any Friday trades are propagated to Saturday (and sometimes Sunday) so the results are inflated.
Second, running the WF with the exact same parameters, returns different results most times.
Repeatedly running with the same parameters returns the same GROUP of results, meaning the same result would pop-up at least once in five runs...
Also, Checking/unchecking the "Break at EOD" screwed things up even more.

>> I demonstrated this using the sample SMA strat so NT did see the issues to log the bug...

As mentioned, a normal optimization seems to work OK so I built an Auto-it (of course) to set the dates, run a normal optimization and save the result grid into a CSV.
This takes MUCH more time to run over 5 months (about 2hours) but I import the result files into SQL to get useable results.
>> as "auto-it" takes over the PC, it is un-useable while processing (and sorry, I won't share the script as it is very complex)

waltFX When you say running the WF with the exact same parameters returns different results most times, under Optimizer do you have this set to default or genetic?

Some of these other issues it sounds like coincide with what's being reported in this thread e.g. repeat evaluations of the same parameter set...

Regarding Break at EOD, whether this should affect reproducibility depends on the bar type. For instance, minute bars would not typically be affected if they're of a size that a day is integrally divisible by e.g. 1-min, 60-min, 30-min, etc. If it's something like a tick bar or a Renko bar, unless it's of the minimum size, it will be completely different bars depending on the starting point if Break at EOD is unchecked...

[waltFX]

all my opts are 1 minute bars... I understand the unique issues surrounding "break at EOD"
I don't use "generic"...
But "the developers are looking at it" LOL!
> I am probably a tiny minority of users actually trying to use NT this way so it follows that it will take forever (if ever) to resolve given all the other bugs that are on the table...

BTW; I've had NT in moth-balls for quite a while and only switched back from Multicharts due to their support (and bugs) being even worse!

[QuantKey_Bruce]

Welcome back. Let's see if we can't get a better understanding of the optimizer bugs and maybe if we can clearly demonstrate and document what's wrong a fix can be implemented. It's true that not many users push the boundaries of what's possible so we may have to be clear and careful in creating a test case that shows what is wrong.

I strongly urge you to do some careful testing and show what's wrong. If you create a test case, I will gladly help you test and confirm any specific issue if I can. It's in our interest to get any issues resolved - or at least very clearly understood - and I presume that if NinjaTrader could clearly reproduce a problem and understand exactly what is wrong, they would fix it. Note that there's a huge difference between a complaint that "something is wrong" or "my results are different" and a careful test case that shows case A with no problem and case B with a problem, and here's the one line of code or setting that's different that demonstrates the problem. Then they have something they can easily tackle without going on a fishing expedition - something that is harder to justify if there are 999 irons in the fire already, like right now when they've just rolled out web and mobile and a new account structure. I am sure they would want to fix anything very clearly wrong when they know exactly how.

[NinjaTrader_Emily]

Question 1: Aren't these last 4 threads unusual?
Question 2: What are they trying to do?

I can see how they might confuse coding logic and potentially cause unexpected results.

Attached are relevant files. I could not attach more than 5 files, so the actual debug files are not included..

Thanks for your reply.

Ray

[P.S. I am not able to edit my typo in the posting title.]

Hello Ray,

Thank you for your post.

Optimizations are run in parallel with each other with one instance per logical processor. Not all instances are used for the test itself; some instances are administrative and relate to the NinjaScript lifecycle. Here is another thread that relates to scripts and their reported states:

https://forum.ninjatrader.com/forum/historical-beta-archive/version-8-beta/85086-why-is-state-terminated-run-when-another-different-indicator-is-loaded?p=726288#post726288

waltFX, and QuantKey_Bruce,

Your discussion here about optimizations and existing behavior that has already been reported does not seem completely relevant to what Ray has started this thread about. If you have any ID numbers related to the reports that have been made to the development team, I can check for any updates if they have not already been posted in the original threads that prompted the behavior to be reported in the first place.
I understand that it can be frustrating not having an ETA on when these items will be investigated and/or when a fix might be implemented; as with other feature requests and bug reports, we cannot present an ETA as they are fulfilled based on the Development and Product Management team's schedule and priorities. Upon implementation, the tracking ID can be found publicly on the Release Notes page of the help guide:

https://ninjatrader.com/support/helpGuides/nt8/release_notes.htm

Please let me know if I may be of further assistance.

[waltFX]

Bruce,
I DID give them specific examples which did reproduce the bugs and they did log them...

Emily,
This was info was just so that Rgordie knew that there are outstanding issues...

[QuantKey_Bruce]

waltFX, and QuantKey_Bruce,

Your discussion here about optimizations and existing behavior that has already been reported does not seem completely relevant to what Ray has started this thread about. If you have any ID numbers related to the reports that have been made to the development team...

My apologies for having a bit of a sidebar discussion there. That should probably have been in another thread somewhere to keep this tightly focused on the original report.

[QuantKey_Bruce]

Bruce,
I DID give them specific examples which did reproduce the bugs and they did log them...

If you would please PM me these threads if you have them and we'll take this discussion outside.

[rgoudie]

Hi, Bruce.

I see the two *.cs files as being attached and clicked on both and they downloaded as expected. Is this not your experience?

Also, what happened to the .cs attachments on this post?

[QuantKey_Bruce]

Hi, Bruce.
I see the two *.cs files as being attached and clicked on both and they downloaded as expected. Is this not your experience?

They're back now. Must have been something temporary with the forum. Sorry to bother.

[rgoudie]

Hi, Emily.

Thank you for your reply.

I did read the NinjaScript lifecycle page as well as the multi-thread page. They were informative. i will implement some of the theories that were recommended.

However, as a software engineer with a few decades of experience, I still cannot help but feel that there are some anomalies in the optimization flow, especially the thread that tries to repeat the pair 2,1. (File: T0012_DEBUG.txt​) I would appreciate if a development person could replicate this test and validate whether that particular thread is appropriate.

I notice that the valid threads all pass through the SetDefaults state followed by the Configure state. None of the later threads do this in this particular order. As a workaround, I will implement logic that stores information in a ConcurrentBag based on thread identifier. I will only process a thread if it passes through SetDefaults followed by Configure. I will ignore all threads that have not done this sequence of calls. This seems reasonable for the time being.

If you have any further suggestions, do not hesitate to post them here.

Ray

Hello Ray,

Thank you for your post.

Optimizations are run in parallel with each other with one instance per logical processor. Not all instances are used for the test itself; some instances are administrative and relate to the NinjaScript lifecycle. Here is another thread that relates to scripts and their reported states:
https://forum.ninjatrader.com/forum/...288#post726288

​Please let me know if I may be of further assistance.

[rgoudie]

I totally forgot to mention my motivation for wanting to implement my workaround!

I compile and store some results of each combination of properties. I need to know when a thread completes so that I can store the results.
I also need to know when the entire optimization has completed so that I can then compile results based on each individual run.