there is a lot that could be improved with nt 7 and 8's optimization engines.

[rtwave]

people with nt, in this thread i will try to share how the experience of using nt as a client is. i will also describe some malfunctions i have run into with nt's optimization engine as soon as i have the time.


at the moment i will just share this screengrab below:




i have 10 different profitable automated strategies and on the 8th of april 2020 i decided to run 4 optimization processes for each one of my strategies. two runs for the m2k contract and two for the mnq contract. i would then select one strategy to trade automated on the m2k contract and another on the mnq contract.

almost 7 full days later i haven't been able to complete these optimization processes. i have only been able to run something like 25 processes in total. i have spent several days on this endeavor and it still has not been enough time devoted to this objective.

the way nt optimization engine works makes it tedious, boring, exhausting and exasperating to run a series or processes. even when i have learned to use strategy analyzer window templates and this does save me a little time, i still have to run one optimization at a time and then save two screengrabs with the results and optimal values for each run.


functions that should be very basic like saving results, saving optimal values, keeping and adding out of sample data or even being able to set a long series of optimization processes and then have the platform run them all sequentially are impossible to use in nt.


i had already shared this information with the people at nt both on these fora and by email, but this particular forum is ideal to report all these matters so i will be expanding this thread when i have the time to do so.


very well, hopefully these reports can gain some traction with other users so that the appropriate improvements can be made to the nt platform. thanks, regards.

[NinjaTrader_Kate]

Hello rtwave,

Thank you for your post.

I understand your frustration. The NinjaTrader optimization engine is optimized to take advantage of as much system resources as possible, and it is advisable not to trigger an optimization during a time where you would need to be using the PC for anything other than optimization. However, there are a number of ways in which you may be able to reduce your CPU and memory usage during an optimization run and possibly speed up the process somewhat.

I don't know how many parameters you are optimizing your strategy on, but strategies with a relatively large number of parameters for optimization can multiply the impact on your PC resources.

When working with strategies with a large number of parameters, avoid using "1" as the increment value for the parameters, to avoid forcing the optimizer algorithm to run the maximum number of permutations. Changing the increment value to as little as "2" can cut the number of permutations in half.

Also, you may want to try testing using the Genetic algorithm instead of the Default. The default algorithm tests every single combination of parameters within the optimization range to find the best possible result. For more complicated optimizations this can take some time to complete, as you've seen. The Genetic algorithm, on the other hand, can be more useful for more extensive optimizations, because it doesn't test every combination - instead, it works on something similar to evolution via natural selection.

Genetic optimization will find the approximate optimal solutions, but since it doesn't test every possible combination it's not guaranteed that the optimization would be the absolute best.

Here's a link to our help guide that goes over Genetic optimization:

https://ninjatrader.com/support/helpGuides/nt7/genetic_algorithm.htm

If this is a strategy you have coded yourself, there is also the IsInstantiatedOnEachOptimizationIteration property you can set in State == State.Configure.

The default behavior of the Optimizer is to re-create the strategy for each optimization run. However, this process requires more time and computer resources to return results, which could impact the amount of time it takes to run an optimization.

When this property is set to false, the strategy is re-used to save time and computer resources. When this is set to false, internal properties are reset to default values after each iteration, but it is possible that user-defined properties and other custom resources may carry their state over from the previous iteration into a new backtest run. To take advantage of performance optimizations, developers may need to reset class level variables in the strategy otherwise unexpected results can occur.

You can find out more information regarding this property in our help guide link below:

https://ninjatrader.com/support/helpGuides/nt8/en-us/isinstantiatedoneachoptimizationiteration.htm

Also, it sounds like you're wanting to automate some of the process so you could run multiple optimizations in a row and save the results, is that correct?

I have added a vote for you to a current feature request to allow UI automation of processes so you could run multiple optimizations more easily. This request is being tracked under the number SFT-631.

As with all feature requests, interest is tracked before implementation is considered, so we cannot offer an ETA or promise of fulfillment. If implemented, it will be noted in the Release Notes page of the Help Guide.

Release Notes — https://ninjatrader.com/support/help...ease_notes.htm

Please let us know if we may be of further assistance to you.

[NT-Roland]

Hi Kate,
Please add my vote to SFT-631.
Can you please explain how to link back SFTs to release notifications?
The latest release seems to address issue 14376, 14088, 14356.
I can't find any SFTs even close to these numbers.
Thx.
NT-Roland

[NinjaTrader_Kate]

Hello NT-Roland,

Thank you for your reply.

I've added a vote for you to SFT-631 as well.

The numbers listed in the release notes are not SFT numbers, and should not be looked at as such. You should focus on the description of the change rather than the number listed in the release notes, as the listed "issue number" is for the actual work ticket from the development team for the changes.

Please let us know if we may be of further assistance to you.

[rtwave]

Kate,




i had not had time to continue with this thread. i have had a ridiculous number of things to get done.



i want to point out that all the optimization processes i run have always used the genetic engine. and it is quite messy as it is, i use 150 for the number of generations and generation size and once it gets started the estimated time to completion always displays 4 to 7 days, then as process is made it goes down to 20 something hours and then in the end it is always something between 2 and 3 hours for the entire process to run its course. the people with nt never use the platform to get any work done, so they are not aware of these situations. as i have documented, it takes multiple days to evaluate different strategies and it is a tedious and exhausting process which needs to be improved significantly.



the suggestion that was mentioned to use instantiated or not instantiated processes without nt support providing some examples is not very helpful. i have no clue what a class level variable is and whether i have ever used them or not. one sample strategy to be optimized using the instantiated option and the same strategy prepared to optimize without instantiating would be necessary so that ordinary users can understand these possibilities.




i also want to report a bad malfunction i have suffered all too frequently in this thread. it is a common occurrence that i run an optimization and get one result with positive profits and the other 19 are blanks. this is just wrong and the problem lies with the algorithm. if there is one result with positive profits, then changing just one of the parameters by one unit will generate a sub-optimal result but it should still be comfortably a positive profit and should be reported. the algorithm is wrong if it can find one local maximum and then nothing else as there will be an abundance of positive results around this point. this can be easily proven mathematically, so nt is doing something wrong with their genetic algorithm. also, the algorithm should start its evaluations by checking out all the values in the middle and at the extremes of the ranges that have been declared for all of the variables.


i include several screengrabs to illustrate this issue, i have collected tens of these:















i couldn't include more than 5 screengrabs but i have tens of these as this is a recurrent malfunction.


i think that i should be able to create one sample strategy that nt support could adapt for instantiated - not instantiated alternatives and i should also a sample strategy that generated this kind of nonsensical blank results. i don't know when i could have these sample strategies ready but i will try to.



very well, regards.

[NinjaTrader_Kate]

Hello rtwave,

Thank you for your reply.

Here's a good publicly accessible link that explains the differences in variable scope and goes over what a class level variable is:

https://www.geeksforgeeks.org/scope-of-variables-in-c-sharp/

Further, there is an example in the link to our help guide I provided previously that shows how to take advantage of performance benefits from setting IsInstantiatedOnEachOptimizationIteration to false. I'm providing the example below:

// A custom trades dictionary is declared when strategy is first optimized,
// but not instantiated until later in State.DataLoaded,
private Dictionary<DateTime, string> myTrades;

// examples of other fields which need to be reset
private double myDouble;
private bool myBool;
private DateTime myDateTime;
private Order myOrderObject;
private Brush myBrushObject;
private SMA mySMAIndicator;
private Array myIntArray;
private List<object> myList;
private Series<double> mySeries;

protected override void OnStateChange()
{
  if (State == State.SetDefaults)
  {
    Name = "My Optimization Test 2";
    Description = "Demonstrates manually resetting a class level variable without re-instantiating the strategy";
    Fast = 10;
    Slow = 25;

    // in this case, we do not need to re-instantiate the strategy after each optimization
    // because we are explicitly resetting the custom trade dictionary in State.DataLoaded
    // This design of re-using the strategy instance comes with performance benefits
    IsInstantiatedOnEachOptimizationIteration = false;
  }

  else if (State == State.DataLoaded)
  {
    // re-create custom trade dictionary on each optimization run
    // we are guaranteed to start with a new object on each optimization run
    if (myTrades != null)
      myTrades.Clear();
    else
      myTrades = new Dictionary<DateTime, string>();

    //Any strategy defaults which are maintained do not need to be reset if they are not mutable as the strategy runs.
    //Any strategy state that would be mutable after State.SetDefaults needed to be reset for the next run.
    myDouble = double.MinValue;
    myBool = false;
    myDateTime = DateTime.MinValue;
    myOrderObject = null;
    myBrushObject = null;
    mySMAIndicator = SMA(14);

    if (myIntArray != null)
        Array.Clear(myIntArray, 0, myIntArray.Length);
    else
        myIntArray = new int[20];

    if (myList != null)
        myList.Clear();
    else
        myList = new List<object>();

    mySeries = new Series<double>(this);
  }
}

protected override void OnBarUpdate()
{
  if (CurrentBar < BarsRequiredToTrade)
    return;

  if (CrossAbove(SMA(Fast), SMA(Slow), 1))
  {
    EnterLong();
    myTrades.Add(Time[0], "long");

  }
  else if (CrossBelow(SMA(Fast), SMA(Slow), 1))
  {
    EnterShort();
    myTrades.Add(Time[0], "short");
  }
}

[Range(1, int.MaxValue), NinjaScriptProperty]
[Display(Name = "Fast", GroupName = "NinjaScriptStrategyParameters", Order = 0)]
public int Fast
{ get; set; }

[Range(1, int.MaxValue), NinjaScriptProperty]
[Display(Name = "Slow", GroupName = "NinjaScriptStrategyParameters", Order = 1)]
public int Slow
{ get; set; }

For the last testing issue, this looks like it may be data related. Are you able to duplicate these results using the built-in Sample MA Crossover strategy? Please ensure also that you're using the latest version of NinjaTrader before testing. You can find the version number under Help > About - if it's the latest version the version number should be 8.0.21.1.

If you have any other version, please update to the latest version prior to testing.

To update NinjaTrader, please follow the steps below:

• First, copy your license key from NinjaTrader under Help> License Key then exit NinjaTrader
• Click on the link: http://ninjatrader.com/PlatformDirect
• Enter your license key and press Submit
• Select 'NinjaTrader 8'
• Select 'Download'
• Critical: Before running the installer, ensure NinjaTrader is closed.

Thanks in advance; I look forward to assisting you further.

[rtwave]

Kate,



thanks.


i see the code you shared.


if i wanted to modify my strategies so that i could use this not instantiated version i would have to completely take them apart and create an entirely different version by hand. i don't think i would be able to as i have had problems even with very simple strategies that duplicate the position sizes they should take. the code that was shared is not helpful either because, ¿would i have to delete all the ninjatrader generated code at the end of all my strategies so that this would work? and from the annotations, one must arrive at the conclusion that one must create private variables for everything inside the strategy and then have the fragments that reset the values for every one of the components in a strategy.

and it is not obvious to me what the benefit would be of doing this either. only if i could run exhaustive optimization processes that took reasonable amounts of time would it make sense for me to go into such a difficult and complicated undertaking.



now, regarding the malfunction i reported, this has nothing to do with the version of nt. nt has generated this kind of failed results since october of 2019 that i started running optimization processes and this has continued with regularity up to now. i have saved screengrabs as evidence and what i ordinarily do is just run the very same process again and most of the time the results will then be 20 positive values and minimally more credible. that's why i usually run at least two optimization processes for every strategy and instrument i'm interested in, because nt returns failed results all the time.

nt's sample strategy has only two inputs so it would never be necessary to use the genetic algorithm and this malfunction could never happen.

i have managed to duplicate these failed results with a sample strategy that i can share with nt support, so i will be sending it by email.



in this thread i have proven that having to use nt's optimization engine is a killer proposition. it takes days and weeks of tedious and boring work to get anything done. and not only that, frequently nt will take hours only to return failed and useless results.

[NinjaTrader_Jim]

Hello rtwave,

The failed optimization iterations look to be what I would expect if NinjaTrader did not have the data to complete the optimization iteration. You wrote in to us over email and we are working with you more closely regarding that case.

Optimizations are known to be very resource intensive. Kate's insight on programming a strategy so it is compliant with IsInstantiatedOnEachOptimizationIteration = false would be our recommendation for designing a script that would be more performant with optimizations. We would also recommend disabling the use of Drawing Objects when possible to improve backtest/optimization performance, but this is not applicable for your strategy as it does not use Drawing Objects.

For the thread's reference, you can check IsInStrategyAnalyzer to control your logic which calls drawing methods, and this will improve backtest times.

IsInStrategyAnalyzer - https://ninjatrader.com/support/help...egyanalyer.htm

We look forward to assisting.