Announcement

Collapse
No announcement yet.

Partner 728x90

Collapse

NT8 - commercial patterns to backtesting and optimisation

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    NT8 - commercial patterns to backtesting and optimisation

    As a Quantative Analyst/strategy Analyse i want to research systems, data series and settings over a portfolio of instruments - and take the least amount of time to do it and to be able to establish a set of tests that are significant and not run the discarded set in the future.

    To do this:
    #1 I want to save the fit settings from the optimiser - as a numeric code or as a text description...
    #2 ideally these should be auto saved and exported at the end of the run... and not have the manual task of having to select and manually save - so to be able to select a checkbox and batch save these to a subfolder with a name and incremental numeric to give unique file names is the easiest way
    #3 I want to group this settings/templates and save them -subfolder for ease - this would be associated with the strategy folder
    #4 later i want to run a strategy test/optimisation of these "fit settings" - by giving the strategy analyser the list of templates to load and run against a list of instruments and data series and optimise - then study the results

    Therefore focusing the efforts on the subset of approved settings and not wasting time optimising parameters that fall outside this set. = the fastest possible and most useful type of optimisation.... as it is the result of a pyramid of time and effort and represent the top....the part that is important...


    I tried this with NT7 - what i found was that i could load the list and walkthrough but NT7 then copied over the values from the optimizer parameters - which had not changed so therefore making the test invalid...

    In NT8 - if would be great if it only set the parameters which changed... which were incremented.... and did overwrite the values of parameters with values out of scope - this allowing back testing of sets of settings - for example by enumerated a folder and load the settings and setting the parameters on the configuration/initialise settings event state.

    Then you got something really valuable...

    Currently you can do this workaround in NT7:
    So then you have to write over the settings again on the first bar - or on start-up, the issue occurs with adding data series if this is not done this way - on the init event

    So for NT8 the thing required from NT is:

    1. UI Changes to allow bulk saving of the settings to a target subfolder with a name
    2. We need to control what gets set - save time in the optimiser by not setting values that are not going to change e.g. Param 1 - 1:1:1 should not be changed-. Only change the params that are marked for optimisation - e.g. .param 2 - 1:100:1 - then this will achieve a window to load settings on the fly e.g.
    A: Param Setting ID in 1:100:1 - this numeric field is based on the enumerated list of templates.... it could be split from the file name stored on disk
    1:Strategy1.xml
    2:Strategy1.xml
    3:Strategy1.xml

    For each optimizer iteration of Param Setting ID - strategy base class will load the associated setting - mapped to the ID - the friendly name will also load in a parameter “template” for ease of so then the template settings are loaded one by one

    B: Another option is to allow to optimise a list of string....
    public string[] templates

    So to achieve this on first instantiate of the strategy use the class defaults to begin with -
    The optimiser should then look at the user set list of parameters that will be optimised...
    Allowing load of Secondary Data series at the appropriate state
    It should load and work from the list of parameters that will change - and then set them iteratively - it should not iterate through the whole property list of category "Optimiser NinjaTraderProperty" and so should NOT set parameters not market for change

    - lightening the load/fast execution than iterating through all params for a strategy - not setting what is not getting optimiser set and allow a more extensible pattern and a huge benefit as you then optimise list of templates - via a numeric...


    I will probably need to supply some screenshots of this process as my writing is probably not good :-)
    Last edited by MicroTrends; 05-09-2015, 11:33 PM.
    MicroTrends
    NinjaTrader Ecosystem Vendor - micro-trends.co.uk

    #2
    Hi,

    I wanted to let you know that this request is on my radar, but would be a pretty big spec implementation on our part and i'm still trying to fully process this request before sending in. If you have any visual aids that you may help fully get your ideas across, please let me know.
    MatthewNinjaTrader Product Management

    Comment


      #3
      Originally posted by NinjaTrader_Matthew View Post
      Hi,

      I wanted to let you know that this request is on my radar, but would be a pretty big spec implementation on our part and i'm still trying to fully process this request before sending in. If you have any visual aids that you may help fully get your ideas across, please let me know.
      ok sure will do -

      NT7 Implementation and background:
      in NT7 we allowed users to save settings in an index with "Zone Setting ID" and "Zone Setting Name" "Zone was the prefixed user to try to push the parameter to the end of the list"

      For Local optimisation and iteration of settings: Really this is best handled as a collection of files in a folder... as you are doing now - not as we did in the following...but below it gives you the flow with screenshots of how we implemented some of it...

      USING an INT index
      An index file can be built locally as a xml link file... id and file name
      to give a consist locally id that allows the optimiser to run through a list of settings files... of int array for example... holes in the index yield no result/settings... or defaults.
      this would be locally unique only - index holes should be kept and not filled to preserve linked data referential integrity

      So here we have a repository of settings: 1 file per strategy
      Free online storage and sharing with Screencast.com. 2 GB of storage and 2 GB of bandwidth per month for free. We won't compress, alter or take ownership of your content.


      Settings are selected from a dataset via the "setting id" from with a strategy or indicator - from the parameters list on a chart or anywhere else...
      So this also works for Strategy Analyzer
      Free online storage and sharing with Screencast.com. 2 GB of storage and 2 GB of bandwidth per month for free. We won't compress, alter or take ownership of your content.


      Setting the Zone Setting ID public param - triggers a property changed event and loads the associated data/settings...


      Optimisation
      optimise via setting the id index - Zone Setting ID
      and so loading the setting from disk or from a dictionary with the objects loaded etc
      Free online storage and sharing with Screencast.com. 2 GB of storage and 2 GB of bandwidth per month for free. We won't compress, alter or take ownership of your content.


      USING Setting Template Name ( file name)
      Alternatives to using "ID"
      The alternative is to use use a string [] of template names
      and use this as a the key and sorted index
      then in the optimiser iterate through this....
      Easier to maintain and achieve i guess :-)
      this would be the least amount of work and still be extensible...

      Bulk Saving - Settings etc
      So next i will add some viso or excel shots to show a batch save from the strategy analyzer... even that could be automated after the manual process implemented


      GATHERING the Settings/Saving
      1. From visual backtests - on a chart
      2. From third party/external
      3. single or batch saved from Strategy Analyzer


      NEW SPINOFF FEATURE: SETTINGS PERFORMANCE STATS -
      The settings file/Key could be correlated to a dataset of performance stats/dataseries instrument - and this saved as an explorable local dataset or even uploaded into some group or public online settings/stats portal -so that settings can be downloaded and used in strategy etc... and that can be automated also to various degrees etc
      ...so this would be an add on to the functionality...described... settings templates and results exported and correlated....... so settings templates can be scored... and explored....and loaded
      Last edited by MicroTrends; 05-12-2015, 10:14 PM.
      MicroTrends
      NinjaTrader Ecosystem Vendor - micro-trends.co.uk

      Comment


        #4
        Hi Microtrend,
        My thoughts along these lines were if they exposed the optimization engine objects then you could implement your custom optimizations any way you want through an AddOn.

        Comment


          #5
          Originally posted by GrumpyTrader View Post
          Hi Microtrend,
          My thoughts along these lines were if they exposed the optimization engine objects then you could implement your custom optimizations any way you want through an AddOn.
          Agreed in part - but no custom optimization needed as such - The loading and optimizing of settings is already achievable - but limited- really the main point of the request is to:

          A: Alter the NT native Strategy Analyzer UI report window to allow bulk save of templates - for use in optimisation later. From the Strategy Analyzer report screen - a column with a check box for example and a save button - or failing that a "save all"

          B: To change the optimiser internally to not overwrite parameters with values from the optimiser parameters that dont change eg 1:1:1 so that loaded template paramters in properties are not overwritten - so that it would work off a list of parameters that will change and not iterate the properties regardless of state -saving cycles and allow extensibility

          c. The Strategy Analyzer - additionally should save a CSV file of perf stats - with a index column to associate the data to the settings template. To allow usage of data such as local ad hoc reporting, data mining and the correlation of results against other data points.....

          The results and template Data could be opened in Excel and would be enormously helpful to people to organise their universe - as currently i doubt 1% work proficiently with these tools i would imagine - these features would allow insight and easy access for all.

          The NT7 user case was only supplied to demonstrate the usage of optimising templates settings as in integer - a string array would also be good - for numerous reasons. The loading and optimising of settings is already achievable - but when loading dataseries it causes issues due to B.

          Custom Optimiser
          If this was not provided then - a custom optimiser might be able to dump out settings and templates at the end of a run.. according to some criteria... Point B is still an issue - and Point A is certainly required - for user time saving -

          The ultimate Goal is to allow
          1. A user to store templates and associated performance stats
          2. to then run further backtest/optimisations against these benchmarks
          3. to expand the dataset and to be able to chose their settings for trading...

          Key advantages
          So you are optimising and testing with an optimal set - the top of the pyramid
          not wasting weeks and months repeating parametesets that are known to be outside the set...etc
          You are able to make empowered and able to clear decisions based on your own database of settings and perf stats... storage, reporting, retrieval -

          So then it possible to leverage the technology to make everything quicker and providing a workflow and approach not available.. so bring that to NT and everyone can benefit... i believe the Add on will not negate point B or provide point A
          Last edited by MicroTrends; 05-13-2015, 09:07 PM.
          MicroTrends
          NinjaTrader Ecosystem Vendor - micro-trends.co.uk

          Comment


            #6
            The optimizer suggestions I will reply here after I finish my research. I feel that this reply will give you what you need programmatically else I will record this feedback for NT to implement.

            Comment


              #7
              Originally posted by NinjaTrader_Brett View Post
              The optimizer suggestions I will reply here after I finish my research. I feel that this reply will give you what you need programmatically else I will record this feedback for NT to implement.

              http://www.ninjatrader.com/support/f...ad.php?t=73940
              Perfect if B can be removed - and yes as Grumpy asks to backtest a basket of systems against a basket of instruments and to kick it off using system automation - save results...et re-run the settings through - to save zillions of wasted iterations with the "outside set" of parameters that can be excluded by running through what is best by settings and system ranking too etc
              Last edited by MicroTrends; 05-18-2015, 08:25 PM.
              MicroTrends
              NinjaTrader Ecosystem Vendor - micro-trends.co.uk

              Comment

              Latest Posts

              Collapse

              Topics Statistics Last Post
              Started by jaybedreamin, Today, 05:56 PM
              0 responses
              7 views
              0 likes
              Last Post jaybedreamin  
              Started by DJ888, 04-16-2024, 06:09 PM
              6 responses
              18 views
              0 likes
              Last Post DJ888
              by DJ888
               
              Started by Jon17, Today, 04:33 PM
              0 responses
              4 views
              0 likes
              Last Post Jon17
              by Jon17
               
              Started by Javierw.ok, Today, 04:12 PM
              0 responses
              12 views
              0 likes
              Last Post Javierw.ok  
              Started by timmbbo, Today, 08:59 AM
              2 responses
              13 views
              0 likes
              Last Post bltdavid  
              Working...
              X