Announcement

Collapse
No announcement yet.

Partner 728x90

Collapse

NT Automation Feasible?

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    #16
    Anytime Dan.

    For me I do something similar but instead of scanning a bunch of symbols looking for certain things I've written that in to the main buy/sell strategy and just run it on ~500 symbols everyday. I gathered my symbol list from filters on barchart.com's Stock Screener for volatility, avg price range, and etc. Then once per year I'll rescan and remake that active symbol list. I created my strategy to only invest X dollars per buy with a min account value left for any account fees and etc. This is also what I believe is the better way to diversify as A) the symbol list is spread across all industries and B) my investments aren't heavily focused on just a small handful of stocks spreading things out even more as in general at any given time 50% of stocks are going down and 50% are going up as people shift money from one place to another.

    This approach, using smaller investments across many stocks, allows you to better play with and be safe with crazy ideas such as buying at a 52 week low and selling at a 52 week high. From an optimists, mean reversionists, approach either the stock will return as that's always the goal of the CEO or the company goes out of business. If it goes bankrupt then no big deal because you only had say 200-500 dollars invested anyways and statistically the others that rebounded far outweighs the companies that went under. I'm not saying to do that but a highly diverse playing field allows for trying all the things the 'big guys' say to never do but could return huge profits.

    One last note on testing unless you've written something specific for a specific symbol is to always test your strategy across hundreds of stocks using the Strategy Analyzer. It is easy to write code that curve fits to a single stock but testing it across say 500 stocks and it still does well says that the code is robust and will mostly likely work well as general market changes slowly occur. Also testing across 500 symbols means your code has pretty much seen all that is possible to be seen with fast dips, gaps up/down, choppy terrain, and long slides up/down. Then just remember that it's a numbers game and that it's impossible to win them all but if you're only using smaller amounts per investment and something tanks it's not so bad to just let it sit there for a year or so and not realize those unrealized losses.

    Anyways have fun.

    Comment


      #17
      Originally posted by antrux View Post
      Anytime Dan.

      For me I do something similar but instead of scanning a bunch of symbols looking for certain things I've written that in to the main buy/sell strategy and just run it on ~500 symbols everyday. I gathered my symbol list from filters on barchart.com's Stock Screener for volatility, avg price range, and etc. Then once per year I'll rescan and remake that active symbol list. I created my strategy to only invest X dollars per buy with a min account value left for any account fees and etc. This is also what I believe is the better way to diversify as A) the symbol list is spread across all industries and B) my investments aren't heavily focused on just a small handful of stocks spreading things out even more as in general at any given time 50% of stocks are going down and 50% are going up as people shift money from one place to another.

      This approach, using smaller investments across many stocks, allows you to better play with and be safe with crazy ideas such as buying at a 52 week low and selling at a 52 week high. From an optimists, mean reversionists, approach either the stock will return as that's always the goal of the CEO or the company goes out of business. If it goes bankrupt then no big deal because you only had say 200-500 dollars invested anyways and statistically the others that rebounded far outweighs the companies that went under. I'm not saying to do that but a highly diverse playing field allows for trying all the things the 'big guys' say to never do but could return huge profits.

      One last note on testing unless you've written something specific for a specific symbol is to always test your strategy across hundreds of stocks using the Strategy Analyzer. It is easy to write code that curve fits to a single stock but testing it across say 500 stocks and it still does well says that the code is robust and will mostly likely work well as general market changes slowly occur. Also testing across 500 symbols means your code has pretty much seen all that is possible to be seen with fast dips, gaps up/down, choppy terrain, and long slides up/down. Then just remember that it's a numbers game and that it's impossible to win them all but if you're only using smaller amounts per investment and something tanks it's not so bad to just let it sit there for a year or so and not realize those unrealized losses.

      Anyways have fun.
      antrux,

      This keeps gettin' better and better. Talk about "Great minds..."

      I use FinViz for pre-screening for symbols. My initial goal was to use pre-screen daily. After more thinkin', I decided that AutoHotKeys could make the process easier to automate the FinViz scan and export process. And it could help make using QCollector easier. So full automation may not be required in the short term.

      Testing against a large number of symbols is a very sound idea. And your ideas about minimizing risk are very interesting.

      One overall analysis strategy that I'm considering is using QCollector to download market data, upload it to a cloud solution (Google, Microsoft, Amazon), and build/test AI models to evaluate more profitable strategies. This is a bit in the future.

      Roughly, my next steps are pretty basic:
      1. Finish setting up NT and data feeds into a workable configuration
      2. Define and implement a smooth workflow, including all of the necessary parts
      3. Define and test workable short-term trading strategies
      4. Automate what can be easily automated now (basic stuff)
      5. Decide next steps


      It's probably not optimal, but I'm working on all of these somewhat simultaneously. IB is my broker and it's been set up for a while. NT subscription is in place and I'm learning the software. IQFeed subscription should be set up by next Tuesday (been using the trial version for the last week). Lots of moving parts that need to work together.

      I'll comment more this evening.

      Again, many thanks.

      Dan.

      P.s. I just installed AHK. Lookin' good so far.

      P.p.s. What is available for trading now is much better than what I used in the late '90's and early 2000's. Rethinking the overall process is an interesting challenge.

      Comment


        #18
        One overall analysis strategy that I'm considering is using QCollector to download market data, upload it to a cloud solution (Google, Microsoft, Amazon), and build/test AI models to evaluate more profitable strategies. This is a bit in the future.

        Never heard of QCollector however after scanning the site I don't really see anything that it does that NT doesn't already do. First off I use Kinetick data because my experience with NT support has showed me that because Kinetick is a sister company of NT they get the priority support and for NT to use Kinetick will pretty much always work. Others that I've used when something changes NT is slow to update code to use the providers new API or etc. Also as for historical data when I use NT to download it you just simply select an Instrument List that you've created say S&P 500, select the data span 1/1/2010 to 1/1/2017 and Last Price and in just a few minutes of my connection pegged at 10megs per second I've got it all and can immediately begin testing on it.

        When NT downloads data it's written to an ascii based file that you can easily scan and do anything with that you want so no need for something extra. QCollector allows for backtesting and etc and so does NT but I'd rather test from the same system that will be submitting orders as in doing so it also tests NT and has no need for any sort of rewriting of code to work with QCollector. If anything must be changed from the NT strategy code to work on QCollector then that is an opening for error in the future when a test on QCollector tested great but when run live on NT fails.

        Lastly as for QCollectors neural networks testing I've been planning on trying something similar in NT. Instead of a genuine neural network framework I however was just going to add a slew of values, SMAs, indicators, or etc for a strategy to process on each bar update storing each variable processed to an array. Then for the buy sell code write something that is essentially a brute force method of testing all the possible combinations together. NT has a really decent Optimization function in the Strategy Analyzer that you can set up so that it will process say a years worth of data on strategy A then when done will change one variable to the next value up and will test that year again and so on till finished. When finished it will show you the top 20 results of the possibly 10k tests it did and can show you a 3d chart of it all. So for example if you're using a SMA(Close, 20) somewhere you make that 20 a variable called X and set it up as a Strategy Parameter. Then the Optimizer can be setup to change the 20 to 21, 22, 23, etc up to Y and will test all possibilities with all other possibilities of other Strategy Parameters that you've defined. I've run the NT Optimization on a strategy where it churned through over 250k tests that took it two hours so I'd watch a movie and come back to review what worked best so I could ponder as to why it worked best.

        To me all this keeps the number of pieces that can break, slow you down, or have to master down to a min. As more time to develop something means less time to make money.
        Last edited by antrux; 07-29-2017, 12:03 PM.

        Comment


          #19
          Originally posted by antrux View Post
          One overall analysis strategy that I'm considering is using QCollector to download market data, upload it to a cloud solution (Google, Microsoft, Amazon), and build/test AI models to evaluate more profitable strategies. This is a bit in the future.

          Never heard of QCollector however after scanning the site I don't really see anything that it does that NT doesn't already do. First off I use Kinetick data because my experience with NT support has showed me that because Kinetick is a sister company of NT they get the priority support and for NT to use Kinetick will pretty much always work. Others that I've used when something changes NT is slow to update code to use the providers new API or etc. Also as for historical data when I use NT to download it you just simply select an Instrument List that you've created say S&P 500, select the data span 1/1/2010 to 1/1/2017 and Last Price and in just a few minutes of my connection pegged at 10megs per second I've got it all and can immediately begin testing on it.

          When NT downloads data it's written to an ascii based file that you can easily scan and do anything with that you want so no need for something extra. QCollector allows for backtesting and etc and so does NT but I'd rather test from the same system that will be submitting orders as in doing so it also tests NT and has no need for any sort of rewriting of code to work with QCollector. If anything must be changed from the NT strategy code to work on QCollector then that is an opening for error in the future when a test on QCollector tested great but when run live on NT fails.

          Lastly as for QCollectors neural networks testing I've been planning on trying something similar in NT. Instead of a genuine neural network framework I however was just going to add a slew of values, SMAs, indicators, or etc for a strategy to process on each bar update storing each variable processed to an array. Then for the buy sell code write something that is essentially a brute force method of testing all the possible combinations together. NT has a really decent Optimization function in the Strategy Analyzer that you can set up so that it will process say a years worth of data on strategy A then when done will change one variable to the next value up and will test that year again and so on till finished. When finished it will show you the top 20 results of the possibly 10k tests it did and can show you a 3d chart of it all. So for example if you're using a SMA(Close, 20) somewhere you make that 20 a variable called X and set it up as a Strategy Parameter. Then the Optimizer can be setup to change the 20 to 21, 22, 23, etc up to Y and will test all possibilities with all other possibilities of other Strategy Parameters that you've defined. I've run the NT Optimization on a strategy where it churned through over 250k tests that took it two hours so I'd watch a movie and come back to review what worked best so I could ponder as to why it worked best.

          To me all this keeps the number of pieces that can break, slow you down, or have to master down to a min. As more time to develop something means less time to make money.
          Antrax,

          I think there is a disconnect...

          First, as far as I know, QCollector is just a download tool that save IQFeed data to CSV text files. Unless I missed something, it doesn't do anything else. No neural network, no back testing, etc. It's just a data collector for IQFeed.

          The major reason that I'm interested in IQFeed and QCollector is a comment by NinjaTrader_ChristopherD in another thread:
          The Kinetick data feed is only available for the NinjaTrader platform and there is no API that would allow you to access the data outside of NinjaTrader. This is one of the main reasons that we are able to offer Kinetick at a discounted cost versus IQFeed: because we have tailored our support to a single platform interface. If you do require an API to access the data with other platforms, then IQFeed may be a better solution for these needs. But Kinetick has been optimized, from both a technology and cost standpoint, to be the best solution for traders who only need to access the data within the NinjaTrader platform.
          I took this to mean that the NT data is in a private repository to which I can't access. If this is not the case and the data is in a "database" file that I can access, then that's a world of difference. I don't care about an API, I just wanted the data for developing and testing new trading models. That said...

          Today, I realized that a better alternative for model development would be to purchase something like a one time dataset of 5 years of historical data for all NYSE and NASDAQ symbols. This should increase the probability of success and be less expensive than using IQFeed and QCollector, or Kinetick.

          The idea is to upload this historical data to an AI cloud vendor (Google, Microsoft, or Amazon), and apply some serious algorithms on it to build trading models that may not be obvious otherwise.

          For example, a strategy might be to look for price/volume divergences based on a linear regression of the closing price and a linear regression of the bar up/down volume for that price. This could indicate that the there is hidden buying or selling that is opposite going on. This could be identified by a divergence in the linear regression "slope". E.g. price going up while buy volume is going down. That's fairly straight forward.

          But what if you want to improve the model to determine if the pattern you're looking for:
          is correlated to only certain stocks, and/or
          it occurs at certain times of the year (e.g. earnings announcement), and/or
          it's specific to certain sectors and industries, and/or
          the strategy works with more or fewer bars, and/or
          the pattern only shows up when using a longer or shorter bar interval, and/or
          it's related to specific market events, and/or...

          There may be potentially millions (100's of millions?) of possible parameter combinations, so processing could take a LONG long time. That's why I'm thinking cloud service that pours serious CPU horsepower at the problem.

          My goal is to build a self learning model that will continue to modify itself to add new data and find the right mix of characteristics. Once this system has defined and tuned one or more models, the "production" model(s) will start scanning large number of instruments looking for these patterns when they pop up.

          If NT can handle even a portion of this kind of backtesting or once developed can run this model looking for instruments to trade, I'll be happy with it. And your points about Kinetick are persuasive. So...

          Couple Questions:
          Regarding NT and Kinetick, you mentioned that the data was stored in a database file. I'd like to take a look. Where is the file located?

          You mentioned that you used Kinectick for data and IB for a broker. Does ALL of your data come from Kinetick? Or does IB provide part of your data (like real-time)?

          Thanks and regards,

          Dan.

          Comment


            #20
            I guess the QCollectors homepage was referring to using the data to be able to do those other things not that it could do those things. I didn't read for very long before a squirrel caught my attention. <grin>

            Well as for the downloaded data I doubled checked and can't for the life of me figure out where I saw it at. In the current user's documents folder under NT is a db folder and I could have swore there were tens of thousands of simple txt files with data in them but now I'm seeing they are all .ntd or .ncd files that are not ascii.

            SOOoo... attached is a stupid simple strategy that will dump all that data to a simple csv file for you.
            Create an instrument list in NT of some or all symbols that you need data for.
            Use Kinetick or whomever to download the data using Tools, Historical Data, Load tab.
            Open Strategy Analyzer, select DataDump strategy, select the date range needed, select the instrument list from above, and click Run.

            A new file will be created for each symbol name and ".csv" at the end.
            Each line will contain: Bar Time,Open,High,Low,Close,Volume
            There's always more than one way to skin a cat. I wonder who used to skin cats though?

            As for IB or Kinetick for data I've stuck with only Kinetick. Not sure how others have only used IB data with NT but even in trying to download data through NT from IB for say the last year of the S&P 500 list either the download would never finish for me or only a couple dozen symbols would ever get downloaded per attempt. IB data is cursed with pacing violation issues if you download too many symbols or too much data all at once that I just said to hell with it and it's $35 per 100 symbol fee that still has to be paced if requesting too much data at one time.

            So all of my data historical and live is from Kinetick and I have to say I've simply never had any issue with it.
            Attached Files
            Last edited by antrux; 07-29-2017, 10:48 PM.

            Comment


              #21
              Originally posted by antrux View Post
              I guess the QCollectors homepage was referring to using the data to be able to do those other things not that it could do those things. I didn't read for very long before a squirrel caught my attention. <grin>

              Well as for the downloaded data I doubled checked and can't for the life of me figure out where I saw it at. In the current user's documents folder under NT is a db folder and I could have swore there were tens of thousands of simple txt files with data in them but now I'm seeing they are all .ntd or .ncd files that are not ascii.

              SOOoo... attached is a stupid simple strategy that will dump all that data to a simple csv file for you.
              Create an instrument list in NT of some or all symbols that you need data for.
              Use Kinetick or whomever to download the data using Tools, Historical Data, Load tab.
              Open Strategy Analyzer, select DataDump strategy, select the date range needed, select the instrument list from above, and click Run.

              A new file will be created for each symbol name and ".csv" at the end.
              Each line will contain: Bar Time,Open,High,Low,Close,Volume
              There's always more than one way to skin a cat. I wonder who used to skin cats though?

              As for IB or Kinetick for data I've stuck with only Kinetick. Not sure how others have only used IB data with NT but even in trying to download data through NT from IB for say the last year of the S&P 500 list either the download would never finish for me or only a couple dozen symbols would ever get downloaded per attempt. IB data is cursed with pacing violation issues if you download too many symbols or too much data all at once that I just said to hell with it and it's $35 per 100 symbol fee that still has to be paced if requesting too much data at one time.

              So all of my data historical and live is from Kinetick and I have to say I've simply never had any issue with it.
              Antrux,

              Many thanks for the code. I'm running out now, but I'll try it out this afternoon.

              And thanks for the feedback about IB. My experience with IB is similar to yours, but I wondered if it was just me as an NT newbie? And how other users had their data feeds set up?

              Finally, I posted a question here: http://ww.ninjatrader.com/support/fo...d.php?t=102682. It's about whether NT is free as stated on the Kinetick web page? I paid $180/quarter. If I switched to Kinetick, would NT be free?

              Thanks,

              Dan.

              Comment


                #22
                I've seen that too before but it's more of a marketing ploy.
                NT is free to do anything with that you please except actually make it send buy/sell signals to a broker.

                As for the NT leasing I'm not sure why anyone would ever want to lease it when:
                (1,000 Lifetime License / 180 Quarterly License = 5.5 quarters)
                IMHO, before anyone buys NT they should have already written their strategies, or intend to trade manually with it, and tested them well on free historical data proving that the investment in NT is worth it. If it is then this world of trading is generally not something one gets in to for the short term only and before you know it 5 quarters will be gone and you'll have spent more than just buying it outright.

                Kinetick is ~55 per month for 100 symbols to stream & ~100 a month for 500 symbols.
                They do give free data but if your strategy runs on tick or min data I don't think the free Kinetick EOD data will work. In my mind just assume you're going to loose ~600 a year for data the first year and use NT free until you tested all that you need. Then if you've got a good strategy buy NT lifetime and just continue the Kinetick subscription. Also, even with a Kinetick 100 symbol subscription you can use that to download however much that you want of historical data and it remains on the PC for future use or use the datadump and permanently store it.

                Comment


                  #23
                  Originally posted by antrux View Post
                  I've seen that too before but it's more of a marketing ploy.
                  NT is free to do anything with that you please except actually make it send buy/sell signals to a broker.

                  As for the NT leasing I'm not sure why anyone would ever want to lease it when:
                  (1,000 Lifetime License / 180 Quarterly License = 5.5 quarters)
                  IMHO, before anyone buys NT they should have already written their strategies, or intend to trade manually with it, and tested them well on free historical data proving that the investment in NT is worth it. If it is then this world of trading is generally not something one gets in to for the short term only and before you know it 5 quarters will be gone and you'll have spent more than just buying it outright.

                  Kinetick is ~55 per month for 100 symbols to stream & ~100 a month for 500 symbols.
                  They do give free data but if your strategy runs on tick or min data I don't think the free Kinetick EOD data will work. In my mind just assume you're going to loose ~600 a year for data the first year and use NT free until you tested all that you need. Then if you've got a good strategy buy NT lifetime and just continue the Kinetick subscription. Also, even with a Kinetick 100 symbol subscription you can use that to download however much that you want of historical data and it remains on the PC for future use or use the datadump and permanently store it.
                  Antrux,

                  Hi. I just spent 30 minutes writing a nice response and then Safari choked. Lost all of it. So this will be shorter.

                  I tested your script. Worked great after I got around some newbie mistakes.

                  The first quarter lease of NT is only to get me past the next three months. If NT works for me, I'll buy it. If not, well... Gotta keep looking. Which may be tough, because I've already eval'd most of the options.

                  My basic strategy requires intraday and daily data. My first year costs will probably exceed $2,000. Cost of doin' business I suppose. I could shave some costs, but that would probably increase frustration and dev/test time.

                  One option is to buy a one-time intraday and daily data download from EODdata.com. I'm still not quite sure how to bulk load the data into NT, but there should be a way especially with AHK.

                  Again, many thanks for all your help.

                  Regards,

                  Dan.

                  Comment


                    #24
                    Thank you for your posts antrux!

                    Dan, historical data can be imported into NinjaTrader from the Control Center's Tools > Historical Data menu. You can select multiple items to import them all at once.

                    The data has to be in a certain format however. If the data is not in the same format, you can request to have the data formatted for you or you can use another program like Excel to get the data into a format you can import .

                    Format for historical data files is documented here: https://ninjatrader.com/support/help...AndDataFormats
                    JimNinjaTrader Customer Service

                    Comment


                      #25
                      Originally posted by NinjaTrader_Jim View Post
                      Thank you for your posts antrux!

                      Dan, historical data can be imported into NinjaTrader from the Control Center's Tools > Historical Data menu. You can select multiple items to import them all at once.

                      The data has to be in a certain format however. If the data is not in the same format, you can request to have the data formatted for you or you can use another program like Excel to get the data into a format you can import .

                      Format for historical data files is documented here: https://ninjatrader.com/support/help...AndDataFormats
                      Jim,

                      Thanks for the excellent feedback. This will help me tremendously going forward.

                      For some of my work, large amounts of historical data may be required. If I can get that directly into NT and use NT for analysis and back testing, that would be great. If not...

                      I may need to get data from other sources to upload to cloud process and/or import it into NT. One possibility is using QCollector to bulk download the data from IQFeed. Another possibility is to get it from a source like EOD Data.

                      With your feedback, I can evaluate these sources to see if they fit the NT format. Even if they don't fit, knowing the specific target format makes it fairly straight-forward to write a C# app to reformat the data into NT format.

                      Thanks and regards,

                      Dan.

                      Comment


                        #26
                        Originally posted by dan_public View Post
                        snip..

                        ... Worked great after I got around some newbie mistakes.

                        snip...
                        Antrux,

                        Just to be VERY clear, the sentence above should have read:

                        Worked great after I got around some of MY newbie mistakes.
                        Regards,

                        Dan.

                        Comment

                        Latest Posts

                        Collapse

                        Topics Statistics Last Post
                        Started by futtrader, 04-21-2024, 01:50 AM
                        4 responses
                        41 views
                        0 likes
                        Last Post futtrader  
                        Started by Option Whisperer, Today, 09:55 AM
                        1 response
                        11 views
                        0 likes
                        Last Post bltdavid  
                        Started by port119, Today, 02:43 PM
                        0 responses
                        1 view
                        0 likes
                        Last Post port119
                        by port119
                         
                        Started by Philippe56140, Today, 02:35 PM
                        0 responses
                        3 views
                        0 likes
                        Last Post Philippe56140  
                        Started by 00nevest, Today, 02:27 PM
                        0 responses
                        2 views
                        0 likes
                        Last Post 00nevest  
                        Working...
                        X