How to call slow database query 1 time only for optimization of 1000's of contracts

[ChiTrader2000]

Hello,

I have a Strategy that needs data from a database that requires a slow database query. It does the query then puts the results from the query into a simple C# generic Dictionary.

I am trying to make my optimizations of 1000's of contracts at once much faster by only calling this slow database query 1 time for the entire optimization, or at least only 1 time per thread, not 1 time per contract, as I have been doing in the Strategy's "Initialize()" event.

I have geen trying to do this by doing the database statement in the static constructor of my strategy, but when I do this only about 20% of the trades occur, as though 1 thread is getting the data and the other threads are not.

What is the best way in code to do a slow database query one time per optimization (or one time per thread), instead of one time per contract?

Thank you very much in advance!

ChiTrader2000

[Jeremytang]

Just a suggestion, but why don't store a value in the DB that is the last time of update? Just check the value in the initialize statement and see if it's been updated today and don't run it if it has been. And then simply update that date value at the end of the query.

Jeremy

[NinjaTrader_RyanM1]

Thank you for the suggestion here, JeremyTang. There is a separate strategy instance created for each optimization test, so this seems like a good way of ensuring the db is loaded only once.

You would want to do this check in OnStartUp() method rather than Initialize()

[ChiTrader2000]

Hi Trader_RyanM,

Thanks for the tip about the new NT7 event called "OnStartUp()".

Unfortunately, I did a single backtest on 590 contracts and discovered the following:

1. OnStartUp() was called 458 times
2. My Strategy's Constructor was called 591 times
3. Initialize() was called 591 times

Since my Strategy's Constructor was called 591 times, it appears to mean that a new instance of my strategy is called for every single contract.

To me, this means that the only way to call a query only once (or not lots of times, at least) in a backtest or optimization of a strategy over 1000's of contracts is to use static variables, perhaps. I have been trying this static variable concept for two days now without success.

Can you or anybody else out there think of any other way than the use of static variables to achieve this result?

Is there a method that is called one time when the "Run Optimization" button is pressed?

If not, is there at least a method that is called once per backtest run in an optimization (i.e., if I do a grid optimization over 2 parameters, each with 20 possible values, this is 400 backtest runs... it wouldn't be so bad to call a method only 400 times, vs. 400 backtest runs * 5,000 contracts = 2,000,000 times, which is very bad for a slow query)?

Thank you very much in advance!

ChiTrader2000

[NinjaTrader_Brett]

Hello,


Let me ask the following question.

What is the purpose of the database.

Database connection in NinjaTrader in the realm of unsupported territory from the NinjaScript support team.

However you say that it only needs to be loaded once.

What does this database store values for, is its purpose mainly for running live or for backtests as well.

Does the database values change per optimize started manually before.

Essentially heres what I'm getting at I'm not thinking of a way to keep these variables/ code the will only run ones during the optimize. What I am thinking is that why dont you add these variables manually from the database if it only needs to query once and add them directly into the code instead of even messing with the database then if you need the database when you run live reconnect the database.

Let me know if I can be of further assistance.

[ChiTrader2000]

Hello NinjaTrader_Brett,

Great idea about hard-coding the data in code. I did this and it worked.

I guess I just needed to think outside the box.

Although this is probably too much to ask, my two cents is that it would be awesome if somewhere down the line NinjaTrader was more optimized for fast optimizations of very large baskets of instruments. For instance, perhaps the Strategy class could be instantiated exactly one time for the entire optimization for all contracts, and it could be the user's responsibility to know what variables need to be changed/reset inside of the Initialize() event, which is called once for each contract for each backtest run. If NT version 10.0 worked this way, it might be possible to greatly improve the speed of these kinds of massive optimizations for complex user strategies.

Thanks again for your help and excellent suggestion.

ChiTrader2000

[NinjaTrader_Brett]

Hello,

Glad this worked out.

Not a problem, I will forward your suggestion to development.

Let me know if I can be of further assistance.