Override Historical Data Loading

[RaddiFX]

I was working on this about a year or so ago. Back then I was asking if it was already done, in the works feature request, or a known project someone had already worked on and the answer seemed to be a no or i don't know to all of those, but I'm back to ask again.

Whatever function is involved here decides when connected to a data feed to request an update to them. I've tried in the past to override from the point I can see which is here:
2021-10-13 03:59:17:431 Cbi.Instrument.RequestBars (to Provider): instrument='PLUG' from='10/12/2021 12:00:00 AM' to='10/12/2021 11:59:59 PM' period='1 Tick'

The problem:
When Ninjatrader loads historical data, if you're connected to a data feed with that data available it will ALWAYS download the requested time frame. The issue is that when I load a stock it can sometimes take up to 3 seconds per day and if i'm trading a news event I've missed it before the chart even loads.

The solution:
I'd like to override the call being made so that it checks the historical database before reaching out to download. I know that this would be faster because I can load a historical chart with 15 days of data in under 2 seconds, but it takes much longer on kinetick.

Does anyone know if this has been fixed/implemented in the last year, someone has completed a project with this utility, or the method I need to override and/or how I would do so to accomplish this?

[NinjaTrader_Emily]

Hello RaddiFX,

Thank you for your post.

This has not been implemented in the last year, and if anyone has completed a project with this utility I am not aware of it and it would be undocumented/unsupported. I can gladly submit a feature request on your behalf (or check if there is an existing request then add your vote to it) if you could describe more concisely what you are looking for in the request.

The following page in the help guide explains the criteria for when NinjaTrader downloads historical data:

https://ninjatrader.com/support/helpGuides/nt8/loading_historical_data.htm

If you were to load any data series with the current day as the end date, this would trigger a request to load data for the current and prior day. If you need to get into the market quickly, such as trading a news event, you could consider using a real-time-only order entry window such as Basic Entry or SuperDOM. You would not have the benefit of having historical data available to view like you would on a chart, but you would get incoming real-time data right after opening the window rather than waiting for a historical data request to be loaded first.

You mentioned, "I know that this would be faster because I can load a historical chart with 15 days of data in under 2 seconds, but it takes much longer on kinetick." What type of data is being loaded? (tick, minute, day) Is Kinetick your only data provider and what do you mean that it "takes much longer" on Kinetick? Is this compared to other data providers or loading historical data for previous days not including today's date?

I look forward to your reply.

[RaddiFX]

I can gladly submit a feature request on your behalf (or check if there is an existing request then add your vote to it) if you could describe more concisely what you are looking for in the request.

I'd like that, Thanks! I don't see anything on the list currently. I'd also be fine with programming this on my own if you could point me in the right direction of what function to override.

real-time-only order entry window such as Basic Entry or SuperDOM. You would not have the benefit of having historical data available to view like you would on a chart, but you would get incoming real-time data right after opening the window rather than waiting for a historical data request to be loaded first.

I do have a strategy that i'm implementing that I've already restricted greatly. I have a high probability if I have 5 days loaded, but I've dropped that down to 1 just due to this loading issue.

You mentioned, "I know that this would be faster because I can load a historical chart with 15 days of data in under 2 seconds, but it takes much longer on kinetick." What type of data is being loaded? (tick, minute, day) Is Kinetick your only data provider

I'm only using Kinetick for stock data. I'm loading Tick data. I have an indicator that requires the use of tick level data to be brutally accurate. When I first started on this endeavor the issue of downloading multiple days worth of data in both minute and tick forms was a major issue, so I made a barstype that uses tick data to generate minute data so that I only need to download tick data. I then assign the minute based indicators to that barstype inside the strategy and focus solely on the tick level.

and what do you mean that it "takes much longer" on Kinetick? Is this compared to other data providers or loading historical data for previous days not including today's date?

I've only tested 2 other data providers and they're roughly as fast. I'm using Kinetick as an example because that's the "in-house" data provider and the one I use. Yes, this is comparing date ranges outside of "live timeframes". So any historical data I have loads much quicker than the request to download.

Operations > Historical Data > Loading Historical Data
https://ninjatrader.com/support/helpGuides/nt8/loading_historical_data.htm

This has given me an idea though. If this is correct, then if I have (up to) yesterdays data loaded then it won't download that data, I can make a script to download the data for all of the tickers I'm interested in at the end of each day to cut out all of those previous days.

[NinjaTrader_Emily]

I'd also be fine with programming this on my own if you could point me in the right direction of what function to override.

We do not have any documented methods or properties to offer support on how to do this. You could consider going to Control Center > Tools > Options > Market Data and unchecking the "Get data from server" option. As described in the help guide, "Enables or disables the retrieving of historical data from the data provider's server while connected. When disabled, historical data will still download when connecting, manually reloading historical data, or manually downloading historical data."

https://ninjatrader.com/support/helpGuides/nt8/options_marketdata.htm

In order to submit this as a feature request, please describe what kind of feature you would like added to the platform. Would you like a different type of option compared to the one above that allows you to stop historical data from being loaded automatically? Are you looking specifically for something you can do in NinjaScript, and if so what are you looking for and how would it work in the proposed feature?

I'm only using Kinetick for stock data. I'm loading Tick data. I have an indicator that requires the use of tick level data to be brutally accurate. When I first started on this endeavor the issue of downloading multiple days worth of data in both minute and tick forms was a major issue, so I made a barstype that uses tick data to generate minute data so that I only need to download tick data. I then assign the minute based indicators to that barstype inside the strategy and focus solely on the tick level.

I asked about Kinetick because there are limitations on tick data; when requesting tick data between the hours of 9:30am ET to 4:30pm ET you will have access to 8 days of historical tick data. If you request the data outside those hours the full 180 days is available. After you request the data, it will be downloaded to your historical database and cache, and you will be able to access this data at any time. Reducing the number of days to load on your charts will prevent you from hitting the 8 day limit during peak hours.

This has given me an idea though. If this is correct, then if I have (up to) yesterdays data loaded then it won't download that data, I can make a script to download the data for all of the tickers I'm interested in at the end of each day to cut out all of those previous days.

If you have up to yesterday selected for the data series, the chart would only display historical data and not any real-time data. You would still see the Ask and Bid prices moving in the Chart Trader and have the ability to place orders, but you would not have live data coming into your chart to view the market movement and how it relates to your positions.

The help guide page mentions the following:
"There are two ways to ensure that the memory contains data for your instrument of interest:
1.Load the instrument into a Market Analyzer window along with an indicator column with the same chart parameters you plan to be loading.
2.Open and maintain a chart with the same data type and days to load that you plan on loading."

What is your current process? Are you opening and connecting to NinjaTrader more or less at the same time you would like to begin trading? This whole scenario could likely be prevented if you open NinjaTrader earlier in the day and leave your charts/Market Analyzer window open to update historical data and continue to load real-time data.

I look forward to assisting you further.

[RaddiFX]

I asked about Kinetick because there are limitations on tick data; when requesting tick data between the hours of 9:30am ET to 4:30pm ET you will have access to 8 days of historical tick data. If you request the data outside those hours the full 180 days is available. After you request the data, it will be downloaded to your historical database and cache, and you will be able to access this data at any time. Reducing the number of days to load on your charts will prevent you from hitting the 8 day limit during peak hours.

I'm only requesting 5 days worth of data, that is enough for my strategy to have fantastic results, but when running live I only use 2 days because i need at least that much.

"There are two ways to ensure that the memory contains data for your instrument of interest:
1.Load the instrument into a Market Analyzer window along with an indicator column with the same chart parameters you plan to be loading.
2.Open and maintain a chart with the same data type and days to load that you plan on loading."

My problem with these two options is that when I have a ticker in the analyzer it will consume a feed slot, which I only get 200 and I may be monitoring ~180 at a time. Unfortunately this one is undoable.

In order to submit this as a feature request, please describe what kind of feature you would like added to the platform. Would you like a different type of option compared to the one above that allows you to stop historical data from being loaded automatically? Are you looking specifically for something you can do in NinjaScript, and if so what are you looking for and how would it work in the proposed feature?

I'm going to see if my idea for the other method works in at least eliminating one day from the download requests.

What is your current process? Are you opening and connecting to NinjaTrader more or less at the same time you would like to begin trading? This whole scenario could likely be prevented if you open NinjaTrader earlier in the day and leave your charts/Market Analyzer window open to update historical data and continue to load real-time data.

I wrote an addon to automatically accept input from a python script I wrote to scan for news and automatically open a strategy of my choosing for that ticker. So all in one shot, it opens a strategy, activates it, downloads the data, consumes a slot on kinetick, then if it doesn't find a trading setup/opportunity within a given time-frame it disables the strategy, and hands back the kinetick slot.

Due to how all of that works, I can't be watching tickers ahead of time, they come to me at "random", up to 50/minute during peak news times(every 30 minutes). And that's where the issue arises, due to the issue of adding tickers as they flow in and the (up to) 3 seconds of download time per day per ticker(I looked at it after your previous comments, it's actually up to 1.5s/day but it always downloads at least 2 days), and it always downloading at least 2 days worth, means that I may miss a trade by up to 3 minutes.

[RaddiFX]

So I was able to program what I was looking for to download the historical data of EVERY ticker I'm looking for using a python script and I programmed an addon for NT to import the list. I need to wait until Friday to test out how many days it tries to load depending on what's in the database.

However, because of how I made this it clutters up the strategies window. I'd like to ask if there is a way to REMOVE a strategy from the strategies grid when it disables. I'm currently using

strategy.CloseStrategy();

to disable the strategies. To add them I'm currently am using

System.Reflection.MethodInfo dynMethod = sg.GetType().GetMethod("StrategyAdd",
System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Static);
if (dynMethod!=null)
    dynMethod.Invoke(sg, new object[] { s });        
                                            
dynMethod = sg.GetType().GetMethod("StrategyEnable",
System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Static);
dynMethod.Invoke(sg, new object[] { s, null, null });        ​

Where s = the strategy object and sg = the strategiesgrid object.

I've tried a few things but none of them seem to be working.
This is documented, but doesn't seem to be working.

account.Strategies.Remove(strategy);

And I was able to find the Method signature for "StrategyRemove" is Void StrategyRemove(StrategyBase strategy), but when I try to invoke it like StrategyEnable it does literally nothing.

Realistically I probably don't need to do this, but I'd like to clean up the appearance without having to do it manually.

[NinjaTrader_Emily]

Hello RaddiFX,

Thank you for the update.

I am glad to hear you were able to achieve your desired result with the Python script.

This is documented, but doesn't seem to be working.

account.Strategies.Remove(strategy);

Where is this documented? I am looking for it in the help guide and I am not able to find Strategies.Remove anywhere. I am not aware of any existing NinjaTrader documentation for a method to remove strategies from the strategy grid programmatically, meaning that what you are asking is undocumented and unsupported by our team. Further, I found the following thread where you discussed a similar topic with one of my colleagues:

https://forum.ninjatrader.com/forum/ninjatrader-8/strategy-development/1172867-delete-strategy-from-strategies-tab#post1172867

Thank you for your time and patience. Please don't hesitate to reach out with any additional NinjaTrader items we may assist you with.