Having to delete historical data from cache

[tarikmedjber]

Hi,

I made a thread https://ninjatrader.com/support/foru...e-of-an-object
a while ago where the outcome of it was to follow these steps:

Removing Historical Data

• Once the data has been deleted close NinjaTrader. Open the Documents\NinjaTrader 8\db\cache folder. Select all files then right mouse click and select “delete.”
• Open NinjaTrader and connect to your data provider. Open a new chart and test.

It did the job.

However this is becoming a major trend recently where I am having to do this or else the orderflow indicators I use are all broken in various ways until I do the above steps.

This thread I am creating here is more of a FYI in case other people are seeing issues.

I am also experiencing very slow processing of my workspace when starting up where it stays on "Calculating" for a very long time. Without even changing any of my indicators I usually have attached. Just FYI.

Thanks,

Tarik

[NinjaTrader_Jason]

Hello Tarik,

You can experience slow and unresponsive behavior if you have many workspaces, charts, indicators and other windows opened depending on market volatility as well. If market volatility increases, the same setup will need more PC resources.

The 'Calculate' setting for indicators and strategies can be important. If set to 'On each tick' or 'On price change', the indicator/strategy will be calculated on each incoming tick or each price change. During high volatility this can be very CPU intensive especially if the indicator/strategy makes complicated calculations. If you set it to 'On bar close', the indicator/strategy will only be calculated at the end of each price bar.

Could it be you have workspaces opened in the background? Please go to Control Center-->Workspaces. If you see a green icon in front of the workspace, it indicates the workspace is opened and displayed on your monitor. If you see a grey icon, the workspace is opened in the background. Meaning you don't see its windows, but it will use PC resources. You can only have 1 workspace display on your monitor at a time. Please close the workspaces running in the background and check if the slow behavior persists.

You can find more tips and suggestions to improve performance at the link below.

http://ninjatrader.com/support/helpGuides/nt8/en-us/performance_tips2.htm

[tarikmedjber]

Thanks for the reply Jason.

Unfortunately this is not the case for me. I monitor my CPU usage frequently and it is always on low and remains under 15%. I only have one workspace that I use all of the time/ in my workspace "folder". It seems to be very intermittent when I suffer these problems. A hard reset of historical data seems to work sometimes for the orderflow issue or sometimes I need to delete charts and create the charts again to fix the calculating and/ or freezes problems.

It is a strange one but I may have to live with it for now and monitor what may be causing the issues.

Thanks,

Tarik

[NinjaTrader_Jason]

Could it be a single logical processor reaches the maximum when you experience the issue? Please right click in the CPU graph in the Windows Task Manager-->Performance-tab and select Change graph to-->Logical processors.

Do you perhaps receive delayed realtime data when the issue occurs? Please disconnect your broker/data feed connection in NinjaTrader when the issue happens again and check if realtime data in the chart immediately stops updating. If so, the issue is delayed realtime data. If realtime data continues to print on the chart (as if it were still connected), the issue is caused by delayed rendering/performance issues.

[tarikmedjber]

Ahh it does seam that some CPUs are working extensively. Such as the last 2. But overall 8/9% utilisation?.

I will check the second part later when I get back on the PC.

Thanks for your help

Tarik

[NinjaTrader_Jason]

This would indicate performance issues. I suggest to apply tips and suggestions from the link below until performance improves.
https://ninjatrader.com/support/help...ance_tips2.htm

If you have Resolution set to Tick for the Order Flow Volume Profile indicator, I suggest to set it to Minute instead and check if this will make a difference.
https://ninjatrader.com/support/help...VWAPParameters

The Order Flow VWAP also has a Resolution setting. Setting it to Standard instead of Tick will improve performance as well.
https://ninjatrader.com/support/help...VWAPParameters

[tarikmedjber]

This is great info thank you.

Is there any tips for improving performance for Strategies and when running backtests?

Thanks,

Tarik

[NinjaTrader_Jason]

Unfortunately there is no link with performance tips specific for backtests/optimizations. You could use less historical data in backtests or less strategy optimization parameters for optimizations. The link below provides information on how optimization performance is affected.
https://ninjatrader.com/support/help...nderstandingFactorsThatAffectOptimizationPerformance

[tarikmedjber]

No problems thank you Jason for your help. It was more just attaching a strategy to the chart (as then it runs through the historical bars the chart has open but takes a while sometimes, however your previous tips may help) I meant to be honest (I didn't word it correctly).

Thanks very much for your help

[ntbone]

I look at a lot of instruments in a given day as I do quite a bit of scanning. I have found similar performance issues and one of the ways of clearing up some of those issues is by wiping out the Cache folder.

The performance issues are more noticeable during market hours but I have also noticed them after hours. I will load my workspace and it will be very very slow, with charts redrawing themselves numerous times before I can interact with the charts. I closed down NinjaTrader, deleted the cache folder and relaunched and suddenly everything was much more responsive. The workspace loads faster and changing charts is faster.

I do look at numerous time frames for each instrument I scan and, from inspection, the cache folder will hold generated data for various time frames inside of it.

My cache folder ends up with so many files, that I use the command prompt to delete as it takes Windows way too long to count up all the files and remove them using Explorer.

Does Ninja Trader ever delete any of the files generated in the cache folder or does the folder just continue to grow over time?

[NinjaTrader_BrandonH]

Hello ntbone,

Thanks for your notes,

NinjaTrader does not do anything to delete files from the Documents\NinjaTrader 8\db\cache folder so the data in the folder does increase over time.

The db/cache folder is where NinjaTrader stores already-processed bars for quick retrieval by the platform.

The cache is used to speed up the platform and should not be deleted as a maintenance step.

Are you able to replicate the behavior you are reporting in a Clean Environment with no custom components used, such as custom Indicators, Bars Types, etc?

Creating a clean environment can be done by following the steps below:

1. Close NinjaTrader 8, and rename the "NinjaTrader 8" folder in My Documents to something like: "NinjaTrader 8 OLD"
2. Uninstall NinjaTrader from the Windows Control Panel and reinstall using the installer from account.ninjatrader.com
3. Add the required indicators and test the template.

If you ever need to switch back to your original platform, you may do so by closing NinjaTrader and swapping the platform folder names.

For example, Close NinjaTrader and rename the new "NinjaTrader 8" folder to "NinjaTrader 8 NEW" and the "NinjaTrader 8 OLD" folder to "NinjaTrader 8." Then restart the platform. Simply put: NinjaTrader 8 will always load the "NinjaTrader 8" folder in My Documents.

[ntbone]

I will investigate and see if I can reproduce it. I cannot reproduce it with a completely clean environment as the cache, minute, tick and day folders from DB will need to be included. I cannot easily generate a large cache folder easily without using my own code as visiting that many instruments would be difficult.

I have noticed performance problems in general with the method for which NinjaTrader stores market data. Performance degrades in a folder that has a large # of files. Its inefficient for storage of the data as well. My cache folder already contains over 60,000 files, and that is after cleaning it yesterday after market close. Since this folder never gets purged, over time it is going to accumulate a very large # of files.

Does NinjaTrader ever enumerate all the files in these folders? How does it verify whether a cache file exists? If it enumerates all the files in a folder for an instrument, this operation can get expensive as the # of files increases.

[NinjaTrader_BrandonH]

Hello ntbone,

Thanks for your note.

To investigate this matter further we would need to be able to replicate the behavior in a clean environment with no custom components used.

The cache should not be deleted as a maintenance step as it is used to store already-processed bars for quick retrieval by the platform.

The way bar caching works is:

1. NinjaTrader starts a realtime subscription to market data with the provider and starts caching realtime ticks in memory
2. NinjaTrader checks if you already have the necessary bars in the cache to fulfill the bars request
   1. If so, go to step 5
3. If not, NinjaTrader checks if you have the necessary data in the repository to fulfill the bars request
   1. If so, go to step 4.1.2
4. If not, NinjaTrader sends out a bars request to the provider/HDS
   1. When this request is returned from the provider/HDS:
      1. The data is saved to the repository
      2. The data is processed through the selected BarsType code to build bars
      3. Then the bars are saved into the cache except in the following circumstances:
         1. Tick Replay
         2. Volumetric Bars Type
         3. BarsBack request
5. When the bars request has been fulfilled, NinjaTrader needs to merge the historical bars with the realtime subscription
   1. By starting the subscription before the bars request in step 1, we try to ensure there is some overlap of data so we can align the two data sources

If the issue involves the use of third-party components, Is the developer of those third-party components informed about how to reproduce the issue from a clean environment? You could consider reaching out to the developer of the third-party components directly to show the developer how the issue manifests/how cache is broken. Or in other words: you could let them know what steps you take where an issue is seen where clearing cache helps so they may investigate it.

[ntbone]

Thanks for the outline.

Step 2 in your process is where things could be affected by having a large # of cached files in a folder. It depends on how the code inspects the folder to determine if the cache data exists or not. If it checks every file, that would definitely slow things down as it grows. If it queries for the existence of a file based on a generated file name, I would hope this wouldn't be impacted by # of files in the folder as it should be a quick look up to see if file XYZ exists.

All 3rd party components in use have been developed by me.

What I have observed is that charts load noticeably faster after a cache that is more than a week old is wiped out then when it hasn't been It can be painfully slow at times to switch instruments or even load the workspace. Once the cache is wiped, all things are noticeably quicker.

I'll wait until I have another week or more worth of cache data collected and then perform some performance tests on the market data, without any custom NinjaTrader code using a clean installation, except the market data.