Unable to Retrieve Historical Data from Backup DB Folder in NinjaTrader

[MSerag]

Hello,

To speed up the performance of NinjaTrader, I delete the DB folder before each trading day and back it up in another location. After the trading day, I return the backup folder to its original location for analysis. I expect that all the historical data stored in the db folder can be retrieved without access to Kinetick connection as the data are already downloaded and stored in the DB folder. However, I have noticed that this is not the case.

When I select the "Days to load" and "End date" in the Data Series window, the data cannot be retrieved even after closing and reopening of NinjaTrader. Instead, I have to connect to Kinetick again to retrieve the historical data, although it is present in the db folder.

Can you please provide me with any clue or suggestion on how to make this work? I appreciate any assistance you can provide in resolving this issue.

Thank you for your help.

[bltdavid]

Which version of NinjaTrader are you running?

When you say delete the DB folder, are you
meaning all contents, like, the entire folder?

There is a very important standalone file in there,
and that is NinjaTrader's main database file. Are
you deleting that, too?

That's technically ok, since NinjaTrader sees the
missing database file and knows to rebuild a new
pristine empty database in it's place.

I'm just curious, why the heavy hand? I mean, there
are only 3 subfolders containing historical data, which
are the 'db\tick', 'db\minute', and 'db\day' folders, so
deleting the entire db folder, well that seems way
overkill when moving around/managing historical data.

If you delete the entire db folder (including the database
file), aren't you losing your historical order history for
that day's trades? That's ok with you?

Just my 2˘.

[bltdavid]

To speed up the performance of NinjaTrader, I delete the DB folder before each trading day and back it up in another location.

Somehow, what you're doing just shouldn't be necessary ...

[QuantKey_Bruce]

Yes, that does seem extreme. I do, however, hear of people who have to clear db -> cache frequently. I suspect this has to do with a bug in how custom bar types are handled or a bug in their custom bar type code that leads to bad cached bar data getting stored but I'm not really sure. I just know that when they clear out the db -> cache folder it gets better.

[QuantKey_Bruce]

See also: https://forum.ninjatrader.com/forum/...cache-clearing

[NinjaTrader_Emily]

Hello MSerag,

Thank you for your post.

Were you advised to perform these steps by a NinjaTrader support representative? I understand you are trying to speed up the performance of the platform, though I agree with bltdavid that what you are doing likely isn't necessary. I suggest reviewing the performance tips here:

https://ninjatrader.com/support/helpGuides/nt8/performance_tips2.htm

When it comes to manually moving/deleting/modifying any files and folders within the NinjaTrader 8 folder on your PC, this should usually only be done when directed to do so by a support representative in order to prevent unintended consequences. You may simply be falling into one of the scenarios when NinjaTrader will download historical data from your data provider:

https://ninjatrader.com/support/helpGuides/nt8/index.html?loading_historical_data.htm#WhendoesNinjaTraderdownloadhistoricaldata

NinjaTrader has an interface for backing up and deleting/editing historical data files and you could try that instead of manually modifying the files and folders. You could create a backup at Control Center > Tools > Export > Backup File. Including Historical Chart Data will add the historical data to the backup file. It is also a tip in the help guide, "Market data files such as Historical chart data can often times be re-downloaded from your data provider. Market Replay data can be downloaded from the NinjaTrader servers for the most popular Futures and Forex instruments. If the data files you have stored on your computer are available from your data provider, you can save time and storage space by excluding these items from your backup and simply re-downloaded the data when needed. Please make sure to check with your data provider to ensure they still carry the type of data for the time period you may require."

Then, you may delete the historical data files after the backup is created from the Control Center > Tools > Historical Data window:

https://ninjatrader.com/support/helpGuides/nt8/index.html?editing.htm#HowToDeleteHistoricalData

After the trading day, you may then import the backup you created at Control Center > Tools > Import > Backup file. For more details and steps related to the Backup and Restore utilities, please see the help guide section here:

https://ninjatrader.com/support/helpGuides/nt8/backup__restore.htm

Otherwise, please explain the performance impact you are experiencing and we may be able to identify a better resolution to help you optimize NinjaTrader's performance.

Please let us know if we may be of further assistance.

[MSerag]

Hello MSerag,

Thank you for your post.

Were you advised to perform these steps by a NinjaTrader support representative? I understand you are trying to speed up the performance of the platform, though I agree with bltdavid that what you are doing likely isn't necessary. I suggest reviewing the performance tips here:

https://ninjatrader.com/support/helpGuides/nt8/performance_tips2.htm

When it comes to manually moving/deleting/modifying any files and folders within the NinjaTrader 8 folder on your PC, this should usually only be done when directed to do so by a support representative in order to prevent unintended consequences. You may simply be falling into one of the scenarios when NinjaTrader will download historical data from your data provider:

https://ninjatrader.com/support/helpGuides/nt8/index.html?loading_historical_data.htm#WhendoesNinjaTraderdownloadhistoricaldata

NinjaTrader has an interface for backing up and deleting/editing historical data files and you could try that instead of manually modifying the files and folders. You could create a backup at Control Center > Tools > Export > Backup File. Including Historical Chart Data will add the historical data to the backup file. It is also a tip in the help guide, "Market data files such as Historical chart data can often times be re-downloaded from your data provider. Market Replay data can be downloaded from the NinjaTrader servers for the most popular Futures and Forex instruments. If the data files you have stored on your computer are available from your data provider, you can save time and storage space by excluding these items from your backup and simply re-downloaded the data when needed. Please make sure to check with your data provider to ensure they still carry the type of data for the time period you may require."

Then, you may delete the historical data files after the backup is created from the Control Center > Tools > Historical Data window:

https://ninjatrader.com/support/helpGuides/nt8/index.html?editing.htm#HowToDeleteHistoricalData

After the trading day, you may then import the backup you created at Control Center > Tools > Import > Backup file. For more details and steps related to the Backup and Restore utilities, please see the help guide section here:

https://ninjatrader.com/support/helpGuides/nt8/backup__restore.htm

Otherwise, please explain the performance impact you are experiencing and we may be able to identify a better resolution to help you optimize NinjaTrader's performance.

Please let us know if we may be of further assistance.

Thank you for your support.
I have tried exporting backup for historical chart data. But it seems that it is not working for order flow tick data (Bid/Ask). It is working for normal open/close/High/Low/Volume data. Can you confirm whether the order flow tick data is included in the backup?
Regarding the performance issue. I confirm that accumulating many sub folders in the db\tick folder renders loading the order flow charts very slow. Once I clear the db folder, everything becomes very fast again.
Thanks again.

[MSerag]

To add more information. I always set 'Days to load at 0' to retrieve only the current day. However, the loading is slow as long as there are many sub-folders in db\tick

[NinjaTrader_Emily]

Hello MSerag,

Thank you for your reply.

It sounds like the reason you are needing to connect to Kinetick again is because the end date contains the current day. Even if you have Days to load set to 0, if data for the current day is needed then NinjaTrader will send a request to your data provider. This is mentioned in the link I previously shared regarding "When does NinjaTrader download historical data?" found here:
https://ninjatrader.com/support/help...tm#WhendoesNin jaTraderdownloadhistoricaldata

The backup should include any bid/ask data that is stored in the db\tick folder. If you would like to look at what files are backed up, you may rename the backup file so that the extension is changed from .nt8bk to .zip. This will convert the backup file to a zip folder that you may open and review the contents. Please feel free to do this and compare the backed up historical data files to those in the db folder of your installation. You may always change the extension back to .nt8bk to convert it into a functioning backup file again.

I would like to further investigate why the loading seems to be slow based on the sub-folders included in db\tick. Please send me your log, trace, workspace, chart data, and db files. You can do this by going to the Control Center-> Help-> Email Support. Ensuring 'Log and Trace Files' is checked will include these files. This is checked by default. Please also expand "Other files" and check the boxes for Workspace files and UI settings, Chart data, and Database file. Please include "ATTN Emily C" in the subject line of the email along with a link to this thread in the body of the email.

I look forward to assisting you further.