Downloading data to speed up backtesting not working?

[HaveGunsWillTravel]

I have read all I can in the guides and done everything correct to my knowledge. I checked the box in SA Properties to use local data for back-testing.

I tried various things:

I read in a post that one way to get data local is to open two charts...for example:
My strategy uses a 50-minute and a 1-tick series (for order placement).
I opened a 1-tick and 50-min chart, custom range back to 1-1-2020 (same range as my back test I want to run).

It took a long time to download the data for the chat, and I see my RAM building up to about 12GB, so the data is in memory.

But when I ran the back test, it just started downloading data again, and I see the RAM building up even more...so it did not appear to work.

So then I try the Download in Historical Data Manager:

I pick my instrument and all the appropriate stuff, tick, minute, day etc.,
I ensure I am connected to Continuum which says in the guide all that data I want is provided,
I click download, and for 1 second a message appears in the lower right corner but then goes away. The Download button remains greyed out, and my computer ramps up and RAM grows, so it appears to work. But I get "Unhandled exception: An item with the same key has already been added" when it finishes.
I tried a back test again, and still, it started downloading data.

So I go into Database manager, update everything just because, and then go back to Download in Historical Data and I actually delete all the historical data for the same ticker in the Edit Tab, just in case, and I start again:

Again, for 1 second a message appears in the lower right corner (as described in the guide) but then goes away. Nothing is showing anything about progress.
The computer worked for about an hour, RAM building up, CPU working, and all the Download parts greyed out with the 'blue busy wheel' present.
It finished and CPU went back to idle.
Same error: "Unhandled exception: An item with the same key has already been added"
I see the RAM reduced down to 775MB for NT8, but the busy wheel is still going. Nothing else is open for windows, just Historical Data, If I try to click the edit tab, the same error pops up.

So I am forced to close NT8 and re-open, when I go back to the Edit tab in Historical data, the data I tried to download is not there, and all of the stuff I deleted previously is back??

I am not sure what I am doing wrong...

[NinjaTrader_Jim]

Hello HaveGunsWillTravel,

Thanks for your post.

Data is downloaded when a request is made and the data is not already present on the platform.

To clarify, do you see the error when downloading data or when running a backtest?

If the error involves running a backtest, do you see the error when testing the SampleMACrossover strategy?

If the error is specific to downloading data instead, I would like to narrow down what steps I would need to take to hit the error from a clean environment of NinjaTrader 8 (Release 25) Could you set up a clean environment to test? See the bottom of my post for instruction.

If the clean environment works, but your existing environment does not, I may suspect cache may be damaged, and we could try clearing cache in the original environment. (This should not be done for maintenance, but if something occurred where cache broke, it may be related. We then would want to try to figure out how the cache broke if that fix resolved, because cache is there to make the platform faster in loading bars.)

^ Let me know if we get to this point and we can continue with additional troubleshooting steps. If you have steps for me to take that can reproduce an issue from a clean environment, please also let me know since this would be something I would like to hit on my end.

Clean Environment Test:

Creating a clean environment can be done by following the steps below:

1. Close NinjaTrader 8, and rename the "NinjaTrader 8" folder in My Documents to something like: "NinjaTrader 8 OLD" Do not delete this folder.
2. Uninstall NinjaTrader from the Windows Control Panel
3. Delete the C:\Program Files (x86)\NinjaTrader 8 folder
4. Reinstall using the installer from http://ninjatrader.com/PlatformDirect
5. Add the required indicators and test the template.

If you ever need to switch back to your original platform, you may do so by closing NinjaTrader and swapping the platform folder names.
For example, Close NinjaTrader and rename the new "NinjaTrader 8" folder to "NinjaTrader 8 NEW" and the "NinjaTrader 8 OLD" folder to "NinjaTrader 8." Then restart the platform. Simply put: NinjaTrader 8 will always load the "NinjaTrader 8" folder in My Documents.

[HaveGunsWillTravel]

Hello HaveGunsWillTravel,

Thanks for your post.....

It is when downloading so I will have to do the clean environment steps. To clarify, when I set this new one up, I will enter my same license key and log into continuum right? Then try the download?

[NinjaTrader_Jim]

Hello HaveGunsWillTravel,

Yes, after following the mentioned steps to set up the clean environment, you can enter the same license key and set up the NinjaTrader Continuum connection.

After this we should be able to test the data requests.

If we hit an issue in the clean environment, I would like a some clarity on specific steps to take and data to test.

If this works fine, we should test the existing environment again, and we may want to look into cache.

I will elaborate on further troubleshooting steps if we get to this point.

[HaveGunsWillTravel]

Hello HaveGunsWillTravel,

If this works fine, we should test the existing environment again, and we may want to look into cache.

OK so it works fine! I was able to download two different symbols. I saw all my choices and contracts pop into the Edit tab Historical Data drop down, and when I removed some they stayed removed after rebooting NT8. So that all works now. The one thing that did not is the progress message in the lower right corner. It appears for a split second then disappears like before. That is not a big deal to me though so long as I can download data and use it. I clicked local data only in SA properties and ran a few random back tests. They all ran great on the local data. I logged every single step I did but did not post it since it works and I figure you don't need it. If you do though, I can post it.

Thanks for the help, let me know what is next...

[NinjaTrader_Jim]

Hello HaveGunsWillTravel,

We can try clearing the cache in the existing environment and try testing again. If this resolves the issue, and the issue comes back, we will want to see what is breaking the cache. Please then clear cache again and take note of what you did specifically that lead up to the cache breaking and the issue coming up. Sometimes we have seen bar cache issues with custom BarsTypes. If a custom BarsType is related, we would need to get in contact with the custom BarsType developer so they may work with us.

Bar Cache is saved in the Documents\NinjaTrader 8\db\cache\ folder. You may delete this folder after NinjaTrader is closed.

If clearing cache does not help, please close NinjaTrader, and move the Documents\db\ folder to a safe place on your desktop, and then run a Repair operation from a NinjaTrader 8 installer (this will essentially test removing all data from the existing environment and restoring the file structure in the DB folder.) You may use the installer downloaded from the link in post #2.

Let me know where we are at from here.

[HaveGunsWillTravel]

Hello HaveGunsWillTravel,

We can try clearing the cache in the existing environment and try testing again. If this resolves the issue, and the issue comes back, we will want to see what is breaking the cache. Please then clear cache again and take note of what you did specifically that lead up to the cache breaking and the issue coming up. Sometimes we have seen bar cache issues with custom BarsTypes. If a custom BarsType is related, we would need to get in contact with the custom BarsType developer so they may work with us.

Bar Cache is saved in the Documents\NinjaTrader 8\db\cache\ folder. You may delete this folder after NinjaTrader is closed.

If clearing cache does not help, please close NinjaTrader, and move the Documents\db\ folder to a safe place on your desktop, and then run a Repair operation from a NinjaTrader 8 installer (this will essentially test removing all data from the existing environment and restoring the file structure in the DB folder.) You may use the installer downloaded from the link in post #2.

Let me know where we are at from here.

OK, should I be restoring my previous environment somehow, I assume? Just replace the 'OLD' folder with the new one created by the 'clean' environment and go from there? You mentioned that so I assume so, but how to clear the cache? I just literally delete Documents\NinjaTrader 8\db\cache\ and that is it?

Thank you.

[NinjaTrader_Jim]

Hello HaveGunsWillTravel,

Yes, you can Close NinjaTrader and swap folder names to switch the environment.

After you swap folder names and before you restart NinjaTrader, you may delete the contents inside the Documents\NinjaTrader 8\db\cache folder.

Then we can restart NinjaTrader and test with cache cleared.

If the issue persists, please close NinjaTrader and test moving the Documents\NinjaTrader 8\db\ folder to a safe place on the desktop, run a Repair operation, and test again.

[HaveGunsWillTravel]

Hello HaveGunsWillTravel,

Yes, you can Close NinjaTrader and swap folder names to switch the environment.

After you swap folder names and before you restart NinjaTrader, you may delete the contents inside the Documents\NinjaTrader 8\db\cache folder.

Then we can restart NinjaTrader and test with cache cleared.

If the issue persists, please close NinjaTrader and test moving the Documents\NinjaTrader 8\db\ folder to a safe place on the desktop, run a Repair operation, and test again.

OK I did this:
RENAMED present folders to NEW then, OLD to present…
DELETED everything inside of Documents\NinjaTrader 8\db\cache
Restarted computer,
Restarted NT8,
Busy wheel present immediately for minutes with no explanation but I see zero resources are being used by NT8, minimal CPU, no RAM build up, no Network.
Despite busy wheel I try to remove old items from edit tab of Historical Data and all seems to delete as expected
I close NT8 and save the workspace.
On restart, the previously deleted ‘ES’ time frames are gone, (unlike before).
Seems despite ‘busy wheel’ operations have stuck.
The ‘busy wheel’ is present all the time still, but it shows no resources being used by NT8 in windows, I am still able to do whatever I click on.
I try Historical Data window to download ES again 1-1-2021 to 3-22-2022. tick, minute, day, ask, bid, last (same as before).
I see resources ramp up as expected, busy wheel is still present when hovering over HD window, Still no progress message in lower right corner.
[A bunch of other notes that are probably not important on resource observations, but ask if so]

Same error as before. 'key already exists etc..." only after I click somewhere when resources stop humming...no other indication of it being done,
BUT!

I look in HD manager and all the data is there that I queued for download.
So I restart, look again, and yes it is all there as downloaded this time.

So it worked, but the busy wheel is always present? Even after restart..which is new

Something definitely seems off with my 'build'? The clean version seemed to work pretty flawless...

[NinjaTrader_Jim]

Hello HaveGunsWillTravel,

Could you email me at platformsupport [at] ninjatrader [dot] com with the text "Attn Jim 3538861" and also include a link to this thread?

I would like to get connected to give more hands on assistance. I can reach out as late as 4:30PM EST today, and tomorrow as early as 9AM EST. (Please include a phone number and a time + time zone that works well for you.)

[HaveGunsWillTravel]

Hello HaveGunsWillTravel,

Could you email me at platformsupport [at] ninjatrader [dot] com with the text "Attn Jim 3538861" and also include a link to this thread?

I would like to get connected to give more hands on assistance. I can reach out as late as 4:30PM EST today, and tomorrow as early as 9AM EST. (Please include a phone number and a time + time zone that works well for you.)

Done, talk tomorrow. Thank you.