Q on Database Reset

[Austiner87]

Hi NT8 strategy support,

Is there a faster way I can reset my database, I am doing a lot of backtesting through market replay and it takes a long time to delete all of the historical orders (which I need to to or else the replay slows down). Can I delete all of these orders in a back end folder instead of doing so through the NT8 software. it would even be faster if I could delete all of my NT8 files (minus strategies and other code), redownload the software and re-install it, as it takes about 6+ hours to reset the database - would the re-downloading the software work. Or is there an easier way?

[NinjaTrader_Jesse]

Hello Austiner87,

If you are using the playback connection the fastest way would be to reconnect to the connection. The playback connection does not store any trades in a database or in files, those exist on the playback account for as long as it has been connected. For a strategy an even faster way would be to disable then re enable the strategy however this also relies on the strategy not placing trades in historical data.

Resetting the database would only be suggested to clear Sim101 orders that had been placed in realtime and are recorded in the database.

[Austiner87]

Hi Jesse, I will try this, however, this is my experience: I run many strategies together on many instruments all on playback connection. Throughout the time period of playback, the max speed slows down (understandable as I have made many trades after, say, a month). When I disconnect and reconnect again to playback connection, this slow speed is the same unless I reset the database. On the other hand, if I reset the database and restart NT8, the playback connection speed goes back a normal pace when on max speed. I believe this is the same if I close out of NT8, restart it, and then connect again to playback without resetting the database - it will be slow unless I reset it.

What I find interesting about your response, is, I have reset the database, where, in between resets, I have not made any trades on Sim101 - only the Playback Account. Yet, resetting the database still takes forever, as if it is deleting something - but your response implies it should have nothing to delete. My Q is: does this sound right? Do you know why I may be experiencing this slowness outside of the reasons in your response?

[NinjaTrader_Jesse]

Hello Austiner87,

What you are explaining wouldn't be expected to happen unless your scripts are doing something which is slowing down the platform. In some situations having a huge number of trades on every bar could potentially present a slowness to render all the execution information.

When you disconnect/reconnect that should clear the previous trades that the strategies made because they are on the Replay101 account. There should be no trades recorded in the database when using playback.

I would suggest trying the SampleMACrossOver on the instrument and for the period of time you test for and see if any slowdown happens or if you find that the database needs reset. That would help to further isolate what may be happening.

[Austiner87]

Hi Jesse,

Looking to re-open this thread - I am having a different issue with my playback connection. When I try to connect, the Playback window is greyed out and never goes green again. I can reset the database in the database management option in the Tools menu, but to delete all the data takes hours. Once it's deleted, I can use the playback connection again. Is there something you can help me with this. If I need to send the log and trace files, can you please send me the instructions for that! thank you!

[NinjaTrader_Jesse]

Hello Austiner87,

As mentioned the database shouldn't have anything to do with playback, the trades you make in playback are not stored in the database. The database would be used for other aspects of the platform like the Sim101 account.

It sounds like you have a lot of data from some other kind of testing that you are doing outside of playback if the time it takes to reset the database is very long. What specific actions are you doing in-between disconnecting from playback and before connecting to playback when it stays greyed out? Are you allowing a strategy to place a large number of Sim101 trades?

[Austiner87]

Hi Jesse, I think I need perhaps to send in my Log and Trace files: here is my experience:.

I connect to playback and enable 1 or more strategies to test performance, they make several hundred to thousand trades, I modify some code, and test again a few times. At some point, the playback freezes the control window, so I need to close NT8 and restart (this would be OK if it happened once and a while / no big deal). When I open up NT8 again and connect to playback, the playback remote control window is greyed out and it never turns green. I open up a new chart and it begins to say 'loading ...' at the top, after a few seconds, the ellipses and chart freezes and it never connects to playback. Sometimes the control window freezes and I can restart NT8 to the same effect, sometimes I can select "disconnect" from the Playback connection, and after I select "disconnect" the Control window freezes and I need to close and restart NT8 again. The only way I can get Playback working again is to reset the database. It takes hours to reset before saying "you must restart NinjaTrader for this to take effect, etc". Note that I didn't even connect to NT8 live and do anything with Sim101 account in this scenario. This is my experience every time I use playback and test a few strategies that rack up a >~1,000 trades.

[NinjaTrader_Jesse]

Hello Austiner87,

The log and trace files won't specifically help with this type of issue, there is nothing logged for playback in the logs. That would be mainly useful for live order submission or if you are seeing a specific error in the control center log tab, that information would be logged.

You mentioned that the strategy is freezing during playback, are you force closing the platform at that point? That could be a reason why the database would be involved. The database may need to be repaired rather than resetting it if for some reason that had corrupted it.

To continue troubleshooting you will need to run the SampleMACrossOver to see if that also can produce the same freezing or end result. Removing the custom strategy from the equation could help to see if that script is somehow causing an issue or if something else is happening in the platform. If the strategy is placing a huge number of trades in a very short period of time that could also relate to the problem if your PC is having performance issues caused by that.

[jeronymite]

Austiner87 This may be useful, notably the third observation I list there: https://ninjatrader.com/support/foru...59#post1193859

The basic approach you may want to consider is:

1. Create a new database file (close NT; rename the existing file to, say, .sdf_old; restart NT)
2. Configure NT as you would want it to be before starting to trade
3. Close NT and copy the new .sdf somewhere as a template for the future
4. When you want to reset: close NT, rename existing db as above, copy the template to be the new usable db file, and restart

Advantages:

• Very quick compared to when you have a large number of orders in the history
• Ready-to-go db compared to re-install and other similarly drastic actions

Disadvantages:

• Suffers from the inherent disadvantage that you lose all history. You can, of course, save all the "old" db files and rename appropriately for review if you want to see the history in them. Messy, and limited to reviewing one file at a time, but possible if you do not want to lose the history entirely.
• You lose all customisations since the template was created, although you can, of course, update the template in a similar way to keep it as you want.
• On a new NT release, it would be wise to repeat the process with the new NT version to ensure you are using the latest db structure

I hope that's useful for you.

Thanks.