Database file too large

[egan857]

I just got the following 3 error messages, some of them hundreds of times (I'm still in the process of clicking through all the dialog boxes):

1. Failed to execute DB job 'OrderUpdateJob': The database file is larger than the configured maximum database size. This setting takes effect on the first concurrent database connection only. [Required Max Database Size (in MB; 0 if unknown) = 257]

2. Failed to execute DB job 'AccountUpdateJob': there is insufficient disk space to complete operation.

3. Failed to execute DB job 'ExecutionInsertJob': The database file is larger than the configured maximum database size. This setting takes effect on the first concurrent database connection only. [Required Max Database Size (in MB; if unknown) = 257]

The file NinjaTrader.sdf is 262,736K, or just over 257MB, so it is over the maximum limit. I have lots of hard drive space available - over 42GB. I just refreshed the database from scratch about 3 weeks ago, and have had to do so every once in a while for various reasons, but never for this reason. One of my old databases is 1.9GB, so somehow the maximum database size decreased dramatically.

I don't need to keep anything in the database, and plan on resetting the database after I finish clicking through all these dialog boxes. But how do I increase the maximum database size so this doesn't happen every few weeks?

[NinjaTrader_Kyle]

Hello,

Thank you for your post.

Please contact me via email at Support AT NinjaTrader DOT com.

In your note, please include a link to this forum thread and a copy of today's trace file. The trace file is located in the Documents > NinjaTrader 7 > Trace folder. The file which I need is named 'trace.20130219.txt'.

[textheis]

Failed to execute DB job: 2 msg windows

This just starting happening to me too....
My sdf file is ~263K.
I do have Real-time data "Save chart data as historical" checked because the last server/data loss NT incurred, it was suggested that in order not to lose "my place" in the market, I would need to store my own with this feature enabled - so I have.

Please advise...Thanks Todd

[NinjaTrader_Jason]

Hello Todd,

Can you please check the disk space of the drive that holds the Documents-->NinjaTrader 7 folder. Could it be there is no space left?

Try repairing the database and check if this will solve the error. You can repair the database at Control Center-->Tools-->Options-->Data-tab and click Repair DB.

If the issue remains, please send me your trace file. Trace files can be found at the following location. Please attach it in your email and send it to support[at]ninjatrader[dot]com. Put 'Att Jason, Database file too large' in the subject header.

C:\-->Users-->*Windows user account*-->(My) Documents--> NinjaTrader 7--> Trace--> trace.20130524 for today. (trace.yyyymmdd for any other date)

[LTWCI]

I get the same error message regarding the data base. When I keep going and click retry, NT is stuck at "initializing database"

[NinjaTrader_Jason]

Hello LTWCI,

Can you please proceed as per my previous post.

[textheis]

Hello Todd,

Can you please check the disk space of the drive that holds the Documents-->NinjaTrader 7 folder. Could it be there is no space left?

Try repairing the database and check if this will solve the error. You can repair the database at Control Center-->Tools-->Options-->Data-tab and click Repair DB.

If the issue remains, please send me your trace file. Trace files can be found at the following location. Please attach it in your email and send it to support[at]ninjatrader[dot]com. Put 'Att Jason, Database file too large' in the subject header.

C:\-->Users-->*Windows user account*-->(My) Documents--> NinjaTrader 7--> Trace--> trace.20130524 for today. (trace.yyyymmdd for any other date)

Thanks Jason - the first one is pretty obvious - HD has 300+ GB open.
I'll go through the "DB clean-up" and let you know.

By the way, I tried adding another chart to my workspace to see the effect and Ninja crashed. When I re-launched it, I had an open order in two of the 3 charts in my workspace that I cannot move, edit, or delete.

I'll be back in a few with an update......

[NinjaTrader_Jason]

Sounds good, I will await the outcome. If the error remains, please send me the trace file.

[textheis]

Update 1

Jason,
Repairing the DB reduced the sdf file to ~253M. So far, so good.

But, I still have that "ghost" order sitting on the chart within the sim account for 1 Sell T1. Prior to resetting the DB, this ghost was qty 1 on one chart and qty 3 on another chart - different time scales but same contract within the same workspace.

Price has gone through this ghost order up/down multiple times without filling. I've exercised some <None> ATM Strategy orders to see the effect. This did not end up well either. Although the DB error msg is now gone, at one time I had 22 contracts in play, used a target order to close them and all but 3 closed. I then just clicked "CLOSE" with no effect. I then set another target to close 3, it filled and finally shows flat - except that 1 Sell T1 that just won't go away.

Still want the trace file?

Thanks, Todd

[NinjaTrader_Jason]

Thank you for the update.

Can you please reset the sim101 account to remove the ghost order.

You can reset the sim101 account as follows. Please go to Control Center-->Tools-->Options-->Simulator-tab. Enter the values you like on the left side and click the Reset button on the right side. More information can be found at the link below.

http://www.ninjatrader.com/support/helpGuides/nt7/simulator_tab.htm

Please send me the trace file if the database error recurs.

[textheis]

Thank you for the update.

Can you please reset the sim101 account to remove the ghost order.

You can reset the sim101 account as follows. Please go to Control Center-->Tools-->Options-->Simulator-tab. Enter the values you like on the left side and click the Reset button on the right side. More information can be found at the link below.

http://www.ninjatrader.com/support/helpGuides/nt7/simulator_tab.htm

Please send me the trace file if the database error recurs.

That did it - thanks Jason.
As always, excellent speed to reply and quick solutions.

For anyone following this repair job thread - in order to clear the ghost order, you have to leave "Clear History" checked or an error msg "Close all positions" stops the effort.

[egan857]

Hi Textheis,

For what it's worth, once you exceed the 256MB limit for the database file, it takes FOREVER to fix the file. And, when you're all done, you've got an empty database. The simplest solution is to shut down NinjaTrader, rename the database file from NinjaTrader.sdf to NinjaTraderOld.sdf, and then restart NinjaTrader. When NT starts up, it creates a new database if none exists, and you're good to go in a total of a couple minutes, instead of spending hours to repair or reset the database. And, you have no zombie trades lurking around, which might exist after a repair/refresh.

Take a look at Windows Explorer when you hit Reset Database - the database grows quite a bit before shrinking. If you're at or near the 256MB limit and try to reset, it makes the database over 300 MB large and then Ninja gets stuck for around 20 hours (on my machine) trying to shuffle everything around before shrinking because it's over the 256 limit. Resetting a smaller database takes just minutes.

I have a very active simulation, so I hit the 256MB limit every few days. I make it a habit to just rename the database every night after I've reviewed the day's activity. I also have my own instrument list since NT has all the futures symbols but only a fraction of the equities on NYSE and Nasdaq. After refreshing the database, you will also have to reset the instruments if you have your own instrument list.

Jason and Kyle, if you're reading this, please ask the developers to increase the database size limit. 256MB is tiny, and it's not uncommon to see laptops like mine with 1 terabyte of hard drive space. Even an increase to 1GB would be nice. The developers probably don't want to allow the database to get larger than the user's computer memory, but I have 16GB of RAM.

[LTWCI]

Any of you guys install Windows update files yesterday? I did and think it might have to do with this.

[textheis]

Much appreciated Egan857!!!!
This was my first instance as my sim activity would seem to be much lighter than yours. Since I was only ~15M past the cap, it repaired in less than a minute on my machine.

However, as I read through your scenario suggestions more than once, I realized that your having to excerpt a lot of manual database management just to keep from getting locked up during trading. The perception on NT is that the backtesting/sim capabilities are one of the big attractions highlighted as #4 on the main web page feature list.

Jason and team - I must concur with my friends request. I understand DB maintenance is a big deal - especially after having walked away from OpenQuant if you know what I mean. As users, if there was ever a situation where we would be expected to manage the DB under any conditions, that should be clearly noted in the manual at least. What comes to mind to deal with the near term would be some kind of script, alarm, meter of use, or something that would clue us in to the condition arising. It's not exactly on the top of my mind when I reading the market for a trade.

And so it begs the question - would this ever occur on a live trading account? If not, why not? If so, what then?

Thanks guys and have a great Memorial Weekend!
Todd

[textheis]

Any of you guys install Windows update files yesterday? I did and think it might have to do with this.

I know what you mean - but not me.
I'm running W7. If I may share, I've learned a couple of things with those updates.
I NEVER let Windows do it automatically - I have a strict "ask permission for anything done" including downloads to my "business" machine. When I do see the request, I only apply after I'm done with Ninja for the day. Given that any open orders that may take more than a hour or so to close, I'll shut down Ninja, apply the MS patches, reboot and then restart Ninja. So far this has worked well for me over the past year. Prior to setting my little procedure, I would end up with NT locking up on me due to Windows OS (presumably) - not consistently, but enough to cause much undue irritation. Hope this helps.