The database file is larger than the configured maximum database size

[NinjaTrader_Bertrand]

gena72, thanks for the reply. To clarify further we have tested your 2 SDF files kindly provided and when we use the DB's without your scripts in our local installations > the sizes drop hugely to the expected levels immediately (around 8MB). This would tell us orders and executions data are not an issue here but the information in the persisted strategy instances are taking in your DB (which are removed if the current local install could not support the needed script, as in our case as we did not have your script alongside to test). I feel it is probably related to your custom data structures used as your pointed out already as suspicion - we have so far not reviewed a case like this and would need to look into with a simple reproducible scenario on our end so we could

a) advice what the issue is
b) potentially involve our development team to review

Thus that would be the next step needed continuing to investigate this case scenario with you.

If you could look into providing a simple script that would mimic the skeleton you have employed we should be able to hit it here as well.

Thanks in advance

[gena72]

Hello Bertrand,

Thank you very much for your response. I also managed to view the DB file with CE SQL viewer, and also noticed that DB files drop in size. One file indeed dropped to close to 8Mb, but the other to over 70Mb. When I checked what's taking up the space, it turned out that it's "userdata" column in "nt_strategy" table. I'm not sure what this is, but it looks like its a serializaion data of all my strategy properties. Am I right?
Not sure why NT stores them after all, especially periodically, and especially every time when an order is executed. Is there any way to disable such storage/serialization? Maybe at least partially, with XMLIgnore property, or something similar?
I can do a memory profiling on the whole thing and identify the largest data structures.
Providing a skeleton may take a while, but if this is indeed a problem, any strategy which allocates over 250Mb of memory would do the trick.
Involving NT development team also is definitely a possibility.

Regards.
Gennady

[NinjaTrader_Bertrand]

Hi Gennady, further research into the item confirms that the strategy data would be definitely persisted as you guessed, however we would expect entries in the strategyData field here, not userData.

I would unfortuantely not have a direct workaround to offer, however would definitely suggest to give the XmlIgnore attribute idea a run to see if it has impact in your case.

[gena72]

Ok, I will try XMLIgnore (after I do memory profiling and identify the largest structures). Also, I'd like to know if there's a way to tell NT to place NinjaTrader.sdf in a different directory? When DB file becomes large, its re-writing takes a very long time, during which NT freezes completely and looses data connections when it un-freezes. So putting this file on a much faster media (SSD/RAM drive) may be a good idea.
Generally, this whole DB issue is a big problem, and I'd expect NT team to provide some kind of solution/patch.

[gena72]

Well, I tried XmlIgnore, and NonSerialized attributed and it didn't help. Probably the persistance is done not through regular serialization, but by using some sort of proprietary mechanism.
I also tried to link the DB file to a RAM drive, but NT refuses to start in such configuration. Constant re-writing of a 250Mb-sized DB slows NT significantly, up to a point where it freezes and looses connections. I also managed to get more large DB snapshots, which are not reduced after compacting. I still see a userdata field in nt_strategy table as being a culprit here. There's no strategyData field I can find in any table.
At this point I'm not sure how to proceed.

[NinjaTrader_Bertrand]

Gennady, talking further to development they believed the XmlIgnore should have aided in sorting this out. What we would really need at this state is a simple script that could be used to reproduce the issue on our end, I understand you can't send your original script, however perhaps a simplified version with your IP removed but using the same approach would be an option? We would highly appreciate that.

[gena72]

Hello Bertrand,

After I added XmlIgnore attribute to certain inner class members of "nested classes" I use, the DB is no longer expands that much. The trick here seems to be is to add this attribute not only to a Strategy class members, but to all nested class members which may have substantial memory footprint. The DB now is around 10-15Mb. I also moved the whole "db" directory to a RAM drive, so slowdown is now minimal.

Thanks a lot for your help.

[NinjaTrader_Bertrand]

Thanks for letting us know Gennady, great to hear about that update.