Debugging my database

[pstrusi]

Hi Ninjas,

I need some help here. I've been optimizing an Algo with my EURUSD database built from Interactivebrokers's daily info. I've never had any trouble, but since yesterday I realized that I've been getting different results with the same input, without changing it previously, which makes me think that's something wrong in the database. As you can see in the chart, obviously something is out of order.

How can I clean it up without change any tick of its integrity ?

A long time ago Bertrand suggested to me that cleaning the folder database-cache would be good enough, but this time it still doesn't work.

Thanks

[NinjaTrader_Bertrand]

pstrusi, judging from this screenshot it seems you're missing historical tick data to build your EURUSD range bars from? If you check for example the Historical Data Manager for those days, what would it indicate in regards to data availability? As the IB connection would not allow for reloading of tick data from the server, a backup 'true' datafeed connection like Kinetick might be helpful to supplement the setup.

[pstrusi]

Bertrand, my database for those days is complete. As you can see it's obvious that it might be a simple issue easily solved with a resetting, I hope.

I'll let u know how it goes after renaming the database

Regards

[pstrusi]

Hi Bertrand, thanks for the solution.

As always the NT support team helped me out in a difficult issue, I posted the solution. As you suggested through the following steps, I was able to build again the complete database in NT; it seems so far, OK.

How to reset the database ( just in extreme case and under NT support team suggestion )

The best deep "cleaning" is through the "rename" your database with the following steps :

Shutdown NinjaTrader and Go to the Start Menu
Select My Documents--> NinjaTrader 7--> DB--> NinjaTrader.SDF.
Right click on NinjaTrader.SDF and select "Rename." *Name it "OLDNinjaTrader.SDF."
Then restart the software and NinjaTrader will create a fresh database file to use.

Unfortunately the following items stored in the old database will be lost – ATM Strategy templates, Session templates, Instrument Lists / Custom Instruments and historical trade execution data
To prevent running into situations where you can potentially lose important data, we advise to run regular backups via our inbuild backup feature in NinjaTrader - http://www.ninjatrader.com/support/h...up_archive.htm

[NinjaTrader_Bertrand]

Thanks for the update Pier and also thanks for including the steps we took in this case. All the best.

[pstrusi]

You're welcome Bertrand.

Let me inform you: the database was created once again with no problem, despite that charts now seem ok, still I have some funny changes with optimization process ( which months ago I didn't ). For example after an optimization or backtest ( for 1 outcome ) when I try to do it again, changing for example the field Max. Profit Factor TO Max. Net Profit, then I have a similar result BUT suddenly one day's results is missing completely; so I must close NT, erase the data cache witihin DB, open again NT and run the whole process again, where then after repeating the same process I can see that day's results again in the outcome, weird ahh?

When market closes today I gonna repair the the database with NT disconnected, searching for bad ticks...I'm running out of ideas about to solve these.

If you have in mind any other process that could help me to evaluate the integrity of the database, please let me know.

Thanks

[pstrusi]

By the way Bertrand, this just happens in the last week of August. For the rest of the database all is pretty good.
So, do you know more procedures in order to evaluate the integrity of the tick database?

Thanks

[NinjaTrader_Bertrand]

Pier, the database file itself would not store any historical tick, minute or daily data - this was the case in version prior to NT7, but now the actual data is externally stored. If you check your Historical Data Manager for the periods where you see the issue, is the data then present / listed in it? Or would you then with gaps in your tick data? As you're working with IB as provider here this would need a 'true' datafeed like Kinetick to provide a backfill from a server.

[pstrusi]

Yes Bertrand, my whole tick data for those days is complete, so far I don't see any gap. Being aware of IB non historical data, but trying to stick my Algo just to the statistical nature of this data, I have my system 24/7 on. I have special alerts programs in case of any disconnection.

So, just imagine that my tick data is well filled, what procedure should I do to inspect ?

[pstrusi]

Another hint to the mystery:

I've just did a backtest until today, with 8/30/2013 missing totally; BUT when I do the same backtest, with the same input and parameters, just changing the final date to 8/30/2013, surprisingly the missing date appears, why those changes?

[NinjaTrader_Bertrand]

What session template do you use in your testing? Can you compare for example to our SampleMA strategy, if the same behavior would be seen? That way we could rule anything specific to your custom script out of the equation.

[pstrusi]

It seems that something that had to do with the session templeate, cause I duplicate the Forex session but when I use the new one, the backtest doesn't give any error.

So does this mean that the session template, in some how after a long number of tests , saturate the database or Strategy Analyzer? Thanks for the tip. I'll keep in mind having a "fresh" session template lol. If you have any suggestion in order to keep these templates in good state, please let me know.

Regards

[pstrusi]

Bertrand, checking out the configuration instruments list, I realized now ( when the test came out good ) that the currency had in the instrument list the session #1, but for the backtest is been chosen the session #2, where session #1 and session #2 are exactly the same. So, does this mean that it should advisable to work with these two same sessions?

[NinjaTrader_Bertrand]

Pier, glad to hear we are getting closer - would yet have to see a corrupted session template, if you check the hours in the Session Manager they do display correct for both and would have the correct timezone as well (EST) like our Forex default one? Usually the backtest would be run with <use instrument settings>, that in turn means it would pickup the default session you have assigned to the master instrument in Tools > Instrument Manager.

[pstrusi]

That's right, both of them have the same hours, days and even the same timezone; but as you say, I suspect that maybe the older session is corrupted. I´ll leave the new one hoping to solve this issue.

Thanks a lot Bertrand