Beta15 Export history bug.

[xTrader1]

I exported a data , performed a clean install and tried to import a data again.
An import failed , I got a message :
$EURGBP: Time stamp line 2689216 is smaller than timestamp of previous line
A problem really exists, here's a snapshot of the text:
20100405 200020;0.88377;1000000
20100405 200020;0.88374;1000000
20100405 200023;0.88376;1000000
20100405 200023;0.88376;2000000
20100405 200023;0.88376;1000000
20100405 200023;0.88378;1000000
20100405 200000;0.88372;1000000
20100405 200000;0.88368;1000000
20100405 200011;0.8837;1000000
20100405 200011;0.88367;1000000
20100405 200012;0.88369;1000000
20100405 200012;0.8837;1000000
20100405 200014;0.88372;1000000
20100405 200014;0.88368;1000000
20100405 200015;0.8837;1000000
20100405 200015;0.88367;1000000
20100405 200019;0.88369;1000000
20100405 200019;0.88365;1000000
20100405 200020;0.88367;1000000
20100405 200020;0.88364;1000000
20100405 200027;0.88366;1000000

I would not complain, would it be an external data source, but this data was created by NT7 Beta 15 export.

I tried it edit history files manually, but an amout of time mismatches makes it next to impossible.

[xTrader1]

Further Investigation:

I thought that the issue is file size detailed, therefore I tried to export smaller pieces of the same data . The error, described above , appears spuriously, every time in another place.

[NinjaTrader_JoshP]

xTrader1,

To clarify, you are able to always export this date for $EURGBP and get that error when trying to import the file that was exported on the same line every single time?

If so, can you please zip up your Documents\NinjaTrader 7\db folder and send it to us? Please also inform us of the date and export type to export to test this on our end. Thank you.

[xTrader1]

xTrader1,

To clarify, you are able to always export this date for $EURGBP and get that error when trying to import the file that was exported on the same line every single time?

I'm always able to export. The problem appears during import and when I see the file , that's a data. And it is not only $EURGBP , also $EURUSD , I didn't try another pairs. And every time I'm getting this error in another time.

It seems , certain race conditions occur, when exporting.

Which files do you want to be sent to you ? An entire folder will be >100mb after compression.

[NinjaTrader_JoshP]

It is still not clear to me. Can you please clarify? If you export the same date does it or does it not create a a text file that has an issue at the exact same part when you import? So, when you export twice the same dates. Do you get two text files that have the exact same errors when importing them?

[xTrader1]

Detailed description.

It is still not clear to me. Can you please clarify? If you export the same date does it or does it not create a a text file that has an issue at the exact same part when you import? So, when you export twice the same dates. Do you get two text files that have the exact same errors when importing them?

OK , I'll be detailed. In the image attached, you see my available tick data , from 8/2/10 to 23/4/2010 .
When I'm exporting all the data , during the import , I'm receiving the first error on these ticks :
20100406 120006;0.88509;1000000
20100406 120006;0.88509;2000000
20100406 120006;0.88509;1000000
20100406 120007;0.88511;1000000
20100406 120007;0.88511;2000000
20100406 120007;0.88511;1000000
20100406 120007;0.88513;1000000
20100406 120000;0.88497;2000000
20100406 120000;0.88493;1000000
20100406 120000;0.88495;1000000
20100406 120000;0.88496;1000000
20100406 120001;0.88498;3000000
20100406 120001;0.88493;1000000
20100406 120001;0.88495;1000000

the error log says:
$EURGBP: Time stamp line 2729616 is smaller than timestamp of previous line

but when I'm exporting from 15/2/10 till the end , I'm receiving afterwards an import error on another tick :
$EURGBP: Time stamp line 2042869 is smaller than timestamp of previous line
and errorneous ticks are:
20100406 120007;0.88511;1000000
20100406 120007;0.88511;2000000
20100406 120007;0.88511;1000000
20100406 120007;0.88513;1000000
20100406 120006;0.88497;2000000
20100406 120006;0.88493;1000000
20100406 120006;0.88495;1000000
20100406 120006;0.88496;1000000

Please pay attention that price /volume is the same but the time is different.

I thought the files are too big to be exported.
I tried to export a data from 1/4 to till the end.
During an import I received the first error :
20100406 140010;0.88366;1000000
20100406 140010;0.88367;1000000
20100406 140007;0.88381;1000000
20100406 140007;0.8838;1000000
20100406 140007;0.88379;2000000
20100406 140007;0.8838;1000000

the same day , another time.

Exporting from 3/4 till the end, gave yet another error.
20100406 200048;0.87839;3000000
20100406 200048;0.87839;1000000
20100406 200045;0.87831;1000000
20100406 200045;0.8783;1000000
20100406 200045;0.87832;3000000

...ETC...

All exports/imports were "ask" price. During import I built minute bars.

Just looking at my mail, I paid attention the problem appear in all cases above at the 1-st minute of an hour.

[NinjaTrader_JoshP]

xTrader1,

Please zip up and send in Documents\NinjaTrader 7\db\minute and then the folder for the instrument you were exporting. Thank you.

[NinjaTrader_Dierk]

The problem is that the data which you have in your repository already is corrupted, meaning e.g. ticks with a timestamp
20100406 120007
... are saved in the wrong file. They should be saved in the file
20100406 1300.ntd
... but are saved in
20100406 1200.ntd

The real question is: where did this data come from? I recall that earlier betas had an issue with recording tick data which should be resolved. So: did you record that data or where do you have that from?

[xTrader1]

The problem is that the data which you have in your repository already is corrupted, meaning e.g. ticks with a timestamp
20100406 120007
... are saved in the wrong file. They should be saved in the file
20100406 1300.ntd
... but are saved in
20100406 1200.ntd

The real question is: where did this data come from? I recall that earlier betas had an issue with recording tick data which should be resolved. So: did you record that data or where do you have that from?

The real answer is that the data arrived from a big (>256MB) backup done on Beta13. I tried to export the data to text file and re-import it again just to avoid backup imcompatibility and huge backups problem in earlier betas. Then I run into this bug. But again, pay attention , there was no problem, restoring a huge beta13 backup onto beta15, I didn't see any data problems in my charts or indicators, even if certain .ntd files violated time sequence.

If this is a problem, you should announce, that in certain circumstances earlier tick data may be not available anymore. Of course, I can write a utility that will do the work, but I would prefer to spend my time on trading.

[NinjaTrader_Dierk]

>> arrived from a big (>256MB) backup done on Beta13
And where in B13 did the data come from? Thanks

[xTrader1]

>> arrived from a big (>256MB) backup done on Beta13
And where in B13 did the data come from? Thanks

From MBT Navigator adapter collected real time data, demo account. I don't use historical servers.

[NinjaTrader_Dierk]

Do you recall which version of NT you had been using to record the data? You could check in the trace files which still should be there... Thanks

[xTrader1]

Do you recall which version of NT you had been using to record the data? You could check in the trace files which still should be there... Thanks

I have no trace files, because I performed clean install everytime upgrading beta. The first data is from 8/2/2010 , I cannot remember beta version. MBT Nav was 11.3.0.4 , that's I remember for sure.

[NinjaTrader_Dierk]

That's odd..

Please let me know as would experience the same trouble again using NT7B15 and MBT Nav as per our web site: http://www.ninjatrader.com/webnew/support_MB.htm

Thanks

[xTrader1]

That's odd..

Please let me know as would experience the same trouble again using NT7B15 and MBT Nav as per our web site: http://www.ninjatrader.com/webnew/support_MB.htm

Thanks

Dierk I lost you. I am using beta15 and and the last version of MBT Navigator. I didn't used neither beta 15, nor MBT Navigator Version 11.5.0.5 , when they were not available yet. As a disciplined beta tester, I am replacing Beta versions, when they are available, and according to your recommendations I perform a clean install. In these circumstances they only way to preserve my data is either backup / restore or export / import. So... How can I help ???