Problem importing tick data into HDM

[adamus]

I have attached the tick data file that is causing problems. It won't import and gives the error:

29/04/2010 10:12:38 Default $USDCHF: Time stamp line 100 is smaller than timestamp of previous line

I can't see a problem with any of the data in the file. This is not an isolated case either - I have come across it before when importing some other data.

The symbol is $USDCHF for Swiss Franc : USD forex.

It took a long time to isolate the data in the original file which was 9 gigabytes, so I hope you can find the problem and fix it.

Thanks

[NinjaTrader_Bertrand]

adamus, which import options were you choosing exactly for this file?

When I imported your snippet, I just got an issue with the volume being zero and thus it was replaced by a synthetic 1, which is expected.

[adamus]

Hi Bertrand,

I am using:

Format: ninjatrader
DataType: last
Time zone: Eastern GMT-5
Generate minute bars checked
Generate daily bars checked

I also see the zero volume warning, but it never stops the import like this timestamp issue.

[NinjaTrader_Bertrand]

Thanks, unfortunately can't reproduce here, with your settings I can import fine here, too. Could you perhaps send a bigger file snippet to me via YouSendit? You could direct it to my Attn to support at ninjatrader dot com.

Thanks

[adamus]

Bertrand,

I've got asynchronous DSL, so uploading it from my machine would take days - it's 300 meg - but I can give you the ftp site where I downloaded it and my password, which is still valid for ten days. However, I can't give it to you here on this forum - do you have a private email address or skype?

[NinjaTrader_Bertrand]

Sure, great - I'll PM you shortly.

[adamus]

I have emailed you the ftp url and password. I hope it displays the bug, although I'm surprised that you couldn't reproduce it.

I tried several times to import the data and every time consistently it stopped with the same error, even with different sized files.

And if the code is the same (beta 14) it must be the PC and OS differences that cause it, but I can't understand how that could affect it, assuming your running the same code. Maybe you are running an internal update of beta 14?

[NinjaTrader_Bertrand]

Thanks, got it - will check into shortly. Perhaps you can try also with a freshly reinstalled .NET framework to iron out any issues from this end?

[adamus]

Same "timestamp is smaller than previous" problem with other data

I've got the same problem with another tick data file I want to import. I attached the small extract from it which causes the problem. The data looks fine, it's only a small file, please have a look at it.

What is probably causing the problem is the missing ten minutes between 19991030 2059 and 19991030 2111.

This is exactly the same gap in the data that is in the first tick data file I uploaded. It's something to do with the Swiss Franc - this is EURCHF and the first one is USDCHF so it's not surprising that there is a gap in the same place, presumably due to technical reasons at the data supplier's collection point.

However this is certainly not a reason for NT to throw an error. Did you look at the first file's data or can you look at this one and tell me if you think there is anything actually wrong with it (i.e. like the error log states).

I also noticed that NT has actually applied +5:00 hours to the tick data, which makes sense because I selected the Eastern time zone in the dropdown and my computer is set to the GMT0 time zone. But then I decided to look at the data with my computer time zone set to Eastern and the HDM displayed the timestamps in exactly the same way as before, i.e. +5:00 hours. So I assume that the tick data is stored in the HDM without a time zone and displayed as it is stored with no time zone conversion. Please could you verify that I am correct? It seems like missing functionality - although it doesn't make any difference to me, I would just like to know.

Thanks

[NinjaTrader_Bertrand]

adamus, I tried both files provided and they imported with an issue over here, the only thing I see again is the log notice about the volume being replaced with 1. NinjaTrader will use the local the local time zone and clock as reference thus it converts the data imported to your timezone, changing the clock after import does not change this, you would then need reimport / redownload the data.

Did you gave my previous suggestion to retry on a clean .NET a try already?

[adamus]

Yes I tried a .NET install. It was an outside chance that it would have any effect and sure enough it didn't.

I just realised that the other difference between my system and yours is that I have my timezone set to GMT+0 / London / Dublin. What's yours?

Could you try doing an import with your OS timezone set like mine?

[NinjaTrader_Bertrand]

I'll it in a sec here on another PC and your timezone, too.

My main PC is in Mountain Standard Time, but as said NT would convert upon importing as needed.

[NinjaTrader_Bertrand]

I tried with multiple time zones on 2 PC's - works here without issues - I'll have a coworker try on his end, too.

Which OS are you using? I tried on Vista Business and XP Professional.

[adamus]

I'm using XP Pro.

You've running beta 14 or 15, presumably?

The data is in Eastern Standard time, that's the choice I make in the HDM timezone dropdown, and Windows is set to London/Dublin.

I'll set my timezone to Eastern and see if it still throws the error.

[adamus]

If I set my PC to Eastern Standard time, I can import the data.

If I set it back to London/Dublin, the import fails again.

Can you replicate that at all?