Problem importing tick data into HDM

[NinjaTrader_Bertrand]

adamus, appreciate your patience on this - I was so far not able to reproduce, neither a coworker. Please try this below -

1. Exit NT, backup your complete DB folder and then move this out of NT7's MyDocuments folder, so you start complete fresh on next restart

2. Now import the data file you posted in the clean DB - can you still reproduce those errors (timestamp smaller than prev one)?

3. If not, please close NT again and restore the old DB folder we backupped in 1.

4. Restart NT and retry the import - can you then also reproduce the issue?

Thanks

[adamus]

Hi Bertrand,

I followed your instructions, despite being very nervous about losing the 10 gigs of data that took me 4 days to import.

Unfortunately, the issue arises at both attempts to import it - both in a new, empty database and in the existing one.

[NinjaTrader_Bertrand]

adamus, thanks for testing this still as outlined - so you ran into the same 'timestamp smaller than previous...' one error? The last data you sent me was time data, is the issue the same for you for time and tick based data? Would you perhaps have a snippet of your tick data so I could test this here?

[adamus]

Hi Bertrand, the error message was the same, as you mention.

The attached files, USDCHF.txt and EURCHF.txt, both in the same format are still on this forum.

I call it tick data, maybe that's the wrong terminology but that's the only format I am using.

Both files are still attachments on this thread above.

They are what I tested with, and are cut out from the original huge files I gave you access to on the FTP website at Disktrading.com.

Thanks

[NinjaTrader_JoshP]

adamus,

Unfortunately we were still unable to reproduce. To clarify exactly what was tested so there is no ambiguity, you did test the exact export files you have attached here on this forum with the clean database correct? And you were able to reproduce with those exact snippet data files?

[adamus]

Yes exactly. I actually downloaded the file I tested from this thread.

[NinjaTrader_Bertrand]

adamus, thanks for confirming that - could you please share screenshots of your regional settings used in the Windows Control Panel? Perhaps we can find the missing link here...

[adamus]

adamus, thanks for confirming that - could you please share screenshots of your regional settings used in the Windows Control Panel? Perhaps we can find the missing link here...

Here you are.

[NinjaTrader_JoshP]

Unfortunately we were still unable to reproduce and our out of ideas right now. Development is currently working on a similar case and in that one we were able to reproduce. Hopefully the fix for that one will be of benefit for this one here too in the next beta version. We will keep our eyes out for more reports along these lines. Thank you.

[adamus]

Here's something for you to try out: causes NinjaTrader to shut down

The attached EURGBP file causes NT7beta15 to pop up a warning message that it must shut down, and it leaves the following trace in the trace files.

The error happens when my timezone is set to London/Dublin. If I set it to Eastern Standard, it imports fine.

It is the last line of the file which causes the problem. If I delete that line, the file imports without the error.

To my eye the data in the file is totally in order. Do you see any problem with it? Can you try importing it?

Thanks

[FONT=Courier New]2010-05-09 11:55:05:578 WARNING: $EURGBP: Volume in line 0 is invalid since it is <= 0. This line and subsequent invalid volume lines have been modified and replaced by a value of 1.
2010-05-09 11:55:12:156 in OnUnhandledThreadException
2010-05-09 11:55:12:156 ********* exception trapped *********
2010-05-09 11:55:12:156 The supplied DateTime represents an invalid time.  For example, when the clock is adjusted forward, any time in the period that is skipped is invalid.
Parameter name: dateTime
2010-05-09 11:55:12:156    at System.TimeZoneInfo.ConvertTime(DateTime dateTime, TimeZoneInfo sourceTimeZone, TimeZoneInfo destinationTimeZone, TimeZoneInfoOptions flags)
   at System.TimeZoneInfo.ConvertTime(DateTime dateTime, TimeZoneInfo sourceTimeZone, TimeZoneInfo destinationTimeZone)
   at NinjaTrader.Data.ImportType.Import(Boolean message, MarketDataType marketDataType, TimeZoneInfo timeZoneInfo, Boolean generateMinuteBars, Boolean generateDailyBars)
   at NinjaTrader.Data.HistoricalDataManager.OnImport(Object sender, EventArgs e)
   at System.Windows.Forms.Control.OnClick(EventArgs e)
   at System.Windows.Forms.Button.OnClick(EventArgs e)
   at System.Windows.Forms.Button.OnMouseUp(MouseEventArgs mevent)
   at System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)
   at System.Windows.Forms.Control.WndProc(Message& m)
   at System.Windows.Forms.ButtonBase.WndProc(Message& m)
   at System.Windows.Forms.Button.WndProc(Message& m)
   at System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message& m)
   at System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)
   at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)
2010-05-09 11:55:19:328 Gui.WorkspaceOptions.ApplicationExit
[/FONT]

[NinjaTrader_Austin]

adamus, I just changed my timezone to London/Dublin (where'd my Sunday go?!?) and tried importing the data and everything worked fine. The only warning I got was that all volumes of 0 were replaced by a volume of 1.

[adamus]

Hi Austin,

you changed your computer timezone setting to London/Dublin and you chose Eastern US/Canada as the timezone in the HDM? Did you also check the create minute bars and create daily bars?

This is really frustrating!

You can of course see that I'm not hallucinating - there's the trace log exception, right?

By the way, great support here - really appreciate it.

[NinjaTrader_Austin]

Ah, with those extra steps the error did indeed occur. I'll send a note along to the developers.

[adamus]

Austin,

just had another crash with another import. This time there wasn't even a trace left in the trace file, so I can't do anything about it as I could earlier when I removed the line which caused the crash.

Can you give me an issue tracking number so i can see whether it gets fixed in the version release reports?

[NinjaTrader_Bertrand]

Thanks for your persistence in reporting and testing, the issue # would be 3734. Please retest on beta 16 - thanks.