Error when using import/export to move data across PCs

[bluelou]

Ryan,
All of the tick data over the past 12+ months has been recorded live. Since the problem with the data set occurs after 10/1/10 I can say for sure that that data was recorded in real time. However here's how I have it set up in NinjaTrader: in 'Connection Options' Gain is the provider and Historical data is from NinjaTrader servers. Let me know if this is clear.

~Lou

Hi Bluelou,

The data has these timestamps, so we need to look at source of data and possibly changing export routine to accommodate these timestamps.

The data may come from one of these sources. Can you please let us know which.

• Data recorded from real time Gain stream.
• Historical Data provided by Gain.
• Historical Data provided by NT.

You can check your account connections to determine what you're using. Click Tools > Account Connection, and verify the historical data setting for your Gain connection.

[NinjaTrader_RyanM1]

Thanks, Lou. Our development team has looked into this and this is expected with your data set. Your local PC clock is likely not in sync with the NT servers, and this can lead to issues with the order of ticks. This is unfortunate drawback when using data that does not time stamp natively and relies on PC clock.

[bluelou]

Ryan,
That NT expects my data set to be corrupted isn't very helpful. This was not an issue until Oct 2010 but now it is. How do I stop the time stamps from being out of sequence?

Following up on your comments:
1) Why is the data usable for backtesting and plotting while it resides on the PC that it was collected on and then only appears out of sequence upon export?
2) How do I get my local PC clock in sync w/the NT servers?
3) How do I get my data to time stamp natively?

4) So, are you essentially telling me that it's pointless to use NT to collect real-time data since it can't be used? Do you recommend another way to collect real-time data so that it can be used in NT?

Thanks, Lou. Our development team has looked into this and this is expected with your data set. Your local PC clock is likely not in sync with the NT servers, and this can lead to issues with the order of ticks. This is unfortunate drawback when using data that does not time stamp natively and relies on PC clock.

[NinjaTrader_RyanM1]

You can use 3rd party tools to sync your pc clock regularly but this does not address the underlying issue.

Yes, as this case demonstrates you are best off with a data only provider for the most accurate historical data. This will ensure your data is time stamped natively. We recommended kinetick for this.

If you want access to real time recorded data, best is to run the recorder on each machine that you want to record. As you've seen, the data is available internally for the program and the problem is seen when trying to export.

[bluelou]

You can use 3rd party tools to sync your pc clock regularly but this does not address the underlying issue.

Yes, as this case demonstrates you are best off with a data only provider for the most accurate historical data. This will ensure your data is time stamped natively. We recommended kinetick for this.

If you want access to real time recorded data, best is to run the recorder on each machine that you want to record. As you've seen, the data is available internally for the program and the problem is seen when trying to export.

Ryan,
No, Kinetick isn't an option for me. Gain is a dealer feed - this isn't a consolidated feed or an exchange feed. I have to use their dealing rates. It doesn't seem like NT should offer Gain connectivity if this is a known issue w/dealer data. Or, at least there should be a warning for NT users if they use forex dealer data.

I still don't understand why the data works on the PC that it was recorded on but becomes out of sequence when exported. Can you explain why the data can be used on the PC it was recorded on?

What are the examples of 3rd party tools to sync the pc clock? Are you talking about syncing the pc clock to atomic time or to the time on NT's servers? I'm only seeing some really amateurish retail software on google. Any ideas?

[NinjaTrader_RyanM1]

Thanks. I will forward your comments and concerns to our development team.

Yes, I was referring to the type of program that syncs your PC clock with an external source throughout the day. Unfortunately there is nothing that will sync exactly with NT servers, but this should be close.

NinjaTrader's handling of data with an incoming stream and within the platform for use in Backtesting is a separate process than export / import. It is more forgiving than the structure that importing files requires.

[bluelou]

Okay. I get it. Thanks for your effort Ryan.

[NinjaTrader_RyanM1]

No problem, Lou. Thanks again for sending all the files needed to look into this.

[bluelou]

Ryan,
I don't think your conclusion that the exported tick data is corrupted due to asynchronous PC clocks is holding up. Recall that I sent you 3 FX pairs and that one of them, AUDUSD, had no issues when exporting/importing tick data. Also, I just exported EURJPY tick data over the same time periods in question from one PC and imported on another PC - again, no problem.

So, it appears that the exported tick data time stamps are out of order only for certain instruments. If the PC clock was the issue wouldn't all instruments (collected from the same provider at the same time) have time stamps out of order?

I'd like to fix whatever is causing the problem since it appears that using NT to collect historical tick data is turning out to be somewhat of a crapshoot.

[NinjaTrader_RyanM1]

Lou,

I submitted your files to development, and they saw that the ticks were already there and not introduced by NinjaTrader.

The issue is with your data files, and not something added during the export process.

Time stamps are a likely culprit since it is difficult to keep consistent when not stamped directly by your provider. Your local machine could have any number of issues during the recording process that can lead to time stamps getting out of sync.

[bluelou]

Lou,

I submitted your files to development, and they saw that the ticks were already there and not introduced by NinjaTrader.

The issue is with your data files, and not something added during the export process.

Time stamps are a likely culprit since it is difficult to keep consistent when not stamped directly by your provider. Your local machine could have any number of issues during the recording process that can lead to time stamps getting out of sync.

Ryan,
Let's pick this apart so I can figure out what's going on. I really need for NT to collect tick data w/o incident as it did prior to 10/2010.

1) What does it mean that "the ticks were already there and not introduced by NinjaTrader"? I use NT to trade real-time and to collect the data that I've traded on. Gain is the data provider and NT servers are used for historical data if needed.

2) You state that the issue is w/my data files and not something "added" during the export process. What do I do with this? I thought this was the starting point. We already know that there's a problem with SOME data files. The key question is WHAT is causing the problem and how do we prevent it. You can't rule out that this may be a problem with NT and it doesn't seem like you've seriously considered this possibility.

3) You state that time stamps are a "likely culprit". If that's true then why are the tick time stamps out of order for only some instruments and not all? Why did this problem not exist at all for data collected prior to 10/2010?

4) I'm only bringing this topic up again b/c the 'customer PC makes bad time stamps' argument doesn't seem to hold up when considering that the time stamps are out of sequence ONLY for certain instruments and that the problem ONLY started around 10/2010 (and not for the prior 12+ months that I had been recording data).

If I had to guess I'd say that something happened w/one of the NT7 upgrades and that the bug is probably still there since some instruments are totally corrupted for every day since 10/2010 and other instruments are not corrupted at all.

This now puts me in the position where I can no longer rely on NT for tick data collection. This is a disappointing loss of functionality b/c I had been using NT to collect Gain tick data for 18+ months w/o issue until recently.

[NinjaTrader_JoshP]

bluelou,

I am having development take a second look at this case. We will come back to you again some time tomorrow. Thank you for your patience.

[bluelou]

Thanks for your attention Josh. As you'll see from the data files I sent to NT support, 2 of the pairs, EURUSD and GBPUSD, have ticks w/time stamps out of sequence upon export. However, another pair, AUDUSD, is in sequence and has no errors. I can confirm from the 12+ pairs that I collect tick data on that some pairs are corrupted from 10/2010 on and some pairs are not.

It seems unlikely that my PC would OCCASIONALLY corrupt SOME files but only after 10/2010. That's the reason for my persistence on the subject. I'm thinking that there could be something happening w/a recent version of NT7 or with the NT-Gain connection.

[NinjaTrader_JoshP]

bluelou,

Development has spotted a gray area location which would require longer term testing. Will come back to you when there is more information to report after those tests are complete.

[bluelou]

Josh,
Could you give me a little more information? What is the nature of the gray area? When do you anticipate getting back to me regarding the longer-term testing?

FYI, I've tested a total of 6 pairs now and 3 of them have tick time stamps out of sequence and 3 are okay. My "test" consists of exporting the NT tick data from one PC and importing it into NT on a 2nd PC. For the 3 pairs w/stamps out of sequence the data after 10/17/10 cannot be imported. This problem doesn't exist for the other 3 pairs.

Thanks,
Lou

bluelou,

Development has spotted a gray area location which would require longer term testing. Will come back to you when there is more information to report after those tests are complete.