I'm currently in the process of importing 10+ years of historical tick data for multiple futures instruments.
Occasionally, there are some corrupted records in the dataset, like invalid bid/ask values or incorrect timestamps.
I want to make sure the data is as clean as possible, so I'm going over the import error messages I get in NT and then adjust the repsective records in the dataset.
To ensure data integrity when reimporting the dataset I have a couple of questions:
1. How does NT handle corrupted ticks on import? So, for example, when I get an error message "... Illegal bid value in line ...", does NT just not import the tick and continue with the next row in the import file?
2. How does NT handle existing tick data? When importing tick data that's already (at least partially) existing in my db, what's NTs policy to deal with that?
A good part of the dataset has been imported successfully already. Now I want to deal with the part that's missing due to errors. However, importing that amount of data takes a while on my machine, so I'm wondering how I should go about the corrpted ticks:
- adjust the corrupted records in the dataset, reimport it and let NT deal with the part that's already present in my db from the prior attempt
- or delete all previously imported data and do a full new import
What's the better option?
Thanks!
