eSignal versus OpenTick datafeed

[pavelevap]

Hello, I wanted to compare ER2 symbol from eSignal and OpenTick.
So, for current symbol ER2 (default in NT) I removed mapping for OpenTick and created new symbol ER2OPEN which I mapped only with OpenTick.

It looks like datafeeds are the same, except problems with more (and duplicate ticks in OpenTick). In OpenTick chart there are all ticks like in eSignal, but for every timestamp there is also their count. But it looks that it is only problem of NT and not OpenTick

eSignal:
20080416 000004;697.1;5
20080416 000309;697.1;1
20080416 000326;696.9;1
20080416 000326;696.9;4

OpenTick:
20080416 000004;697.1;5
20080416 000004;697.1;5
20080416 000309;697.1;1
20080416 000309;697.1;1
20080416 000326;696.9;1
20080416 000326;696.9;4
20080416 000326;696.9;5

So for timestamp 20080416 000309 there is one tick (volume = 1) in eSignal. But in OpenTick there are two ticks. Also for timestamp 20080416 000326 there are two real ticks (with volumes 1 and 4), but in OpenTick there is also third tick with count volume value (so there is volume 1 + 4 + 5).

As you can see on attached screenshot (Range 10 Chart), there are doubled volumes from OpenTick. Source data directly from OpenTick (without using NT) do not have these counted values...

Thank you for your help...

[NinjaTrader_Dierk]

Provided you are in latest NT6.5.1000.1: Sorry, there is nothing what we can go, since OT explicitly tells on when a new tick comes in.

However, I will notify OT support about the oddity.

[NinjaTrader_Dierk]

How did you create this output?

OpenTick:
20080416 000004;697.1;5
20080416 000004;697.1;5
20080416 000309;697.1;1
20080416 000309;697.1;1
20080416 000326;696.9;1
20080416 000326;696.9;4
20080416 000326;696.9;5

[pavelevap]

Yes, I have latest version NT6.5.1000.1.
This output is Historical data - Export - my symbol ER2OPEN (only for OpenTick) and specify the same day as for eSignal ER2 symbol...

I thought that it might be problem in NT because I also get data from OpenTick in Ensign Software and these odd ticks were not here. It looks like that for every unique timestamp is count of related ticks (volume) added as a new tick...

[NinjaTrader_Dierk]

- I verified with OT support that NT coding logic is correct
- I recorded a few minutes of ES 06-08 (standard instrument setup as provided by NT default DB, no change in mapping) chart data and exported it
- I verified visually on T&S window

Everything works as expected, no duplicates

[pavelevap]

New problem appears:
I removed my new symbol ER2OPEN and add back OpenTick mapping for default symbol settings ER2.
So, it should be default situation now.
But the problem persists.
So when I usually use eSignal as my default datafeed, everything is OK.
But when I Reload tick data (I use 1 Tick Chart for better comparison) from other (secondary) datasource (OpenTick), duplicate ticks are here again.
Example: So, I have stored eSignal data in ER2 symbol for 2008/04/21.
I display them on Chart (Tick 1, start and end date is 2008/04/21, everything is right), then I connect to OpenTick and Reload data.
And duplicate ticks appear.
I think, that is any bug in NT, because there should be possible handling datafeeds for one symbol.
And now also my default symbol ER2 has broken (duplicate) ticks...

[NinjaTrader_Dierk]

I suggest trying on standard setup by using a fresh DB with unmondified symbol mappings. Please delete your DB file <my documents>/NinjaTrader6.5/db/NinjaTrader.mdb.

Note: You will loose all your trade data and historical data stored so far.

[pavelevap]

OK, I deleted DB, but problem persists.
I made some more tests for default ER2 ##-## (or ER2 06-08) and there are still problems in ticks from OpenTick.
You can try it, for example for date 21/4/2008. Open 1 Tick Chart for this date for one connection (for example eSignal). Then export these tick data. Reload the same symbol and date for other connection (OpenTick) and then also export tick data. Compare these two ASCII files.
But it is not neccessary to reload data, you can simply download only ticks from Opentick. Export them and look at them carefully. There are many duplicities from OpenTick (for every unique timestamp). But I do not know if it is problem directly from OpenTick, or some kind of bug inside NT. I use free OpenTick historical data (not realtime).

Also when you tried to compare tested symbol ES 06-08 (for 21/4/2008, historical data) from eSignal and OpenTick, you will see many duplicate ticks in OpenTick datafeed. I did not test realtime (or delayed) data, because I use OpenTick only for historical free data.

Thank you for your help.

[NinjaTrader_Dierk]

Sorry, I'm out of ideas. As pointed out below everything works as expected. Unfortunately there is nothing we can do.

Please contact OT support if you feel there are issues with their historical data.

Thanks for your understanding.

[syswizard]

OK, I deleted DB, but problem persists.
I made some more tests for default ER2 ##-## (or ER2 06-08) and there are still problems in ticks from OpenTick.
There are many duplicities from OpenTick (for every unique timestamp). But I do not know if it is problem directly from OpenTick, or some kind of bug inside NT. I use free OpenTick historical data (not realtime).

I used the Histcsv.exe utility and noticed the same. It appears that level I data is not being filtered properly...nor is there a command line option to do so. I had as a project to refactor this utility. What must happen here is the zero-volume entries must be removed. I pulled down just 1 day of tick data for the YM contract and there were over 100,000 records.

[pavelevap]

But problem is only in NT.
I tested the same with Ensign software and there is no similar problem.
Also HistCSV from OpenTick gives me right ticks, so there is no need to contact OpenTick, I guess.
Only problem is in NT with ticks data from OpenTick.
I tried also for example changing regional settings to US, but it did not help.

I am also out of ideas, where could be the problem...

[NinjaTrader_Dierk]

Guys,

After further consulting with OT support we added a filter to the historical tick data which should address your issues below. This code change will be available with next update (likely next week).

Thanks