Data price disruption that makes no sense

[NinjaTrader_PatrickH]

The data is either being removed or was never saved. Check your PC once again for any programs that may remove files (such as CCleaner) or any programs that would prevent connections. Also verify in the NinjaTrader Log that there are no disconnects.

[pstrusi]

I don't have Ccleaner or even do any cache or files remove procedure at all. I have selected in NT "Save chart data as historical " then if I see the data until 5pm, it should be in the database saved, in fact, this issue just happens with data on Fridays, when I close all programs and turn my computer off.

I've already told this, but since it's very noticeable, I repeat it: Last Friday I did a backup with all data completely loaded till 5pm, when the issue happened , I decided to restore that backup with the whole data, and the surprise was that even after the restore the missing data continues to be, how's that possible? That must be something else that is affecting the database.

Thanks for these suggestions, but still the issue persists

[pstrusi]

After much research, I have just one possible suspect. I connect to IB through the Gateway of version TWS 944.3 ( approved by you ) , BUT despite I see data loaded into my charts perfectly normal, something happens eventually that messes with the database. I don't still know what could it be. This is beginning to make me feel very frustrated and angry. I have all my platforms in order, everything connecting and loading well but still the database suffers a mess eventually without any warning.

1. Is it because the connection through Gateway triggers some instruction disabling the NT database?

2. Could it be something wrong in the SQL in my pc ?

3. I have set all clocks and time sessions to E.T ( UTC-5hrs ) but I live in europe, is it possible some unthinkable interference with internal NT database ?

My next steps:
1. A totally fresh reinstall of NT
2. Connecting through the standard TWS 944.3 ( instead of its gateway ) for a week and watch if the issue still persists.

Please pay attention to this problem, and help me out here, I'm struggling here with my limited knowledge in this platform.

[NinjaTrader_PatrickH]

pstrusi,

TWS 944.3 is supported in NinjaTrader 7, not the Gateway API. The Gateway API is supported in NinjaTrader 8.

1. I am not sure as the Gateway API is not supported in NinjaTrader 7 and thus any behavior is unexpected.

2. Potentially, but I cannot say for certain. You could attempt to re-install SQL.

3. No, the databases would use your PC time. The physical time zone plays no part.


If you have another PC that you can test NinjaTrader on, I would do so just to confirm if it is something with the PC specifically.

[pstrusi]

Hi Patrick, thanks for your attention and help. Since I've updated to windows 10, changing some pc's configurations might create some problems with SQL, some I decided to reinstall Windows 10 from zero, hoping not to see this annoying issue anymore.

[pstrusi]

Hi Patrick.

In order to address this issue, I decided to do a complete reset of my pc:

- Reset Windows 10 and update it
- Fresh install of NT 7.0.1000.30
- Fresh install of Interactivebrokers TWS 944.3 which is the supported by you.

While I sit in front of my pc, I've verified a totally normal data loading into my charts, without any problem or weirds gaps, so I was hoping not to see the issue any longer, but once again, during a backtest, I realize that the issue arise once again, two times of big chunk of data ( around 13 minutes are gone, disappeared from Database) despite I've had a stable and reliable connection, with the right configuration, saving chart data as historical...etc.
Searching carefully for possible cause, I've found something that maybe might give you some hints:

- This issue ONLY happens to $EURUSD ( Forex ) the days when I run "live" simulations with the account Sim101. I've checked the ES chart and it's completely normal. I suspect that when I reset for the account Sim101, his procedure maybe is causing some problem in the database, erasing unwillingly loaded data.

- Most of the time, the data disappearing are for a timespan of 14 minutes, and almost always from XX:45:00 until (XX+1):00:00 , so this proves that the issue comes from some "internal system call" .

- Sometimes there are simply a tremendous price gap within a very quiet time period ( around 15 halfpips ) and continuous as a very quiet data loading but at the new price range, above the original. So it's not cause volatility or market related.

- The day 12-31-2015, I had my system stable, connected; and under a distraction, the auto log-off of Interactivebrokers did the process at 05:10:00 AM ( E.T or UTC-5:00 ) . I reconnected the system once again at 05:22:42 AM, so it would be expected to not see any loaded data during that timespan, well, inexplicably there's data loaded !, from 05:10:00 till 05:19:29, which without any other data source, how is this possible ?; so I begin to suspect that the "Interactive connection configuration" has gotten anything that is causing this issue. I'm suspicious about "Historical Data: Use Interactivebrokers server" Box checked. I think that maybe that configuration is triggering some attempt to download bad data from some server, who knows. Attached to this post, there's a copy of that chart.

Please, I need your help here, because I'm running out of ideas to solve this error. My database is building with these flaws, which make Back tests unreliable.

Thanks in advance

[NinjaTrader_PatrickH]

Hello pstrusi,

Thank you for the additional details.

We will look into this on Windows 10. Do you have another PC you can test this on? And perhaps another Operating System?

[pstrusi]

Hi Pactrick.

I have only this PC, however as you can see, I've been doing more trials and observations which I've posted as HINTS. I believe that for experts like you, this detailed info might hopefully lead to conclusions and the root of the failure.

I suspect strongly, that something happens when you "Reset" the account Sim101 after a live simulation, that even more, it could trigger some weird attempt in the IB connection to download historical data from server, which IB doesn't provide, and this might explain why those big chunks of data disappeared.

Looking forward for your research's conclusions, because this has stopped my heavy work of develop Algos.

Thanks in advance

[NinjaTrader_PatrickH]

pstrusi,

We have been unable to re-create this scenario on our end nor have we ran into such behavior in the past without a connection or database item being the root cause.

[pstrusi]

I see.

At least are you able to answer why this ? ( it was in an earlier post of this thread )

- The day 12-31-2015, I had my system stable, connected; and under a distraction, the auto log-off of Interactivebrokers did the process at 05:10:00 AM ( E.T or UTC-5:00 ) . I reconnected the system once again at 05:22:42 AM, so it would be expected to not see any loaded data during that timespan, well, inexplicably there's data loaded !, from 05:10:00 till 05:19:29, which without any other data source, how is this possible ?; so I begin to suspect that the "Interactive connection configuration" has gotten anything that is causing this issue. I'm suspicious about "Historical Data: Use Interactivebrokers server" Box checked. I think that maybe that configuration is triggering some attempt to download bad data from some server, who knows

How a chart was populated with data when NT was without any data source at that time?

[NinjaTrader_PatrickH]

What instrument and period type was that chart?

[pstrusi]

EURUSD at 1 tick to watch it better, that very same info is in the chart

[NinjaTrader_PatrickH]

Do you normally see a gap in data at those times or is it always filled in?

[pstrusi]

Patrick my system is on 24/7, so normally is filled up when data connection of course

[NinjaTrader_PatrickH]

During the nightly maintenance and the force disconnect you would see a gap in data, no?
To once again confirm, you are using TWS and NOT the IB Gateway? Meaning you have to start TWS in order to connect, correct?