Charts and Analyzer unreal disconnection issue from Database

[pstrusi]

Hello pstrusi,
Thanks for your reply.
Please go to Tools> Historical Data Manager> Export
Export Last Tick data for the $EURUSD on 4/5/2016
Attach the file to this thread so I can test this further on my end.

The file size is about 15mb and it when I try to upload it gives me an error message that I didn't have a token

[NinjaTrader_Brandon]

Hello pstrusi,
Thanks for your reply.
Please send it using www.wetransfer.com to platformsupport[at]ninjatrader[dot]com
In the message put ATTN BRANDON TICKET 1498374

[pstrusi]

Hello pstrusi,
Thanks for your reply.
Please send it using www.wetransfer.com to platformsupport[at]ninjatrader[dot]com
In the message put ATTN BRANDON TICKET 1498374

Done.

By the way Brandon, how should I proceed to erase the complete last week historical tick data from my database ? Historical Manager only allows to edit tick by tick. I'm interested of erasing the whole last week in the most secure way.

[NinjaTrader_Brandon]

Hello pstrusi,
Thanks for your reply.
We will answer this item in the other forum post you made: http://ninjatrader.com/support/forum...ad.php?t=84721

[pstrusi]

Hello pstrusi,
Thanks for your reply.
Please send it using www.wetransfer.com to platformsupport[at]ninjatrader[dot]com
In the message put ATTN BRANDON TICKET 1498374

Hi Brandon,

I'm repairing manually my historical database as good as I can, I'm more interested in the conclusions of your search about possible causes for this issue so I can prevent them. Don't forget that this mess only happens whenever I've done live simulations with account sim101. The mess in the database is shown after the NT platform is closed and restarted it again.

Thanks in advance

[NinjaTrader_Brandon]

Hello pstrusi,
Thanks for your reply.
I have not been able to find a way to replicate this. In your files, I see a poor connection which could be related, however, no other errors.
I would recommend exiting NinjaTrader and restarting your modem/router.
If you have an antivirus/firewall installed you will want to add exceptions to the following directories.
C:\Program Files (x86)\NinjaTrader 7
C:\Users\piers\Documents\NinjaTrader 7
If you are not plugged in hard wired, do so if possible.

[pstrusi]

Hello pstrusi,
Thanks for your reply.
I have not been able to find a way to replicate this. In your files, I see a poor connection which could be related, however, no other errors.
I would recommend exiting NinjaTrader and restarting your modem/router.
If you have an antivirus/firewall installed you will want to add exceptions to the following directories.
C:\Program Files (x86)\NinjaTrader 7
C:\Users\piers\Documents\NinjaTrader 7
If you are not plugged in hard wired, do so if possible.

Hi Brandon, thanks for your answer. I appreciate your observations, however they don't look like as possible cause of this database mess. Basically because it doesn't happen until I do a live simulation.

- I have my connection to the router by wire, in order to prevent wi-fi issues.
- IB has a brief couple of moments at noon where they reset their servers for 1 or 2 minutes, there's no way to avoid this.
- The antivirus just scans certain type of files real.time, but it doesn't have any task filtering internet connections.
- Obviously I have a FireWall which is correctly set to allow all connection with NT and IB


As I told you before, they key is the afterwards on having done a live simulation, something happens that affects some cache of historical data, that should be the main focus in this search. I really hope that you consider to find out what's happening, cause if this happens to me, there will be more people in the future.

Please let me know any helpful further findings.

Thanks in advance

[NinjaTrader_Brandon]

Hello pstrusi,
Thanks for your reply.
These items have been taken into consideration. If I am able to replicate this and/or find the cause I will update you with my findings. If you find a way to reproduce this, please let us know.
I will continue reviewing your files and testing for a way to replicate this.

[pstrusi]

If you really want to go closer to replicate the issue, take my advice:

- Install NT7 within a Windows 10 64 bits
- Set a connection to IB
- Initiate a live simulation every day.
- At market close do some backtest and save results
- When finish the simulation, let's say for a couple of days, close the whole platform, and restart it again, Once NT is opened again, do the same backtest previous closing the platform. Compare with the last saved backtest. Watch for differences. Watch for database alterations.

If you do this, for a while without having any issue, it will be for me a big problem to solve.

Trying to replicate issues without following the conditions above, will be a different search.

Let me know how it goes.

[pstrusi]

Hello pstrusi,
Thanks for your reply.
I have not been able to find a way to replicate this. In your files, I see a poor connection which could be related, however, no other errors.
I would recommend exiting NinjaTrader and restarting your modem/router.
If you have an antivirus/firewall installed you will want to add exceptions to the following directories.
C:\Program Files (x86)\NinjaTrader 7
C:\Users\piers\Documents\NinjaTrader 7
If you are not plugged in hard wired, do so if possible.

Hi Brandon,

One last thing cause I'm curious. Are you telling me that the data that I've sent you is totally clean? Please go to the historical data manager, select 4-5-2016, and then go to the bracket from 9:00:00am until 10:00:00 and check if you have tick data AFTER 10:00:00am, which it shouldn't be there from that bracket, but yes for the next one, which is from 10am until 11am.

Looking forward what you find out

[NinjaTrader_Brandon]

Hello pstrusi,
The data displays for me the same way as the data displays for you per the screenshot.

[pstrusi]

Ok thanks for confirming that.

[pstrusi]

If you really want to go closer to replicate the issue, take my advice:

- Install NT7 within a Windows 10 64 bits
- Set a connection to IB
- Initiate a live simulation every day.
- At market close do some backtest and save results
- When finish the simulation, let's say for a couple of days, close the whole platform, and restart it again, Once NT is opened again, do the same backtest previous closing the platform. Compare with the last saved backtest. Watch for differences. Watch for database alterations.

If you do this, for a while without having any issue, it will be for me a big problem to solve.

Trying to replicate issues without following the conditions above, will be a different search.

Let me know how it goes.

Hi Brandon.

I hope that this week you're trying to replicate the issue according to the set of conditions above, so eventually you could see the same issue and work that out with the developers team.

Thanks in advance and I'll be waiting for your findings.

[NinjaTrader_Brandon]

Hello pstrusi,
Thanks for your reply.
Yes, I am attempting to replicate this based on this information.

[koganam]

Hello pstrusi,
Thanks for your reply.
Yes, I am attempting to replicate this based on this information.

Of course you will generate it, and it does not even need all that elaborate method. The issue is a pretty simple and long-standing issue with lost information. ALL snapshot data feeds are ipso facto losing information in the transmission. The net effect is that the Historical data will almost always contain more information than the live data, so the charts are very unlikely to look the same.

This has been discussed and pointed out for many years, both on this forum and other places. Here is one such discussion, and the very first post demonstrates the issue, as refreshing the chart loads up historical backfilled data, any recording settings in NT notwithstanding.

ref: http://www.elitetrader.com/et/index....uality.233301/

It is a standard tenet of information processing. Two data sources that are not identical will almost always produce different information from identical analysis. In this case, the historical information reloaded, is not the same as the live data, as the live data is snapshot, and the historical data is backfilled. Ergo, different data, different results.

This has nothing to do with what data recording settings are being used. If the data gets back-filled, it will be different from any recorded, or otherwise, snapshot data.The poster has already questioned the presence of extra data. By definition, such extra data cannot not be there if the only data there was the recorded data from the live snapshot data.