NT Reliability

[Harry]

NInjaTrader demons at work

By mistake I tried to load an instrument, for which IB no longer provided data. Result: Complete Crash of NT. Cannot be connected after restart and no charts can be be opened . Needs complete new installation.

This is really bad. If NT does not get the data it destroys everything including itself. I hope that NT 7 is more stable and reliable.

[NinjaTrader_Kyle]

Hi Harry,

Thank you for your reply.

Please send a note to Support [AT] NinjaTrader [DOT] com with "ATTN: Kyle" in the subject line.

In the message, please include the following:
1.) A link to this forum thread.
2.) Your trace file from the date this occurred.
You can find the file here: My Documents > NinjaTrader 6.5 > Trace > trace.YYYYMMDD.txt

[Harry]

NT and IB with limited compatibility

Hi Kyle,

thanks for your answer. Do not need to send any files. Have already checked them. The only useful information comes from the logfile

"16.09.2009 17:57:41|0|2|Error in requesting historial data from IB: No security definition has been found for the request(200)"

The tracefile just stops in the middle of HistoricalDataRequest and is not showing any useful message. Have gone through this many times - my NT installation already crashed a couple of hundred times on backfill problems. Very often. when IB does not send the data NT asks for, NT simply crashes (this could be a time out of IB's backfill server or backfill throttling limitations). Alternatively the famous "Saving" popup will show and tell me that NT has ceased working. As this usually destroys data base and chart templates and/or the NT installation, I am used to reinstalling NT and restore my backup from the previous day.

I think there is also limited compatibility between NT and the API of IB. Since January (9 months) it is not possible to trade energy futures with NT through IB, just beacuse a different definition of "expiry month" is used. NT uses the contract month (October for CL 10-09) and IB the month, during which the contract expires (September for CL 10-09). Isn't this ridiculous?

Such a great piece of software, but so many little nuisances!

[NinjaTrader_Dierk]

On your crash: it certainly would help if you could provide more details like how exactly NT "crashed"
a) was it stuck but responsive?
b) was it stuck but not responsive?
c) did it terminate showing the box "NT has to terminate..."?
d) did it terminate without further notice?
e) else?

On the CL issue: As explained in other threads on that subject IB changed the behavior of their API earlier on this year. IB announced to provide the relevant information (which now suddenly was missing or incorrect) again on a future update of their API. Any API update provided by IB so far unfortunately would not be sufficient to resolve the issue. As explained before that issue is beyond our control. Nothing has changed on that regard. We will keep you and other effected users updated as the issue would be addressed in the IB API and in NT.

Thanks for your understanding

[Harry]

Detailed information on the problem

Thank you for your replying, Dierk.

1. I changed the instrument of one of the 4 connected charts to 6B 06-09 by mistake. Actually I wanted to look at 6B 12-09, but because of the contract merging problems I had temporarily reput 6B 09-09 and 6B 06-09 on the default list, and my mouse was faster than my sight.

2. This triggered a one-lined response by IB (see logfile below).

3. After this, 3 of the 4 connected charts remained empty, the other one was showing old data from 6B 06-09 (which was still in the data base, as NT does not allow to delete it). A detail: the tick chart was also affected, although not connected to the others.

4. After this NT was still running without any popup message (no negative message within the Windows Task Manager either).

5. I then wanted to rebuild the charts, but the destroyed ones could not be closed. So I tried to close NT, which also was not possilble. Actually I had to kill the NinjaTrader process in the Task Manager, because I could not remove the NT windows from the Task Manager applications list.

6. When I restarted NT, the Control Center showed up large in the middle of the screen (usually it is a small frame in the lower left corner of my screen). This already tells me that NT is dead. I then closed the charts, because they did not display anything and tried again to rebuild them via my templates. However it was impossible to create a single new chart. New charts would remain empty and read "Instrument" in the upper left corner instead of showing the name of the instrument.

7. All this tells me to reinstall NinjaTrader, which I did, actually upgrading to 6.5.1000.13.

8. However, I did not have to restore the database. After installing NT my original workspace came up as usual without any problems. My conclusion: NT simply committed suicide, but this time preserving the database.

[NinjaTrader_Dierk]

Thanks for your feedback. Could you please try this:
- save all your workspaces (e.g. by NT backup File->backup)
- close all UIs, just have CC opened
- connect to IB
- now pull up a chart on the offending instrument "6B 06-09"

Does it work? Would it ever come back or does it get stuck in "Loading data"?

Thanks

[Harry]

Unfortunately not reproducible

Dierk, I did what you said. Unfortunately I could not reproduce the error. NT behaved as expected, even with 4 connected charts of 6B 06-09. Alternately displaying "Loading" popup and error message due to non-existence of data, 4 times with some waiting time in between due to IB data throttling.

Is it correct that NT loads intraday data for each chart if the lookback periods of these charts are different? Can't NT choose the chart with the longest lookback period and then fill the other ones? I have a sure feeling that the problem is somehow related to IB's unwillingness to supply data for the same instrument three or four times "de suite". The problems have always occured when backfilling multiple charts with different lookback periods (on a 60 min chart I would need 250 days of intraday data, but I do not want to kill my PC with 250 days of 5 minute data).

One new problem occured afterwards. Could not restore the backup. So I have to reinstall NT once more. Every time I reinstall, the 35 MB of NinjaTrader are downloaded again. This is sometimes a pain, and this evening your servers are particularly slow (I admit this is rarely the case). I would prefer an installation file that I could keep on my PC, just in case. Have to restart my system now to complete installation of NT. If problems persist, I will come back later.

[Harry]

The data base demon strikes again

Today I merged contracts again, I paid attention to everything and all contracts were successfully merged and displayed on my chart, even the daily data.

Now what did the database demon? Look at the chart below: I had merged TF 06-09 into 09-09 at June rollover date, then merged TF 09-09 into 12-09 at September rollover date. But after connecting to IB the middle section of the merged data that originated from the TF 09-09 contract disappeared, while the TF 06-09 and TF 12-06 sections were maintained.

NT can no more surprise me.

[bridenour]

I have been having some of these same issues...in particular, i try to collect real time data but i just randomly get "holes" in my data all of the time (meaning, missing partial or half days). I am literally only monitoring and collecting data for the ES, but it is tick/volume based.

Anyway, I have now started deleted the NT mdb each and every time i start up NinjaTrader....the corruptions are so frequent that its the only thing that seems to, at least, partially work. Sometimes deleting the MDB is not sufficient...today I had to actually reinstall...not sure why.

Anyway, i don't know what conditions cause these problems...I use IQFeed with NT, and all I do is monitor the ES on three different volume bar-charts. Definitely get corruptions daily doing that....

Not expecting an answer, just thought I would relay my experiences.

[NinjaTrader_Dierk]

Guys, I suggest not confusing issues. Please open separate threads on the different issues you experience.

Harry:
- on the merging issue: We never had a report similar to yours. However, we are aware that manual merging in NT6.5 could be cumbersome. That's why NT7 will come with a completely different merging logic where you no longer should need to merge manually. We will provide details as we roll NT7
- on the 6B 06-09 issue: we needed a reproducible scenario to understand if and why IB would not return from requests for historical data which NT sends. Please let us know as you would be able to provide such scenario.
- on the backup: what in particular did you experience? Is the issue reproducible? If so how?

Thanks

[Harry]

Another example

The main problem with the database issues is that they are not reproducible. Today I could not load my workspace, so I had to restore the workspace from two days ago. After restoring I shut down NT as usual, without saving. When I wanted to restart again, it was impossible, because NT was still running in the background (got the message that I cannot run a second instance...). So I had to remove NT . although shut down before - from the Windows Task Manager. Things like this really happen often. I am now going to start the merging of the contracts all over, because my workspace form yesterday is broken.

[NinjaTrader_Dierk]

Sounds like you're having massive trouble. Although definitely other users have DB trouble as well, yours are outstanding. It may sound ridiculous but have you ever considered e.g. trying NT on a different machine with a clean Windows setup, may be as few as possible other applications installed just to eliminate variables.

NT7 definitely will paint a different (and as we believe better) picture, but if the issues are related to your particular machine/setup then the likelihood that NT7 would make a difference could be negligible.

[Harry]

Another timestamp issue detected for TF

I had once reinstalled the machine following your suggestion. The machine is ok, it is a Dell Xeon Workstation with XP Professional. I am running other software, and no problems.

The trouble is not massive. Crashes only occur, when NT locks up on backfill. The merging of intraday data always works. Daily data sometimes does. Today everthing went fine with the exception of FDAX, FESX and TF. I repeated the merging for these three and now everything is ok with exception of TF (see below).

TF seems to be a special case, as the daily bars are not correctly displayed. Yesterday's bar is displayed as today's and today's bar is missing. This is an old bug that had already occured with FOREX and COIL, but has meanwhile been solved for those instruments. The reason: The time stamp of the daily bars of TF by Interactive Brokers is 6 PM ET. If you convert this into my local time (CET), this is exactly midnight. As a consequence NT displays the wrong dates. This issue is clearly related to the session times of TF and the (late) time stamp of the daily bars. It does not affect other futures contracts. So it is similar to the energy futures, where only those are affected where the contract month differs from the expiry month.

On the charts below you can see that for TF NT displays yesterday's daily bar as today's. Maybe this is also related to the trouble in maintaining merged daily data.

[NinjaTrader_Dierk]

>> This is an old bug that had already occured with FOREX and COIL, but has meanwhile been solved for those instruments
To clarify: I believe we NT have not changed /fixed anything in that regard (as we felt it was an issue with IB's data). Are you saying that IB has corrected that issue? If so, then the TF issue actually might be resorted to the same category.

[Harry]

The small difference

Indeed I had opened a ticket with IB and asked them to modify the time stamp, which they did. However, there is one inconvenient. FOREX is now timestamped 5 PM ET (which is 11 PM CET) and COIL is timestamped 10 PM GMT (which is also 11 PM CET). For TF the session only closes at 6 PM ET (which is 12 PM CET), but they cannot give it a time stamp earlier than session close! No way of achieving this.