Market Analyzer slow

[arnonmoscona]

Hi!

Let me preface by saying a few things to bear in mind before anything else:

• I know that I operate MA far outside what it normally gets, but that's what I need for my trading strategy
• I am a very experienced computer professional

Now to some other operating parameters:

• My machine has 4 cores (i7) and 24GB RAM and drive C: is a regular disk (not SSD). Running Win 7 Pro
• I use IQFeed as a data source
• I load up MA with about 500 instruments - all the constituents of the S&P 500
• I have 5 indicators loaded (all of them custom)
• All indicators are configured to calculate on bar close (not tick)
• The indicators look generally back between 15 and 120 bars, with max look back on one of them set to 360
• I have 3 charts open all of them linked to the selected instrument on the MA and each with a bunch of indicators - all of them 1min charts
• Importantly, NT is configured to save chart data as historical as well as to record for market replay
• I've been using IQFeed with a custom Java application for the last two years for the same instruments, and was logging the data stream while running indicator logic in the application. The system had no problem keeping up with the data and was sustaining 120ms latency between transaction time and write time. I was using my own storage format, but clearly it shows that dealing with the amount of data should not be a problem for a system that is designed for this number of instruments (same system, same network, same everything - just a different application)

And some symptoms:

• Indicators on the MA can be up to 5 minutes behind real time
• Charts seem to be updating OK
• CPU utilization is pretty low (never above 20%)
• IO on the C drive seems to peg often due to seek time (not throughput) with the files experiencing the worst response time all being in the NT DB. This high IO is only if the MA is open. If I close it then it calms down (not surprising). It appears to be tick data that's getting write delays most of the time.
• NT as a whole (the whole application) would enter a semi-frozen state every once in a while (maybe once an hour or 2) but remains generally responsive most of the time. The problem seems to be affecting almost only the MA

I would venture to guess that the MA is affected directly due to saving chart data as historical (have not experimented yet with turning that off for reasons I will explain). This would imply that the data saving (especially ticks) is done in the same thread as the indicators, thus delaying their execution. The system is clearly IO bound, and more specifically bound by disk latency and not throughput.


Now, in parallel to this, I've been having a different problem with the historical data manager. It has not been stable downloading data for the same list of instruments and hangs entirely quite often (sometimes hanging the entire NT process). For the data manager, I get no problems if I download only minute and daily data. But if I ask for ticks or I ask for ask/bid data then it almost never manages to complete the download. This too, is something I do regularly for my Java app with no problem (I backfill my DB on a tick level from IQFeed to my custom DB and I have no problem). So I am really relying on MA to collect the data for me as i cannot rely on the historical data download. I need the historical data for backtesting (market replay is not very important to me actually, but strategy testing is very important).


So I would assert that I have no problem with the machine, network, or data provider - as all of these are eliminated by the trouble free Java implementation. I would concede that it is likely that switching to an SSD for storage may solve the problem (as it is a seek time problem) - but that would be both painful and expensive.



So after this long introduction, here are my questions:

1. Given that I do want to record the data for strategy backtesting purposes, in the options/data tab what is safe to turn off and would it have any effect? (e.g. turn off record for market replay and leave save chart data as historical on).
2. Is there any plan for making MA more scalable than it is in a future version (e.g. do these writes in a background thread, or use a less seek heavy storage format)
3. Is the problem with the historical data manager downloads familiar to you? If so, is there a solution for that or alternatively, can I do strategy back testing for periods that have only minute data and no tick data?

Thanks
Arnon

[NinjaTrader_Jason]

Hello Arnon,

1. Yes, it will improve performance if you disable 'Run market replay recorder'. If your data feed provider supports historical tick data, we recommend to leave 'Save chart data as historical' disabled as well. These suggestions are listed in the performance link below under 'Miscellaneous'.

http://www.ninjatrader.com/support/helpGuides/nt7/performance_tips2.htm

Downloading historical tick data for multiple instruments at once can cause freezing behavior. Tick data consists of much more data than minute or daily data when downloading. I suggest to download one instrument at a time or perhaps split your instrument list and download the data in two times. Please note that all historical data loaded in a chart will be automatically saved in NinjaTrader. You can view all historical data you have present in NinjaTrader under the Edit-tab of the Historical Data Manager.

2. I am not aware of such changes, however I will forward your feedback to development to put it on the list of possible future enhancements.

3. If you perform a backtest, it will automatically download the data you need (you will need to be connected to your data feed). No need to download the data first in the Historical Data Manager.

[arnonmoscona]

Thanks for the response. However, your suggestions seem to make matters worse, not better. I tried turning off the two data saving features - both recording for market replay and save chart data as historical. I restarted NT, and now it basically goes into hangup, as soon as it starts. It appears that as soon as a message in the status bar shows "loading data for ..." for any of the stocks in the market analyzer, the whole UI becomes unresponsive.

It is so bad in fact that the only way I can terminate the process is from the task manager. Since it tries to load my workspace on startup, I cannot even turn those two features (or one of them) on any longer as the UI is unresponsive from the start. So now I am cornered. NT does not work at all.

How do I back out from this situation?

[NinjaTrader_Jason]

Hello Arnon,

Can you please go to (My) Documents-->NinjaTrader 7-->workspaces and transfer the workspace files to a temporary different location. Start NinjaTrader. It does not have to load data anymore and you can make the changes you like. Exit NinjaTrader and transfer back the workspace files.

Am I correct you have your connection configured to connect on startup? How many workspaces do you load on startup as well? If you keep experiencing issues during startup, I suggest to disable to 'Connect on startup' option and have NinjaTrader only open 1 workspace.

Can you please go to Control Center-->Tools-->Account Connections, highlight your connection and click Change. Disable 'Connect on startup' and finish the connection.

Please go to Control Center-->File-->Workspaces. Check what workspaces are listed below 'Save Workspace As'. The workspace with the checkmark is the workspace opened on your monitor. Any additional workspaces listed, are opened in the background. Please select a workspace without the checkmark. Subsequently, go to Control Center-->File-->Workspaces-->Close Workspace. Repeat this procedure until only one workspace is opened.

Restart NinjaTrader and check if it starts without issue.

[arnonmoscona]

OK. I Restored functionality by following the instructions in another post:

• Kill NT
• Go to Documents\NinjaTrader 7\workspaces
• Move my workspace to a temp directory
• Restart NT, turned save chart data as historical back on (left market replay off)
• exit NT
• restore my workspaces file
• Restart NT

The result is that NT now works.

• I/O utilization is lower, but at the end the disk is still pegged with the same problem, and still many files that get bad response time as reported by the resource monitor are tick files for stocks that are in the MA
• However, the indicator values in MA are now actually worse than before. While before they were a few minutes behind - now they give plainly wrong numbers. When I open a chart with the same indicators on it, the chart shows very different results - and in most cases does not show the numbers that are in MA as having been there at any time today. For instance an indicator that on MA shows 56, on the chart never shows more than 23 (with the same parameters)

[arnonmoscona]

Sorry - got this reply while I was writing the other reply... See the problem there with indicator values

[arnonmoscona]

Note also that all my indicators in the MA are set to calculate on bar close, so I am not sure why I have any tick data being recorded at all.
Also note that the same indicators on the chart are set to calculate on bar close. So i have no explanation as to how they end up with values on the MA that they never get on the chart...

[NinjaTrader_Brett]

Hello,

One thing I would like to try. Remove all indicators from the MA and charts. as a quick test.

Do you have the issue even with all custom calculations now removed?

Meaning does the Last price reflect correctly in the MA as it does in the chart.

-Brett

[arnonmoscona]

Thanks Brett, this is an interesting experiment. It also alerted me to some additional facts.

So, the first thing I noticed while doing all this (BTW using a new instance of NT for each experiment to make sure that we don't get "cross contamination of results") is:

* The process that is getting long response time is mostly not NT itself, but the system process.
* The files that are getting the most traffic are the master file table and the NTFS volume log
* The first thing I suspected was that there is an antivirus program active (this will kill I/O on any system with very active I/O). Verified that MS Security Ess. does not have real time protection on (there is no other anti virus installed on this system)
* I also verified that I have no real time backup enabled

So on to the experiment you suggested. The setup is one chart open with no indicators, plus MA open with no indicators but my default instrument list loaded (500 of them). It has the 4 default columns on it (instrument,ask,bid,price)

I/O utilization is lower than my working workspace, but after a while the disk still gets pegged on I/O waits - maybe not quite as badly as my working workspace. The files getting the long waits are now mostly teh ask/bid files. So removed these two columns from the MA to see what happens.

Note that I wait each time I shut down NT until the I/O calms down (as it is the system process pegging the I/O for some unknown reason, it takes a bit of time until that waits stop pegging the disk).

So - if I remove the ask/bid columns from MA (leaving only last price) then I still get I/O pegging after a little while and it's still the ask/bid files that are affected. Apparently NT records those even if i don't show the data anywhere (chart trader is off and the ladder is also not open).

So much for I/O - but mind you, I/O is not my biggest issue here (it seems) as the system as a whole is responsive and even though the disk shows that it's pegged on waits (not on throughput) things seem to function OK. In the worst case, this I/O problem could be solved brute force by using and SSD (espenssive, but can be done)

My biggest problem is that I get indicators that show different numbers on the MA than they do on the chart.

I actually have to leave now and will have to continue my experimentation tomorrow.
FYI - my plan is as follows:
* Start from a "clean slate" - a bare bones workspace (not even last price on MA)
* Build up my workspace by adding one indicator at a time, until the problem is reproduced
* Thereby I expect to get the minimal workspace that reproduces the problem and report it back to you

In the mean while it would be helpful if you can make some comments on the I/O findings from using the bare bones workspace...

Thanks

[NinjaTrader_Brett]

Hello,

I could agree with your I/O finding.

As far as the BID ask this is expected.

There is one other thing I'm worried about and that is with Indicators OnBarUpdate for all indicators will run in a single thread. So depending on indicator complexity with an extreme load this could make NT unable to calculate fast enough and then unable to keep up with the data feed and cause lags and delay.

It is possible its an I/O limitation or a CPU limitation. You are for sure at a limitation with NinjaTrader however with hardware specs and NinjaTrader coding practices.

We are looking in our next major release of NinjaTrader to bring in more multi threading support to take advantage of multiple CPU's. However this is still sometime down the road and I could not confirm any feature or give a estimated release date.

-Brett

-Brett

[arnonmoscona]

Thanks for the info. The fact that all indicators in the MA run in a single thread is quite useful to know.
Is there information anywhere about overall thread allocation in the system (e.g. does each chart run in a separate thread. Is each indicator NT generated code thread safe from other instances of the class (obviously my own code can introduce its own problems - so I am just asking about the generated code and internal supporting structures such as caching).

Anyway - this morning I've been rebuilding my environment slowly, one item at a time working towards reproducing the problem. So far I have almost everything like I had yesterday, except that I did not include one indicator, which has the heaviest calculations and is also a multi-instrument indicator (on top of that it is one that I did not write and did not come from you folks - so I am not very familiar with its implementation and so decided to keep away from it for this debugging activity).

So far no very severe problem has been reproduced this way. The only thing for today is that I do display a total line for two of my indicators in the MA, and there does seem to be a lag in the total display. It does not seem to be a huge lag, but when the market is moving fast - it can be quite pronounced (although it may not seem like that it did today on the market charts, they were short bursts of high activity, and I am not talking about the early morning, where generally things move very fast)

I will switch back and forth between my normal workspace and this debug workspace to see if I can reproduce this problem as strongly as it showed yesterday.

One thing I did not realize at an early point yesterday is that when you switch workspaces, the other workspace remains active - just not displayed. You must actually close it for it not to be active. This made a big difference in the I/O observations (the ones in the thread are mostly after I realized that)

[NinjaTrader_Brett]

Sounds like you are on the right path now.

Let me know if any questions.

[ccl3093]

arnonmoscona:

I am very curious about what you were doing.

Are you still adding 500 symbols on ONE MA, and having any performance issue?

Do you send order from any of those custom indicators on the MA for trading, or just screening?



Thank you for sharing your experience advance.

ccl3093