NT7 memory consumption

[NinjaTrader_Dierk]

We applied several optimizations with NT7. However the benchmark covers standard C# stuff which NT has no impact on. Please contact the author for details.

[eDanny]

Ok thanks Dierk, that makes sense.

[NinjaTrader_Ray]

Hi All,

We have been working on a "caching" logic has drastic performance improvements when working with bar series. You will notice serious improvement in areas like:

- NT start up
- Backtesting/optimization
- Working with charts etc...

What NT7 will do is this:

- You request a 10 tick chart
- NT aggregates 1 tick data into 10 tick bars and write this to your disk
- Next time you request 10 tick series, its 10x faster since it does not have to re-construct the bars.

The following video is a comparison of performance on a backtest.

It executes a backtest on about 10 million ticks.

http://screencast.com/t/ID0EbV19o

[whitmark]

Cool Cache

Thanks, Ray, for the update and the video is a nice touch. Yes, it is a serious improvement. If you care to, can you address the following questions on this topic:

1) What is the persistence of the cached data? Is it only for this instance of the Strategy Analyzer?

2) How might the caching work when using the walk forward optimizer? Would it attempt to cache the entire data range up front or would it cache the data incrementally as it walks forward.

Thanks!

Whitmark

[NinjaTrader_Dierk]

1) Available whenever bars series are built. The cached data then would be available throughout NT.
2) Really does not matter, since cached data is either used (when available) or build (when not available) whenever bars are requested (no matter when that actually would be) = transparent cache

[Gains]

Excellent communications

Thanks a lot Ray for keeping us updated like this! It is very much appreciated.

Gains

[syswizard]

1) Available whenever bars series are built. The cached data then would be available throughout NT.
2) Really does not matter, since cached data is either used (when available) or build (when not available) whenever bars are requested (no matter when that actually would be) = transparent cache

Will NT7's caching accomodate the situation where the 10 tick bars were built say 5 days ago, but now need refreshed with the most recent 5 days of tick history ? Or will it rebuild the entire series all over again from the requested start date ?

[NinjaTrader_Dierk]

>> but now need refreshed with the most recent 5 days of tick history
correct

[KrisS]

We applied several optimizations with NT7. However the benchmark covers standard C# stuff which NT has no impact on.

NT Dierk,

Is this to say that NT7 will not run this benchmark with significantly improved hyperthreading and multi-core support because the benchmark uses C#?

I would have expected a significant improvement in calculation time based on the multi-core support in .NET 3.5

[NinjaTrader_Dierk]

You need to contact the author in order to understand what this benchmark actually measures and what not.

[zoltran]

NT Dierk,

Is this to say that NT7 will not run this benchmark with significantly improved hyperthreading and multi-core support because the benchmark uses C#?

I would have expected a significant improvement in calculation time based on the multi-core support in .NET 3.5

Hi Kriss
Just a general observation... Using multiple 'cores' or CPU's is a pretty complicated topic.
Many people think it's just a matter of using a new library, or 'dotNet' or OS, but it really is a lot more complicated than that.
Programs in a multitasking OS like Windows break work into many threads or processes. All else being equal, Windows can assign each thread to a 2nd parallel core. But this is not as easy as it sounds, as many things need to be done serially vs parallel.
For example, the calculations in an indicator/strategy need to start from the begining and progress to the end..., and usually require information from the prior bar.
Take a simple moving average over a 400 bar chart .. Splitting that calc into 4 chunks or threads of 100 bars isn't feasible, as each 100 bar chunk needs the value of the all the prior chunks. Plus, there's the overhead of coordinating all these tasks... I used to work on a large transaction processing system and we found the this inter-task/thread co-ordination often stymied the benefits of running multiple threads.

This is one of the reasons why massively parallel super-systems with 10's to 1000's of cpu's are not in common practice.

So in practice, it's unlikely that any given single task, such as an individual chart window will benefit from a multi-core system.

But ... it should be possible to design a system such that each chart or task does have it's own thread, so if you run multiple charts it could be spread over multiple cores.

The NT team has also said they have made multi-tasking improvements in the optimizer, That sounds like some pretty sophisticated programming

[kman1]

Im wondering why my Murrey Math indicator does not work with Range charts. Only happens on NT. Works fine on other programs. Any help would be appreciated

[NinjaTrader_Kyle]

Hi kman1,

Thank you for your post.

I suggest contacting the indicators author for more information.

[djohnsonhot]

Resources

It appears that this may be a good forum on system resource improvements...those that will determine whether I buy NT 7.0 Lifetime licenses or not. Although NT is far superior in many ways compared to other platforms, as it stands today, I cannot use NT without memory dumps, crashes, lost connections and so on. Since I tried using NT as a reference platform, naturally I use my feed and place sim trades. Yesterday, I entered a trade and it took over 8 seconds to appear (right click on chart, Lady-voice confirmation, trade entry line appeared)...that is just ridiculous. Furthermore, the loading times are unreal. I will wait over a minute to fully paint the static workspace, and THEN click connect, and I wait another 30 seconds to a minute for bars to appear. I am using ThinkorSwim's platform and I need to say this: it is not a terrible platform, and in fact it has some things that are better than NT (their trading DOM is simply better as it displays updated volume accumulations that NT does not - - which allows traders who pay attention to VOLUME to note clusters of accumulated large block trades which indicate zones to watch carefully). And, they do have on-chart trading but it is not refined. I know the President there and they claim they are working on it, but NT is far superior in this way. I love every aspect of NT in this area of programming development. Ray, if you are reading this, I am waiting and hoping that you jump past BETA soon and give me a platform that is not a memory hog, offers better OCO "bracket orders on a single click" and other new stuff to make this thing my platform for the long run. Thanks for listening.

[kman1]

Here, Here, very well said. Im sure you speak for the majority of NT users.
Wont renew unless new version is up and running before next quarter.
Too bad, great platform though.