Is there an off-by-one error in ChartControl.GetXByTime at the 01s time interval?

[carnitron]

Consider the image below. The time pointed to by the top arrow (19:09:23.488) is the time received in OnOrderExecute for the Order that exits me from the trade. You will notice that the magenta NT8 indicators for this time point are drawn a second later, at 19:09:24. (Ignore the ms on the first time, I truncate those internally before drawing the salmon-colored callout line you see.) Now the reason the salmon callout line is drawing at 19:09:23 (the correct time slot) is that I am subtracting 1 second from the time I pass into ChartControl.GetXByTime(). So my line is correct, b/c I am compensating for this seeming bug, whereas the NT8 indicator isn't correct, b/c it isn't compensating. I am on version 8.0.27. If this is not a bug for some reason, and there is some option or scale factor I am missing, I'd love to know.



One thing to note. I have a separate worker thread that does this:

Dispatcher.InvokeAsync(() => ChartControl.InvalidateVisual());

In a busy wait loop at 42ms intervals, so that I can improve the perceived frame rate at which the chart draws. Is it possible that this is causing a draw that somehow is using the time from the prior "frame" in some places and times from the new "frame" in others? That might explain the issue. I.e. GetXByTime() has the position from the previous frame, where the bar drawing has the positions from the next frame.

Cheers, Peter

P.S. As an aside, if there is a better way to get 30 or 60 fps out of the chart? I got this of the forums here, but it's kind of a brute force method, obvs that seems like it might be leaving me vulnerable to unexpected madness. I have to lock ChartControl both in my thread and in my main render to prevent a thread contention exception, which works -- but who knows what else wants to work with the ChartControl... Since we're on top of DirectX, I have a somewhat beastly video card, and the overall draw complexity is so trivial, it seems like we *should* be able to get truly ridiculous frame rates. While it might seem unnecessary, given that we only get new market data injected at 1 fps, it's definitely possible to get much cooler, more engaging indicators that draw the eye to the right place rendering like a video game.

[QuantKey_Bruce]

The time 19:09:23.488 is contained in the 1-second interval ending at 19:09:24.000 which is why the execution marker is positioned there. The timestamp of the 1-second bar is the timestamp of the end of the 1-second interval it contains.

On your P.S., trading applications typically update the screen on a timer basis rather than "each event" because sometimes during busy market conditions, there could be thousands of events arriving per second and to try to draw them all "immediately" would be a mistake. Human perception is not such that drawing the candles more than a certain rate makes any difference, so the application is architected to maintain its speed and responsiveness even in the face of an onslaught of rapid events. Games are not designed this way because video games have different considerations. Nevertheless, the speed with which the screen is drawn should be considerable under normal circumstances.

[carnitron]

Ok so I am familiar with the time phenomenon you are talking here after building out my data ingest system. Basically, if we are on the 1 minute scale, we get the event for time 19:04:00 on the 01m BEFORE we get 19:03:02 on the 01s scale, b/c we have to build out the data for 19:04 on 01m with the tick data from 19:03:01 to 19:04:00. I.e. 19:04 on the 01m "opens" the time interval rather than "closing" it.

Put another way, the market gods decided by fiat long ago that 19:03:30 is part of minute 19:04, not minute 19:03.

This sort of feels like the wrong choice from a UI perspective, since intuitively people will look for times like 19:03:XX at 19:03, not 19:04. But I could grudgingly accept this reality if it is known that all trading tools work like this, and we have confirmation from NT8 that this is specifically what is being done here.

(I'd love it if someone had insights into why the market gods made the time choice they did here, like if there is was a good reason for doing so, that overmatches the UI consequences I mention above.)​

[carnitron]

Bruce - So I definitely hear you that painting after every event is a bad idea, lol. Painting on a timer seems like an odd choice though, b/c if you are done repainting long before the text timer tick, you are twiddling your thumbs when you could be drawing. This is why video games start on the next game loop as soon as the last one is done. (It gets more complicated/nuanced than this in practice, but that is the main idea.)

Regardless, when I turn off my chart control invalidator thread, my animations look terrible. It looks like I'm getting a frame rate of maybe 2-3 fps. When the invalidator thread is on, it is significantly smoother and definitely looks north of 20 fps, though I have not timed it specifically.

What I want to know is if NT8 has specific blessed techniques for getting smooth animation on the chart that are better than the one I am using. Or if there are plans to render this thing like a video game at some point. Modern machines more than have the rendering horsepower and it is possible to make awesome next-gen looking stuff with that kind of visual fidelity.

[QuantKey_Bruce]

The goal in a trading app is not to maximize frames per second - that would be wasting CPU. In a video game, the presumption is that the console or PC can give the game its full attention, so there is no such thing as wasted CPU - you want to maximize its appearance regardless of how busy the CPU is. In a trading app, you want to keep the CPU free so that it can respond to incoming market data events which you cannot easily predict.

If you're only getting 2-3 frames per second, you're doing something wrong. I'm not sure what you mean by "animation on the chart" but it sounds like you're trying to draw something on the chart that is going to monopolize the CPU just to try to get something to display as smoothly as possible but lead to the CPU being unable to respond rapidly to incoming data because it was too busy trying to make your animation slightly faster. That's not generally what a trader wants - they want the screen to be updated "fast enough" so it's faster than their perception (generally 1/60 sec is plenty fast and often 1/4 sec is even fast enough) and yet the UI remains 100% responsive so when they drag a scroll bar or open another window, nothing slows down at all. If it's using 100% CPU to render your cool looking graphic, that will not be the case and the trader will be quite upset.

[QuantKey_Bruce]

The reason minute bars are timestamped with the end of the minute is because to timestamp the data from 19:03:30 as 19:03 implies that at 19:03 you knew the data, and you did not. You did not know it in its entirety until 9:04, which is when the bar is timestamped.

[NinjaTrader_Jesse]

Hello carnitron,

Regarding rendering, bruce was correct that the platform has a fixed interval for rendering, that is to deliverer the best overall performance. When using a faster interval in a fast market across many charts with many updates you could end up slowing down the overall process which effects everything being done. Its best not to try and animate anything in the chart because it was never intended for that purpose. Charts are essentially a static image of a segment of calculated data, a new tick will cause an update to the underlying data which then gets updated on the next render pass. Thats another static image containing the updated data. Keeping the amount of render passes to a minimum is in your best interest to keep overall latency low.

On a side note calling invalidate on the chart panel can lead to deadlocks which would prevent using the parts of the platform and make cause you to unable to trade. This is mentioned in the helpguide to avoid doing that.

It is NOT recommended to invalidate the chart control directly as this could cause issues with threading which result in dead locks.

https://ninjatrader.com/support/help...htsub=invalida

[carnitron]

Jesse/Bruce. So I hear you, but it doesn't have to be this way. Video games draw at high frame rates when there is CPU/GPU availability to do so. When important gameplay calculations or large amounts of network traffic come in, game engines switch gears and spend most of their time working on that and much less time drawing, which is why you see the frame rate go lower during those times. Drawing at high speed is "optional" whereas handling core gameplay is not.

Trading apps are no different -- market events are like gameplay events, they cannot be ignored and drawing a lot would take a backseat during times of heavy traffic. But when the market traffic is low, letting 95% of the machine's CPU/GPU clock be idle when we could be having higher fidelity chart visuals is a waste.

It's unsurprising that NT8 works the way it does, I doubt the developers had a prior career in video games, so they wouldn't have known there is a better way.

Anyhow, it is what it is. I AM getting 2-3 fps without the invalidator thread and I don't think I'm doing anything wrong. I'm drawing some lines, boxes and text during OnRender in my indicator, that's it. (I transition in some chart callouts with time-based alpha blending and it's all super simple, not that this is relevant to my FPS.)

If there is a way to somehow increase the timer frequency NT8 uses, I would love to know what that is. PerfMon shows all my cores being wildly underutilized at all times, which isn't surprising, given that it I develop on a pretty beastly machine.

Jesse - it sounds like I should NOT use my invalidator thread when live. I have definitely seen stalls on the chart window when it is on that can last for half a minute before they mysteriously self-correct. It seems like these might be the deadlocks alluded to in the help guide comment you provided.​

[carnitron]

Bruce -

"The reason minute bars are timestamped with the end of the minute is because to timestamp the data from 19:03:30 as 19:03 implies that at 19:03 you knew the data, and you did not. You did not know it in its entirety until 9:04, which is when the bar is timestamped."

That is a great succinct explanation, TY.

[carnitron]

Jesse - could you confirm that it is known and intentional that NT8 draws trade entry and exit indicators as discussed above? I.e. a trade that completes at 13:05:30.500 will show up on the chart on the bar for 13:05:31?

[NinjaTrader_Jesse]

Hello carnitron,

it sounds like I should NOT use my invalidator thread when live. I have definitely seen stalls on the chart window when it is on that can last for half a minute before they mysteriously self-correct. It seems like these might be the deadlocks alluded to in the help guide comment you provided.​

That would be a general result from deadlocking the chart. You can also see it not unfreeze or cause major problems like crashes if this gets out of control. If your script is deadlocked it may not be able to drive its logic to produce trades or close trades which may result in losses.

NinjaTrader is not designed to be a video game and has no engine that would be similar to that so the comparison between the two is not really an apples to apples comparison. There is no want/need to update at that fast of a rate for many reasons. One of the main reasons is that a game you have 1 window open ever, in NinjaTrader you may have 50 charts open which all have their own render loop. NinjaTrader is a trading application that is designed to be run at high performance from most any machine. A large majority of users do not have powerful pcs and some opt to run the platform from server environments which have limited visual capabilities. This gives the best UI experience and prevents overloading the CPU for no reason. If a single chart was run unrestricted its "game loop" the rendering pass would happen as fast as the CPU would let it which means you could really only have 1 chart open and may have slowness on the PC and in the application.

Using some type of game engine would also limit the types of machines it could run on due to high end requirements. Games have many optimizations to render complex visuals like 3d models or use shaders for effects which are not needed in a trading chart, that would be a lot of unnecessary overhead to display simple data. NinjaTrader has the ability to use complex rendering with directX by means of SharpDX in a 2d space. That would limit the rendering to a simple render loop for items that can be rendered by direct2d at a fixed rate to avoid maxing the CPU for no reason other than to display something slightly faster.

The above is correct about how time works

[QuantKey_Bruce]

I disagree with the premise that the way video games are designed is a "better way" - that's a different problem and it requires a different solution.

[QuantKey_Bruce]

I also strongly disagree with the idea of invalidating the chart thread because you are trying to boost the fps. That's silly, and is not a concept that belongs in trading. The emphasis here should be on the application being responsive, including keeping resources free for the next burst of incoming market data, which you cannot anticipate, so you should not use all of the resources just because they are there to make the chart refresh a few fps faster. This is not a game, it is business, and reliability and responsiveness are FAR more important than smooth animation. By a factor of double.PositiveInfinity.

[carnitron]

Jesse - thank you for the info and confirmation.

I do come from a background in video games, so to have all the power of SharpDX but then only be able to get 3 fps is a little bit of a bummer for someone like me. But I can use this invalidator thread so that I can create visuals for myself during development and just keep it off when running for real.

It IS possible to get smooth animation running on top of DirectX in many windows in a scalable way on all manner of hardware. It's what web browsers do with modern HTML. Create a bunch of windows with links like the following WHILE you have NT8 running in the background to see what I mean:

https://codepen.io/creativeocean/pen/oNaWjjo

https://codepen.io/pavlovsk/pen/dygOVgj

https://codepen.io/MackFitz/pen/QWZpyMa

And click and drag here for a treat, my fav lol:

https://kabam.com/

Finally, here is a DirectX based trading tool that creates super useful visuals that I came to rely on when trading live:

https://bookmap.com/en

It absolutely blew the doors off of anything I could find in seven different trading front ends.

But I'm not here to convince you guys or win an argument. I'd like higher fidelity visuals for some of what I want to do. But it sounds like my path forward is to write my own chart control in a separate window or app at some point. It'll just be more work, no biggie.