Make Indicator to Work on historical data the same way as it is on live data

[sinpeople]

Hi folks,

I am looking for your help to get the indicator to work the same way on historical data as it does on live data.

In my indicator code, I maintain an internal list of bars, which is classified as either UP or DOWN according to some other indicators. I try to put the same "UP" or "DOWN"
bars into the same list. When a new bar is closed with a different "UP" or "DOWN", the decision is deferred until the close of the 2nd bar. If the 1st bar is the only different one, then
the difference is considered as noise. In this case, both two bars will be added into the latest list. If the 2nd bar is classified as the same of the 1st bar, then a new list will be created
to replace the previous list.

I use the below code to manage the list
===============================================
//calculate indicator values to be used internally in this custom indicator,
calculations are here on every tick;

if(FirstTickOfBar)
{
//when bar close,
//Manage the new incoming bars and add them into the list
//If 1st different bar is incoming, then wait till the 2nd bar close to decide.
//Otherwise, simply add the incoming bar to the list
//A new list is to be created only when 2 consecutive bars are classified as different to those in the latest list.
//in this case, a new list will be created at the close of 2nd bar.
}
else
{
//predict the new UP/DOWN of a ongoing bar against the latest list on every tick.
//STRANGE
//NEW LIST IS ALREADY CREATED WHILE THE 2ND BAR IS STILL TICKING!!!!!!!
}
===============================================

I believe I have something missing in the code. I am expecting to do some prediction using the latest list.
It seems FirstTickOfBar is already true while the 2nd bar is still ongoing, which leads to different decision making with historical data.

Thank you for your suggestions.


Best Regards
David

[bltdavid]

It seems FirstTickOfBar is already true while the 2nd bar is still ongoing

That's wonderful news.
That says the feature known as FirstTickOfBar is working.

When COBC=False,
What do you think it means when you get the 'first' tick of that 2nd bar?
When Historical is true, it means current bar has just closed.
When Historical is false, it means previous bar has just closed.

In other words, when COBC=False,
When Historical is true, current bar has just closed is your 1st bar.
When Historical is false, previous bar has just closed is your 1st bar,
and the 2nd bar is now the current bar being processed.

But, terms like 'current bar' vs 'previous bar' (and in your case, 1st bar
vs 2nd bar) are actually relative, because the definitions can depend
upon your point of view. That 'point of view' is defined for you by
using CalculateOnBarClose.

When COBC=True, 'current bar' is defined as the most recently closed
bar. The 'previous bar' is the second recently closed bar, and so on.

COBC=False redefines what 'current bar' means. Therefore, you have
to adjust your thinking (and your BarsAgo indexes) to account for that.
But it gets tricky, because Historical also has a say in what 'current bar'
means as well.

In real-time data with COBC=False,
FirstTickOfBar means the previous bar has just closed,
because, yes, you're exactly correct, you're already
processing the first tick of the new current bar.

In real-time data with COBC=False,
First tick of the new bar is exactly the same thing as 1 tick past the
last tick of the previous bar, so FirstTickOfBar tells you a bar has just
closed and a new bar has started.
The bar that has just closed is the previous bar, eg, Close[1].
The bar that has just started is the current bar, eg, Close[0].

In historical data and COBC=False,
First tick of new bar doesn't exist, per se, since historical data does
not process on every tick, but on every bar close, so FirstTickOfBar
is true when the current bar has closed, but you have no visibility into
the historical ticks, whether first tick or last tock, previous bar or
current bar -- all you know is the fact the current bar has closed.

When COBC=False, to get the close price of the most recently closed bar,
When Historical is true, use '[0]' BarsAgo index, ie, Close[0] is close of current bar.
When Historical is false, use '[1]' BarsAgo index, ie Close[1] is close of previous bar.

So, in real-time data when Historical=False and COBC=False,
if you want to know the tick by tick value of the Close price as the
current bar develops, well, that is exactly what Close[0] is giving you.
But if you ever need to use the close price of the most recently closed
bar (by now that bar is considered the previous bar, since the current
bar is the new bar being built) you need to use Close[1].

int FirstBarsAgo = Historical || CalculateOnBarClose ? 0 : 1;
if (FirstTickOfBar)
{
    ... your code ...
}
else
{
    ... your code ...
}

every where you use '[0]' BarsAgo indexing, use '[FirstBarsAgo']' instead.

Hint,
I use 'B0' instead of 'FirstBarsAgo', because 'Close[B0]' is easier on the
eyes for readability and on the fingers when typing. Ie, I use Close[B0]
for close of previous bar and Close[0] for close of current bar.

[EDIT: Actually, Close[B0] really means close price of most recently
closed bar, regardless if that bar was the previous bar or the current bar.
See my next post below for some even deeper insights.]

[EDIT2: These corners of NinjaTrader are dark and musty. To expose them
to the light, you need lots of Print statements, lots of careful reading of docs,
but you also need a tremendous amount of study of prior existing code with
lots of your Print statements sprinkled about, and then when you re-read the
docs over and over, you realize just how barren they really are with regards
to these really deep intimate details. Welcome to the club! ]

[NinjaTrader_PaulH]

Hello David,

Thanks for your post.

I've edited my reply here because I did not observe that you were using NinjaTrader7 and my reply was based on the Tick replay capability of NinjaTrader8 that does not exist in NinjaTrader7.

Member bltdavid has provided a solid reply, thank-you.

[bltdavid]

Let's put some stabilizers on this boat.

Let's start by putting these definitions in concrete.

Close[0] is close price of current bar
Close[1] is close price of previous bar.

Ah, there's the rub. Do you see it?
The underlying definitions of 'current bar' and 'previous bar' are
now the parts that need further refinement.

Sometimes 'current bar' means the most recently closed bar,
and sometimes it means the active bar under construction.

How to know the difference?
That's where CalculateOnBarClose comes to the rescue.
Sort of. Let's call it a 'partial' rescue.

When COBC=True, 'current bar' means the most recently closed bar.
When COBC=False, 'current bar' means the active bar under construction.

So why 'partial' rescue?

When processing historical bars (eg, when Historical is true)
your OnBarUpdate is being called only on the bar close of the
most recently closed bar -- so the semantics are Close[0] is
the close price of the current bar.

Woah, woah, woah .. hold the presses ... you thought the value of
CalculateOnBarClose was calling the shots for 'current bar', right?
Well, almost but not quite. Historical also participates.

Why is that? When Historical is true, it overrides the COBC value and
forces the same semantics as if COBC=True. Why? Because, well,
your OnBarUpdate is only being called on bar close for historical bars,
so even though COBC might be False, or True, when Historical is True,
the value of COBC won't matter -- so, you gotta account for that when
you process historical bars, ie, the semantics will change on you when
you change to real-time bars.

After OnStartUp is called, COBC never changes from true to false or
false to true -- it is stuck -- so we can add COBC to the concrete list.

After OnStartUp is called, Historical is true while processing historical
bars on the chart, and if the chart transitions to real-time data, there will
be exactly one transition where Historical changes from true to false.

But now, after Historical changes to false, which means you're now
processing real-time data, assuming COBC is also False, all of a sudden
your OnBarUpdate is now called for every tick -- and immediately you
must switch your semantics so that Close[0] is close price of active bar
under construction, and switch to Close[1] to get close price of most
recently closed bar.

I call this the historical to real-time transition, or more specifically, I use
the expressions LHB and FRB to refer to the Last Historical Bar and the
First Real-time Bar, and then I use shorthand LHB -> FRB to mean
that transition.

As you get into studying COBC=False and using helper index variables,
like B0 and B1, you'll find the LHB -> FRB transition can be a big deal.
Why? Because, well, among other things, it can affect the value of
CurrentBar.

That is, I have seen plenty of instances where the LHB and the FRB
each have the same value for CurrentBar. Yeah, pretty weird, right?

Add a Print to your OnBarUpdate that dumps these values when
FirstTickOfBar is true,

Bar=1400 COBC=False Historical=True
Bar=1401 COBC=False Historical=True <-- this is LHB
Bar=1401 COBC=False Historical=False <-- this is FRB
Bar=1402 COBC=False Historical=False

WTF? Yep, it can happen.

I've seen cases where OnBarUpdate processes a second bar with
the same value of CurrentBar -- thankfully, this duplicate situation only
seems to happen during the LHB -> FRB transition -- but it doesn't
seem to happen every time -- which is another weirdness.

When you get this deep into these dark and musty corners, you're really
doing some serious NinjaTrader coding, and these details may (or may
not) matter to you -- lol, so it's all relative, right?

Hopefully, this extra insight will not be needed by your indicator, but if
you do need it -- solutions do exist.