Better practice to store a collection of data

[efeuvejota01]

I would like to store the price and the bar number (CurrentBar) of the ZigZag edges or angles.
In terms of machine utilization and performance what would you recommend
A) Using two Lists
B) Using one ISeries<double>( I will know the price and the bar since only the bars where the edge appears are going to be with a valid number, the rest will hold 0 or null)
C) Using a C# array, with increment each time I add a new value
Thank you very much
FVJ

[bltdavid]

A)
ok, well, maybe B) ... see below ...

I vote for 2 lists, but you gotta keep the Highs separate
from the Lows, right? Keep reading below for my trick.

Start by defining your two lists,

private List<int> zzBar = new List<int>();
private List<double> zzPrice = new List<double>();

[This technique is commonly called parallel arrays.
In fact, did you know that all Series<type> managed by
NinjaTrader, regardless of type, comprise a giant set of
parallel arrays? Read that 5 times, let it sink in. Yep,
Series<type> are an example of parallel arrays. The benefit
for Ninja is one giant data set for each bar, split out across
multiple (and parallel) series. But I digress.]

Thus, I recommend two lists, and keep them in parallel,
which is actually pretty easy -- when you Add to one, you
must also Add to the other. Obviously, you don't Add to
these lists on each bar, otherwise, you'd just use a Series,
right? The idea is to call Add sparingly, only when needed.
That's the memory savings these 2 parallel lists give you.

-=o=-

Here's an easy trick for keeping track of High vs Low.

When you find a ZigZag high point, save it like this,

zzBar.Add(CurrentBar);
zzPrice.Add(High[0]);

When you find a ZigZag low point, save it like this,

zzBar.Add(-CurrentBar); // <-- negative
zzPrice.Add(Low[0]);

See that?
Low[0] is saved with a negative CurrentBar.

How come?
Later, when you examine these lists, that's how you'll
know when zzPrice[n] is a Low point. That's the
trick -- a negative value at zzBar[n] means that
zzPrice[n]is a Low. If zzBar[n] is positive,
then the price at zzPrice[n] is a High.

Make sense?

-=o=-

Finally, a word of caution.

Why go through this extra trouble?
Why not just use one (or two) Series and be done with it?

I mean, you have friggin tons of memory compared to
the minuscule memory savings you get from these two
parallel lists -- you know that, right? The best way to
decide what to do is how you plan to access the data.
With Series, it's backwards, starting at index '0'. But
many values will be 'unset', whereas w/parallel lists all
values added are valid and it's forward looking, like a
regular array. But that means the most recent ZigZag
bar/price data pairs will be located at the end of the
lists. [Hint: Store your List in a class, then create
indexers, so you can also use the '0' index to get to
the most recent value. This, too, works great.]

That said, I've used this parallel list technique to great
advantage -- it helps to corral my brain cells around the
data better, esp for cases like what you're doing.

Good luck!

[efeuvejota01]

Thank you very much bltdavid!!!
I was guessing that is the way, but I wanted to be confirmed by someone with more experience (like you!!)

One more question ( to verify that i am understanding correctly)

how do you

Store your List in a class, then create
indexers, so you can also use the '0' index to get to
the most recent value. This, too, works great.

??

I have created the lists exactly as you mention

private List<int> zzBar = new List<int>();
private List<double> zzPrice = new List<double>();

what else shall I do in order to create indexers??
Thank you again!!!
FVJ

[bltdavid]

--- Introducing ListSeries<T> ---

If you want Series-like indexing, you'll need to create a class⁽¹⁾. The
concept of an indexer is part of the C# language. I invented the class
ListSeries<T> as a wrapper class for a List<T>. The difference
is I wanted to make the index accessors access the internal List<T>
backwards, so that accessing '[0]' element is actually the last element
in the list.

That is, since I wanted to use 'backwards' indexers just like Series<T>,
I had to invent a new class to do that -- thus ListSeries<T> was invented.

The backwards indexers give ListSeries<T> the same access semantics
as Series<T>, which is the whole point. That is, when we access the '[0]'
element, we are accessing the most recent element added to the list, just
like a series. The wrapper class ListSeries<T> is used to hide the math
to get the real index to access the internal privately maintained List<T>.

That means,
for 'List<T> zz' accessing 'zz[0]' is the first element in the list.
for 'ListSeries<T> zz' accessing 'zz[0]' is the last element in the list.

I'll give you the code I use in a moment. First, let's discuss how to use it.
Then you can read the code at the bottom and fill-in any brain gaps as a
exercise for yourself.

--- How to Define Your Variables ---

I call my class ListSeries<T>, and you define variables of this type
just like you would for List<T>. For example,

private ListSeries<int> zzBar = new ListSeries<int>();
private ListSeries<double> zzPrice = new ListSeries<double>();

But, ListSeries<T> supports a Name property, which is used if you
have an index out of range exception on the internal list, so, in practice,
in my own code, I always define them like this,

private ListSeries<int> zzBar = new ListSeries<int>() { Name = "zzBar" };
private ListSeries<double> zzPrice = new ListSeries<double>() { Name = "zzPrice" };

If you want to specify an initial capacity (like what List<T> allows), you
could define your variables like this,

private ListSeries<int> zzBar = new ListSeries<int>(128) { Name = "zzBar" };
private ListSeries<double> zzPrice = new ListSeries<double>(128) { Name = "zzPrice" };

See that 128 in there? That's the initial capacity, same as List<T>.

When you look at this carefully, you can tell that I've used ListSeries
everywhere you would normally specify List.

The idea is that ListSeries<T> is almost a drop-in replacement for List<T>.

--- How to Add New Elements ---

Notice I said almost.

Remember, a ListSeries<T> is a kinda like a convenient hybrid of
a List<T> and a Series<T>.

Nevertheless, I decided to use slightly different semantics when adding
elements to a ListSeries<T>,

Normally, with List<double> zzPrice you would add an element to
the list like this,

zzPrice.Add(High[0]); // <-- List<T> append element to end of list

But with ListSeries<double> zzPrice, you add elements to the
list using the Set method,

zzPrice.Set(High[0]); // <-- ListSeries<T> append element to end of list

For anyone familiar with NT7, you can tell where the idea of 'Set' came
from -- these are the semantics used in NT7.

--- How to Access Elements ---

Now we get to the whole point of why I use my ListSeries<T> wrapper:
the ability to access the most recently added item with the simple semantics
we enjoy with a Series<T>.

Let's give a running example. Given this definition,

private ListSeries<int> zzBar = new ListSeries<int>();

with these values added to the ListSeries,

zzBar.Set(102);
zzBar.Set(236);
zzBar.Set(528);
zzBar.Set(623);
zzBar.Set(747);

Then zzBar.Count is 5, and the following code,

for (int indx = 0; indx < zzBar.Count ++indx)
    Print("zzBar[" + indx + "] = " + zzBar[indx]);

produces this output,

zzBar[0] = 747
zzBar[1] = 623
zzBar[2] = 528
zzBar[3] = 236
zzBar[4] = 102

That means, every time you add an element using zzBar.Set(x), you
can immediately retrieve the value x by using zzBar[0]. If you next add
y using zzBar.Set(y), then y is at zzBar[0], and x shifts to zzBar[1].

These 'shifting' access semantics is exactly what happens with every
Series<T> -- that is, just before the OnBarUpdate, a new slot is added
to every Series<T> (remember, all Series<T> slots live together in
parallel) and inside OnBarUpdate all indexes for every Series<T> have
automatically been shifted by 1. With ListSeries<T>, this shift by 1
happens because of you -- only when you call Set.

Obviously you can see your ListSeries<T> will probably have dramatically
fewer elements that a Series<T>, which may or may not be material to
their use. I treat the memory reduction as a small bonus, the much larger
bonus is using zzBar[0] and zzBar[1] to access the two most recently
values added to zzBar.

This can result in some very simplified code, because the most recent
values of zzBar and zzPrice are the values that are probably of the
highest interest -- and these values are immediately accessible in your
code via the '0' index using zzBar[0] and zzPrice[0].

-=o=-

Why do I like ListSeries<T>?

It makes the code more elegant, and (for me) easier to understand. Using
a ListSeries<T> borrows the same brain cells for backwards accessing
we all do with a Series<T>, so seems like a natural fit.

If zzBar were a List<T>, the two most recently added elements would be
accessible with zzBar[zzBar.Count-1] and zzBar[zzBar.Count-2],
so being able to instead use zzBar[0] and zzBar[1] simplifies the semantics
and beautifies the code.

Just remember, a Series<T> gets a new 'slot' for every bar, and that slot
may or may not have a valid value.

A ListSeries<T> gets a new 'slot' every time you call Set, and every value
in every 'slot' is valid, because Set(x) always stores x in the new slot.

Because of the backwards indexing, I think ListSeries<T> can be produce
much simpler code, ie, a much more elegant solution than a plain List<T>.

For example, say your code needs to look at the two most recent swings
from the ZigZag indicator, well, you would look at zzBar[0] and zzBar[1],
as well as zzPrice[0] and zzPrice[1].

--- The ListSeries<T> Code ---

And finally, the code, which you'll need to copy and paste into your partial class
library, or your base class library, or directly into your indicator,

[EDIT: See post 20 below, I made ListSeries<T> live in file MyCommon.cs]

#region class ListSeries

// same as List<T> but with DataSeries like semantics
// eg, property index [0] retrieves most recent item
// eg, property index [Count-1] retrieves oldest item
public class ListSeries<T>
{
    protected List<T> ItemList = null;  // list of data items
    protected int Index0 = -1;          // fast access to index of last item
    private string name = null;         // user-defined variable name

    public string Name
    {
        get { return name ?? GetType().Name; }
        set { name = value; }
    }

    public const int InitialCapacity = 32;

    public ListSeries() : this(InitialCapacity) {}
    public ListSeries(int initialCapacity)
    {
        ItemList = new List<T>(Math.Max(initialCapacity, InitialCapacity));
    }

    public int Count { get { return ItemList.Count; } }

    // careful, do not add/remove items through this reference
    public IList<T> ToList() { return ItemList.AsReadOnly(); }

    public T this[int indx]
    {
        get {
            if (indx == 0)
                return ItemList[Index0];
            else if (indx > 0 && indx < ItemList.Count)
                return ItemList[ItemList.Count - indx - 1];
            else
            {
                string s = Name+"["+indx+"] Get failed: Invalid index: Count="+Count;
                throw new IndexOutOfRangeException(s);
            }
        }
        set {
            if (indx == 0)
                ItemList[Index0] = value;
            else if (indx > 0 && indx < ItemList.Count)
                ItemList[ItemList.Count - indx - 1] = value;
            else
            {
                string s = Name+"["+indx+"] Set failed: Invalid index: Count="+Count;
                throw new IndexOutOfRangeException(s);
            }
        }
    }

    public virtual void Set(T NewItem)
    {
        Index0 = ItemList.Count;
        ItemList.Add(NewItem);
    }
}

#endregion

​

--- Final Thoughts ---

Now let's full circle back to your question about ZigZag.

When using ListSeries<T>, I would recommend you use 4 of them, like this,

private ListSeries<double> zzHigh = new ListSeries<double>(128) { Name = "zzHigh" };
private ListSeries<int> zzHighBar = new ListSeries<int>(128) { Name = "zzHighBar" };

private ListSeries<double> zzLow = new ListSeries<double>(128) { Name = "zzLow" };
private ListSeries<int> zzLowBar = new ListSeries<int>(128) { Name = "zzLowBar" };

That makes dealing with the High vs Low events in the ZigZag indicator a lot easier.

Enjoy!



Footnotes:

(1) Technically, indexers are supported by class, struct, or interface.
But, since I was writing a wrapper class around List<T>, I didn't see
the need to mention the other two. I don't like to digress.

[efeuvejota01]

Thank you very much!!!
I have to process all the info you have provide me... ( which is a lot!!)
let me try in the code and I will post any question may arise...
FVJ

[efeuvejota01]

Hello bltdavid,
I started to test the approach that you kindly proposed me, but i got stocked at the begining of the process


I searched and noticed that is required to use System.Collections.Generic; and it was added:


What am I missing??
Thank you
FVJ

[QuantKey_Bruce]

What you're missing is that you didn't paste in the ListSeries code he posted above so that class is not defined.

And finally, the code, which you'll need to copy and paste into your partial class
library, or your base class library, or directly into your indicator,

You did not do this.

[efeuvejota01]

Hi bltdavid,
I almost did it!! It has been a game changer your advise and help.
I have 2 more questions that i tried to find but have had no luck!!.
f zzBar were a List<T>:
A) how can I know how many element does the ListSeries<> has? or what would it be the following command

zzBar[zzBar.Count]

related to ListSeries<T>. Considering that I do not set an initial capacity for the ListSeries<T> or a List<T>
B) How would you translate the following command

Print(string.Join(", ", zzBar));//

to apply for a ListSeries<T>
Thank you very Much!!!
FVJ

[bltdavid]

I almost did it!! It has been a game changer your advise and help.
I have 2 more questions that i tried to find but have had no luck!!.

Excellent news!

A) how can I know how many element does the ListSeries<> has?

Just use the Count property.
ListSeries<T> supports Count, just like a List<T> does.

Count is maintained in real-time, it always reflects the current
number of elements.

When Count reaches Capacity, the allocator grows the list by
another chunk of elements, and when Count reaches that new
capacity, the allocator grows the list again ... and the cycle
repeats.

or what would it be the following command

zzBar[zzBar.Count]

related to ListSeries<T>.

zzBar[zzBar.Count] is incorrect, this should zzBar[zzBar.Count-1] because
the List<T> and ListSeries<T> indexing is zero-based, just like arrays.

Let's assume
A is a List<T>
B is a ListSeries<T>

A[0] is the first item added to the list.
B[0] is the last item added to the list.

A[A.Count - 1] is the last item added to the list
B[B.Count - 1] is the first item added to the list.

Study List vs ListSeries until you understand the above
statements and why they are true.

Considering that I do not set an initial capacity for the ListSeries<T> or a List<T>

Which is totally fine, you don't need to specify that value.
Whether you specify it or not, it makes no difference to how
you use the List<T> or ListSeries<T>.

B) How would you translate the following command Print(string.Join(", ", zzBar)); to apply for a ListSeries<T>

Print(string.Join(", ", zzBar.ToList()));

[efeuvejota01]

Thank you bltdavid!!!

One more last question...

In order to update the last ZZ bar ( thinking we are in a trend) it is necessary to remove the last (no longer valid) bar number, And to add the new CurrentBar.

I tried to do it though the following command

zzBar.ToList().Remove(zzBar.Count-1);

but the following error arose

Indicator 'ZZUTG_RPV_V06ListSeries': Error on calling 'OnBarUpdate' method on bar 11: Collection is read-only.

How do you workaround this issue?

FVJ

[bltdavid]

Ok, no problem. Let me explain.

Uh, so first, I never had to remove an item in a ListSeries.

Just like a regular Series, you don't really do this, so I
never provided that feature.

You can, however, replace values in a ListSeries.

Let's say you add an item to a ListSeries<int>,

Let's say your code last did this,

zzBar.Set(CurrentBar);

and now you want to replace that value. You already
know it's been stored at the 'slot' currently available
at index [0], so just do this,

zzBar[0] = NewValue;

which updates the slot containing CurrentBar with
the new number NewValue.

-=o=-

See that?
Just like a Series, you're not supposed to be able to
remove 'slots', but you can update the values in any
'slot' just by re-assigning a new value to it. This is
how a Series works. I duplicated that behavior.

For a ListSeries, it helps to think in terms of 'slots'
(not elements) because that's how you think about
a Series.

Remember,
OnBarUpdate adds a new slot to each Series, doing
this behind the scenes.

The Set method adds a new slot to a ListSeries,
which you must call manually.

Here is your important light-bulb moment:
In both cases, once the slot is created, you can
assign values to this most recently created slot
using normal assignment,

mySeries[0] = ... some value ...
myListSeries[0] = ... some other value ...

or change the value in the previous slot,

mySeries[1] = ... some value ...
myListSeries[1] = ... some other value ...

or the value 5 slots ago,

mySeries[5] = ... some value ...
myListSeries[5] = ... some other value ...

In NinjaScript, we call that 'bars ago', but with my
ListSeries, we gotta stick with the phrase 'slots ago'.

Make sense?

[bltdavid]

Ready for some more history?

Ok, let's go down memory lane a bit.

-=o=-

Yeah, you called ToList() ... but the list returned
was read-only, and an exception was generated.

Excellent!

This is good. The API is working precisely as I
intended. I made the returned list read-only on
purpose.

Why read-only?
Precisely to catch 'invalid' situations like what you
were trying to do, or what I might have wanted to
do were I not thinking straight some late night.

From the point of view of using a ListSeries, I
tried to mirror what you can do with a Series.

You can't make a end-run around a Series using a
List, so neither can you do that with a ListSeries.

But getting access to a List was extremely useful
on certain occasions (as you have discovered), so
I added ToList, but made it readonly.

-=o=-

I'm a proud papa.

How so? When I first wrote ListSeries, it was for NT7,
so I was actually mirroring a DataSeries, since that is
what NT7 called it back then.

Anyway, the concept of a ListSeries survived the NT7
to NT8 porting with no changes, and it matches and aligns
quite well with the NT8 concept of a Series.

In fact, it aligns better with NT8 than it did with NT7, which
I thought was pretty damn cool.

A ListSeries is simple, and elegant, and borrows the Series
semantics as wrapper for a List -- it turns out that, for NT8,
it mirrors a Series very very well.

-=o=-

Some final thoughts.
You gotta think in terms of slots.

Except for that Set method, which is how you create a new
slot (which comes from NT7), they're very similar.

See that?

I said 'Set' creates a new slot -- think in these terms and
it aligns with a Series.

If I say 'Set' adds a new element -- then thinking in these
terms and it aligns with a List.

Well, hey, it's a ListSeries, so both ways are correct.

But I prefer to think in terms of adding new slots ... so that
it aligns better in my brain, because like I said, when you
use them, a Series and ListSeries are very similar.

Enjoy!

[efeuvejota01]

Yes Thank you!!!
FVJ

[efeuvejota01]

Hello!! Thank you for your attention:
I use the concept of the "MySharedMethodsAddonExample.zip (3.6 KB, 693 views)"

https://forum.ninjatrader.com/forum/historical-beta-archive/version-8-beta/86475-how-to-avoid-file-errors-using-streamwriter-on-multiple-instrument-backtest?p=733956#post733956

to communicate between Indicators when there is no other way.
If I would like to copy a ListSeries<int> form one to another Indicator.
What would be the Syntax to achieve it??
Thank you very Much
FVJ

bltdavid,

[NinjaTrader_BrandonH]

Hello FVJ,

Thanks for your notes.

bltdavid is correct in the advice they shared regarding using C# Lists. Please note that using C# Lists fall under C# education and in the support department at NinjaTrader we do not provide C# education services in our support so this would go beyond the assistance we could provide you with.

That said, this forum thread is open for bltdavid and other community members to share their insights on the topic.