How to wait till last bar on Multitimeframe Strategy

[NeoAkira]

Hi Ninja Trader Support Team,

As you might think I have a little Problem.

I have programmed a NTStrategy that collects Data from multiple Indicators and different Timeframes (10 at the moment) that I added via Add(PeriodType) and saves this data to a csv table. All timeframes are based on daily data (eg. 1D,2D,3D,4D,...).

After this table is created (this means that all timeframes are on the last bar) I want to call some functions that use the data in this table to create further tables and do some more calculations.

The main algorithms work and I can create and collect my data and also create further tables without problems. This part works flawless.

My main Problem is that my further functions are not called on the last bar when all previous calculations are finished and the table is created over the full data in the chart. eg. I have data from 2000 to 2014 on the chart and the timeperiods i load but the generated table stops at 2004 because the if clause (see below) seems to get triggered.
I know that this is the problem because without this if clause the table is generated and saved complete from 2000-2014.

I already tried a couple of things but without success.

e.g.
if ((Count - 2) != CurrentBars[lastBarArrayIndex]) return; // wait to last bar;
or
if ((Count - 2) != CurrentBars[0]) return; // wait to last bar;
{
MyFunctionCalls
}

Is there any way (function or code) to wait in OnBarUpdate() till we are on the last Bar in all timeperiods that are added to the strategy?

Thanks in Advance

[NinjaTrader_JC]

Hello NeoAkira,

We do not really have any methods that will do what you are looking for. Count is the best supported way that I can think of that may accomplish this.

Do you know what the values are returning at this point to see why it is returning?

[NeoAkira]

Just for an Update.
I figured it out based on your Info and other Threads based on this topic.

My Problem was that i also used BarsInProgress to only react to relevant Bars in my BarsArray and that mixed up the Count and CurrentBars values.

But I still have another Problem that I couldn't find a solution for.
Is there any possibility that the Periods I add via AddPeriod use my local historyDB for the Data instead of the server from my Data Provider?

When I use my strategy and table generator only with the current Period on the Chart it creates my table from 1993 till current day based on the data I have in my local history DB.
When I add additional Periods via the AddPeriod Function the Table generated has only Data from 2002 till current Date.
I know that the history Data that you can normaly get from FXCM and your Servers are only till 2002 or end of 2001 (Daily Data).
I also tried to switch off the Option in Tools/Options/Data "Get data from server" but without success.

I hope that you can give me a positive Solution for this Problem.

Thanks in advance

[NinjaTrader_JC]

Hello NeoAkira,

NinjaTrader will try to grab Historical Data from your Local Files inside of NinjaTrader if they are not available on the server for the dates that you are requesting them from.

Do you see the Historical Data for the time and type of data that you are adding inside of the Historical Data Manger? You may check this by going to Tools -> Historical Data Manager -> Edit tab.

http://www.ninjatrader.com/support/helpGuides/nt7/editing.htm

[NeoAkira]

The historical Data is in my local Database and I see it in the Historical Data Manger.
In my case I use Daily Bid Price and in the Data Series of the main Chart where I start my Strategy from I have defined a customRange with StartDate 1990 and EndDate 2014(currentDate).

I have multiple Functions in my strategy that generate csv tables based on historical data and the indicators i use. As long as i call only Functions that use the base period on the chart my tables are generated from 1993 till today. when I use Functions that add other Periods via AddPeriod() to the chart the generated tables start in 2002.

This indicates to me that somehow the added Periods interfere with the loaded Data. All Periods are based on Daily Data (main chart 1D, added Periods 5D, 10D...100D) overall I add 10 Periods to the Chart via NinjaScript.

Hope this helps you to understand my problem better.

Thanks in advance.

[NinjaTrader_JC]

Hello NeoAkira,

I believe this maybe due to the strategy waiting for 20 bars of data on all of your series before it is starting to process and since your highest time frame is 100 days it maybe causing it to not load some years worth of data.

Can you try to change the "Min bars required" inside of the Strategy Parameters to 0 and retest this?

[NeoAkira]

That was one of my first thoughts and therefore I already set "Min bars required" to 1. I need at least one Bar in my Strategy because some function look 1 Bar back. If I set it to 0 I get an Error.
The setting of "Min bars required" helped that the resulting tables now start from 2002.
Before i set "Min bars required" to 1 they started in 2005.

But how can it be that almost 10 Years of Data (from 1993) are ignored with only 1 Bar required...that is the Mystery.

Do you have more thoughts on this problem?

Thanks in advance.

[NinjaTrader_JC]

Hello NeoAkira,

If changing "Min bars required" made the dates go back farther then we know that it and any checks that are done inside of OnBarUpdate() to wait until there are enough bars of data is going to be the cause of this.

Please let me know what exact periods you are adding inside of Initialize, any checks that you are doing inside of OnBarUpdate(), and what data you are trying to access inside of your OnBarUpdate so that I can try to assist you further.

[NeoAkira]

This is what I have in my Initialize()
The useXXXX are Parameters that are selected by the user before the strategy is started.
barArrayIndex is an Array where I collect all the selected Periods for later usage.

        protected override void Initialize()
        {
            if (collectionFormat == MyDataCollection.PTrend )
            {
                if (use150D)
                {
                    barArrayIndex += 1;
                    Add(PeriodType.Day, 150);                
                    periodBarArray[10,0] = 150;                    // Add Period 150D to Chart
                    periodBarArray[10,1] = barArrayIndex;       // Define BarsArray Number
                }    
                if (use090D)
                {
                    barArrayIndex += 1;
                    Add(PeriodType.Day, 90);                
                    periodBarArray[9,0] = 90;                    // Add Period 90D to Chart
                    periodBarArray[9,1] = barArrayIndex;        // Define BarsArray Number
                }    
                if (use050D)
                {
                    barArrayIndex += 1;
                    Add(PeriodType.Day, 50);                
                    periodBarArray[8,0] = 50;                    // Add Period 50D to Chart
                    periodBarArray[8,1] = barArrayIndex;        // Define BarsArray Number
                }    
                if (use030D)
                {
                    barArrayIndex += 1;
                    Add(PeriodType.Day, 30);                
                    periodBarArray[7,0] = 30;                    // Add Period 30D to Chart
                    periodBarArray[7,1] = barArrayIndex;        // Define BarsArray Number
                }    
                if (use020D)
                {
                    barArrayIndex += 1;
                    Add(PeriodType.Day, 20);                
                    periodBarArray[6,0] = 20;                    // Add Period 20D to Chart
                    periodBarArray[6,1] = barArrayIndex;        // Define BarsArray Number
                }    
                if (use015D)
                {
                    barArrayIndex += 1;
                    Add(PeriodType.Day, 15);                
                    periodBarArray[5,0] = 15;                    // Add Period 15D to Chart
                    periodBarArray[5,1] = barArrayIndex;        // Define BarsArray Number
                }    
                if (use010D)
                {
                    barArrayIndex += 1;
                    Add(PeriodType.Day, 10);                
                    periodBarArray[4,0] = 10;                    // Add Period 10D to Chart
                    periodBarArray[4,1] = barArrayIndex;        // Define BarsArray Number
                }    
                if (use005D)
                {
                    barArrayIndex += 1;
                    Add(PeriodType.Day, 5);                
                    periodBarArray[3,0] = 5;                    // Add Period 5D to Chart
                    periodBarArray[3,1] = barArrayIndex;        // Define BarsArray Number
                }    
                if (use003D)
                {
                    barArrayIndex += 1;
                    Add(PeriodType.Day, 3);                
                    periodBarArray[2,0] = 3;                    // Add Period 3D to Chart
                    periodBarArray[2,1] = barArrayIndex;        // Define BarsArray Number
                }    
                if (use002D)
                {
                    barArrayIndex += 1;
                    Add(PeriodType.Day, 2);                
                    periodBarArray[1,0] = 2;                    // Add Period 2D to Chart
                    periodBarArray[1,1] = barArrayIndex;        // Define BarsArray Number
                }    
                if (use001D)
                {
                    barArrayIndex += 1;
                    Add(PeriodType.Day, 1);                
                    periodBarArray[0,0] = 1;                    // Add Period 1D to Chart
                    periodBarArray[0,1] = barArrayIndex;        // Define BarsArray Number
                }
                
                lastBarArrayIndex = barArrayIndex;
            }
            
            CalculateOnBarClose = true;
        }

And this is a snippet of my OnBarUpdate

        protected override void OnBarUpdate()
        {
            
            if (collectionFormat == MyDataCollection.DTMA16 && BarsInProgress != 0)
            {
                return;
            }
            else if (collectionFormat == MyDataCollection.DTMA16 && BarsInProgress == 0) 
            {
                //The Code in this Part works perfect and I can Collect all Data in my Table
                //from 1993 till today
                 //If this Part is handled there also will be no Additional Periods added to the chart
            }

            if (collectionFormat == MyDataCollection.PTrend && BarsInProgress != 0)
            {
                //In this part is the code for collecting Data from the other Periods
                //Here it collects only Data from 2008 till today

            else if (collectionFormat == MyDataCollection.PTrend && BarsInProgress == 0)
            {
                if ((Count - 2) == CurrentBars[0])
                {
                //This part should be executed when we are on the last bar in the Chart and all Periods
                }
                else return;
            }

Hope this shows you my problem a little bit better and I hope that you can help me.

Another Indicator that it must have something to do with the provider Data is that when I connect to the FXCM Server in my Connection My Tables show Data from 2002 till today. If I connect to The NT-Servers (FXCM) I have only Data from 2008 till today.

Thanks in Advance.

[NinjaTrader_JC]

Hello NeoAkira,

While FXCM's servers may have more Historical Data then NinjaTrader's servers, this is still going to be two separate items not related to each other. As long as you have the Historical Data downloaded to your Local PC, even if the servers do not have Historical Data you will still be able to access the data as long as it you are requesting it in the Data Series.

So your issue maybe one of two different combinations. The first is that you are returning, inside of OnBarUpdate from your different DataSeries.

if (collectionFormat == MyDataCollection.DTMA16 && BarsInProgress != 0)
{
return;
}

You may want to separate your scripts out so that it is easier to understand what is being process as this can be returning and missing some updates. Also, it is going to be easier to debug a simplified script as well.

The other is that you are adding a 150 Bar Series so NinjaTrader has to make sure that there is at least the 150 bar is available before processing the rest of your series as well which 1 bar is almost going to be a year. Adding this series and not processing this until the bar has closed is going to put the results most likely in 1994 since it requires that a 150 bar be present.

If you try this script it may help to understand what is going on.

        protected override void Initialize()
        {
                        Add(PeriodType.Day,150);
			BarsRequired = 0;
			CalculateOnBarClose = false;
        }

        protected override void OnBarUpdate()
        {
			// Checks to ensure all Bars objects contain enough bars before beginning
			Print("CB: "+CurrentBar+" BIP: "+BarsInProgress+" BarsRequired: "+BarsRequired);
			Print ("Time: "+Time[0] );
    		if (CurrentBars[0] <= BarsRequired || CurrentBars[1] <= BarsRequired )
        		return;
			
			Print("Pasted Return");
        }

[NeoAkira]

Hi JC,

While FXCM's servers may have more Historical Data then NinjaTrader's servers, this is still going to be two separate items not related to each other. As long as you have the Historical Data downloaded to your Local PC, even if the servers do not have Historical Data you will still be able to access the data as long as it you are requesting it in the Data Series.

But how can this happen?
If what you say is true it shouldn't change the result of the Table I create. I have the Data local in my History from 1993 till today. But when I connect via NT-Servers my created table has only Data from 2008 till today and when I connect via FXCM Servers my created table has Data from 2002 till today. This is reproducible. I don't change anything else...only change the server in the connection and reconnect.
This tells me that the Add.Period Function isn't working with local Data but with the Data that comes from the Servers of the DataProvider or at least some settings that are provided from the DataProvider. There is no other possibility as I see it.

So your issue maybe one of two different combinations. The first is that you are returning, inside of OnBarUpdate from your different DataSeries.

if (collectionFormat == MyDataCollection.DTMA16 && BarsInProgress != 0)
{
return;
}

You may want to separate your scripts out so that it is easier to understand what is being process as this can be returning and missing some updates. Also, it is going to be easier to debug a simplified script as well.

You are right that I have two loops in the onBarUpdate but they never will be working at the same time since collectionFormat can only have one Value. Either MyDataCollection.DTMA16 or MyDataCollection.PTrend. This means that only one loop is active at any given time. This is selected with a Parameter when you start the Strategy. I'm Pretty sure that they don't interfere with one another. It would be logically impossible and the Loop where collectionFormat == MyDataCollection.DTMA16 is working properly as it should be only the other loop is not working as it should. Actually I'm thinking that it also works properly but the only thing that is not working is the Add.Period Function.

The other is that you are adding a 150 Bar Series so NinjaTrader has to make sure that there is at least the 150 bar is available before processing the rest of your series as well which 1 bar is almost going to be a year. Adding this series and not processing this until the bar has closed is going to put the results most likely in 1994 since it requires that a 150 bar be present.

I also disabled the 150D Period and the only thing it did was that I had 4 to 5 Months more Data in the Table..before was Nov. 2002 now it is June 2002 as the starting point. this can't be the reason that almost 10 Years of Data are missing.
Also the available Data tells me that it must be related to the DataProvider. As far as I know FXCM just has Daily Backdata till 2002 via the normal Download mechanism if I'm not incorrect and also that the NT Servers have only Daily Backdata till 2008.
I could only get the Backdata till 1993 from FXCM with their historical Downloadtool.

If you try this script it may help to understand what is going on.

JC, I really know what I'm doing. I'm in the Software business and a Developer for over 20 Years. I really want to believe you that the Error is on my side but there is ample evidence that it is not (only) my error. I could understand when a few months or maybe one year or two were missing but over 10 Year of Daily Data could not go to nirvana without something else happening.

Also the behavior with the two different Servers should/could not happen if your solution would be correct and the Add.Period Function is using only local Data. How would this possible?

Thanks again for your continued work and I really would appreciate if we can find a solution for this problem.

Thanks in Advance

NeoAkira

[NinjaTrader_JC]

Hello NeoAkira,

FXCM has multiple Historical Data Servers so it may vary but I believe around 2002 is the most common date range yes.

I believe I need to go about this at a different way as I think we are not talking about the same items.

I am not familiar with the FXCM Historical Data Download tool, but looking into it it appears it will just download data that is importable inside of NinjaTrader.

Once imported into NinjaTrader it should be all of the same as accessing data from your local PC.

To try to help me get aligned with you are doing, how are you enabling your strategy so that your data is exported? Are you doing this inside of a chart, strategy analyzer or strategies tab?

Also, since your Strategy has the Primary series as 1 Day, are you able to see your Price Data back to 1993 on the chart no matter if you are connected to FXCM or not connected at all?

Note that to see a chart if you use your strategy inside of the Strategies tab, you may right click on your Strategy, select Strategy Performance, select your strategy, select Real-Time and Historical and you should see a Chart tab.

Happy to be of further assistance.

[NeoAkira]

Hello JC,

FXCM has multiple Historical Data Servers so it may vary but I believe around 2002 is the most common date range yes.

Ok, then I'm at least right about one thing..

I am not familiar with the FXCM Historical Data Download tool, but looking into it it appears it will just download data that is importable inside of NinjaTrader.
Once imported into NinjaTrader it should be all of the same as accessing data from your local PC.

This is correct. I can see all the Data in the Historical Data Manager after I Imported it and I see the Data also in my Chart.

To try to help me get aligned with you are doing, how are you enabling your strategy so that your data is exported? Are you doing this inside of a chart, strategy analyzer or strategies tab?

The Strategy is enabled in a EURUSD Daily Chart. Like I wrote at the beginning the Data Series has a Custom Range from 1990 till today. And all the Data is shown on my Chart. I right click in the Chart and go to Strategies. I Select my Strategy and then select my Parameters and klick on OK. My Strategy starts as usual and if I select the Parameter with the additional Periods I see in the Status Line of the Control Center that it loads the Data (EURUSD). This Loading message does not appear when I deselect "Load Data from Server" in Tools/Options/Data. But the outcome is the same (only Data from 2002 till now or 2008 till now if NT-Servers). After Loading this Chart Data the Strategy collects all the Data I want and creates a csv Table on the Disk. All this Functions are working fine. My only trouble is the missing Data as soon as I Add Periods via Add.Period.

Also, since your Strategy has the Primary series as 1 Day, are you able to see your Price Data back to 1993 on the chart no matter if you are connected to FXCM or not connected at all?

Yes, all Data is shown till 1993 regardless of connection status.

Note that to see a chart if you use your strategy inside of the Strategies tab, you may right click on your Strategy, select Strategy Performance, select your strategy, select Real-Time and Historical and you should see a Chart tab.

As described above I don't use the strategy Tab since It is not really a Strategy...only a Data Collection and Data Export Tool.

Happy to be of further assistance.

Happy to be further a pain in the A...

Thanks for your hard Work and that you still help me to find a solution for this really bothering topic.

[NinjaTrader_JC]

Hello NeoAkira,

It is not going to be the missing any data as you see on the chart it is all there but, when you are accessing it programmatically NinjaTrader has some under-the-hood handling to ensure that the series have enough data.

Could you send me the script so that I may take a look at it to better assist you? If you do not want to post it on our Forums you may send it to me by sending an email to "Support [at] NinjaTrader [dot] com" with ATTN: JC in the subject and the a reference to this thread in the body.

[NeoAkira]

Hello JC,

It is not going to be the missing any data as you see on the chart it is all there but, when you are accessing it programmatically NinjaTrader has some under-the-hood handling to ensure that the series have enough data.

But it is missing Data. You still didn't answer my question that if this all is really so why my Tables show different start dates when I connect to FXCM Servers (Data from 2002 onwards) or to NT-Servers (Data from 2008 onwards). This behavior should not happen if what you say is really the case. As I wrote before I didn't change anything else only changed the Servers in the connection. And if I change them back and for it always changes the starttime of my table. How Do you explain this behavior if the Add.Period is really using the Data from my local History DB at least the startdate should always be consistently the same. Please answer this question as I think herein is the answer to this behavior.

Could you send me the script so that I may take a look at it to better assist you? If you do not want to post it on our Forums you may send it to me by sending an email to "Support [at] NinjaTrader [dot] com" with ATTN: JC in the subject and the a reference to this thread in the body.

I don't know if I can do that since it is for a custommer of mine and I have an NDA signed for his proprietary algorithms. Also there are some special Indicators you would need to run the script.
But the Script is not the Problem since it really performs as wanted. The only Problem is that as soon as I add Periods via Add.Period It is not working any longer with the local Data. I am also sure that I read about this behavior in another thread regarding an MTF Strategy or Indicator. And one of your collegues wrote something like that behavior (That Add.Period uses Data from the DataProvider like that). I just wanted to know if there is an option to disable this behavior.
I also published the relevant Parts of the script regarding this behavior in an earlier post.

Thanks again for your ongoing support.