SP500 Daily 200 MA and a Futures Tick-based Strategy

[NinjaTrader_JoshP]

David,

Unfortunately I am out of ideas since I cannot reproduce on my end with the exact same parameters as you. Try disconnecting from all data feeds before running a backtest. Other than that all I can suggest is try a 100% fresh reinstall.

[NinjaTrader_Ben]

Hello,

Someone will reply to your post on Monday. Sorry for the delay.

[dbw451]

Bertrand,

My clock was synced on 12/31 and I've rebooted a few times since then while working on this thread's issue. The tick data I'm using for backtesting is all historical. I do not have my live broker feed turned on.

Regards,

David
dbw451

[NinjaTrader_Bertrand]

Hi dbw451, just a quick idea - did you try synching up your PC clock and then restarting the PC and NinjaTrader? Can make a big difference when working with tickdata...Have a great weekend!

[dbw451]

Josh,

I'm at a loss of ideas also. I have Ninja v6.5.10000.8. To recap:

1. We have the same Ninja version.
2. We have the same strategy.
3. We have ES tick data from the same source.
4. The daily ^SP500 data might be sourced differently (mine's from OpenTick and Yahoo), but I cannot see how the daily data would make a difference.

I did notice something interesting with my strategy Initialize() procedures, but I'll start a separate thread because I'm not sure if it relates to this daily SMA calculation issue or not.

Regards,

David
dbw451

[NinjaTrader_JoshP]

Backtested on ES 12-08 tick 500. My tick chart only goes back a month.

0 12/18/2008 7:53:30 PM 1203.56765 891.5
0 12/18/2008 11:59:17 PM 1203.56765 889.25
1 12/18/2008 12:00:00 AM 1201.47235 885.28
0 12/19/2008 1:06:44 AM 1201.47235 891.5
0 12/19/2008 2:19:42 AM 1201.47235 889
0 12/19/2008 3:33:34 AM 1201.47235 887.5
0 12/19/2008 4:11:13 AM 1201.47235 886.75
0 12/19/2008 4:42:19 AM 1201.47235 887.75
0 12/19/2008 4:56:00 AM 1201.47235 889

Line in red is the SP500 updating on daily chart. You can see the SMA updated to 1201. All subsequent calls from 500tick ES acquires the correct value of 1201.47 as determined from 12/18. A check on the daily chart of ^SP500 and a SMA(200) confirms the value at 12/18 was 1201.47.

Also, in your screenshot you are comparing the wrong thing. At 12/05 your prints should return you the 12/04 SMA(200) value, not the 12/05 value. The 12/05 value is not known till after 12/05 is over.

As far as why yours is off and mine is not I am out of ideas. Please ensure you are on 6.5.1000.8. If you are please try a fresh reinstall. Do not forget to clear your browser cache when you redownload NinjaTrader.

[dbw451]

Josh,

I get similar results to my previous post (see attachment). I now suspect that the issue could be either:

1. the amount of tick data (i.e. 3 months) in the ES futures contacts.
2. using a tick timeframe instead of a minute based timeframe.

Does your AAPL have more than 200 days of data? Maybe you could replicate the issue if you create an ES 12-08 instrument and download tick data from the Ninja historical data servers. Then back test using a 500 tick timeframe of ES 12-08 contract.

Regards,

David
dbw451

[NinjaTrader_JoshP]

David,

I reran with multi-instrument, multi-time frame. No problem.

Series 0 = 1min AAPL
Series 1 = 1day ^SP500


The chart on the left is daily chart of ^SP500. The print out is from the strategy as ran from the Strategy Analyzer on 1min AAPL. You can see it prints the value of SMA(BarsArray[1], 200)[0] perfectly. On 12/18 you get the proper 12/17 SMA 200 of ^SP500.

On your chart you have a bunch of extra things. I do not know what those are and for simplicity sake please just try a very basic ^SP500 daily chart. Throw SMA 200 on it and then compare. Everything matches up on my end.

[dbw451]

Now I know why I've been having problems communicating. The strategy should be multi-instrument and multi-timeframe. The daily SP500 data is a different and separate instrument than the ES contract tick data. Each ES contract instrument has 3 months of tick data. The daily SP500 data has 50+ years of history.

I changed the Add line in your strategy include the SP500 instrument:

Add("^SP500", PeriodType.Day, 1);

The results I get are similar to my original strategy (see attachment).

Regards,

David
dbw451

[NinjaTrader_JoshP]

David,

I cannot comment on your exact code, but on my end it works perfectly. Please look at this screenshot. It works as expected.



On 12/30 I get the SMA(200) value of 12/29. The value is accurate. I checked several different days and they are all correct.

In the screenshot you can clearly see the switch from BarsInProgress from 0 to 1. The update of the daily bar series happens on 1 and then you can see all subsequent calls to SMA(BarsArray[1], 200)[0] gets ammended as well.

Find the strategy attached. I suspect you are still comparing apples to oranges. The reference I have put together is as simple as it gets and it works as expected, suggesting that there is no problem with NT's handling of this scenario.

[dbw451]

Thanks Josh. I understand how NinjaTrader references the data values between different timeframes in a strategy. The help file on this subject is well written and very clear. Thus, I understand that intraday data on 12/30 would be referencing the daily 200 SMA calculated at the end of 12/29.

A 200 day SMA changes very slowly. Referencing the chart I attached in my previous post, you can eaily see that the 200 day SMA is not changing more than +- 2 ES points a day. However the calculated SMA(BarsIndex[1],200)[0] in my strategy example is over 113 ES points different than the correct 200 day SMA value. That difference is much more than (using your example) referencing the 12/29 200 SMA value intraday on 12/30.

I think that the SMA(BarsIndex[1],200)[0] function in my strategy is not looking at 200 days of data. I think it is looking at the number of days since the tick data starts flowing. If you look at the SMA indicator code, it does not wait for the input number of 'periods' to elapse before it starts calculating values. The SMA indicator starts calculaing the simple moving average at CurrentBar == 1. Thus at 30 days, the SMA value is a 30 day moving average and at 60 days, it is a 60 day moving average.

I added a 30 day SMA indicator on the daily SP500 chart to approximate the one month time period I use in my strategy. The 30 day SMA value on Feb 29 (the date referenced in my previous posted chart) is 1351. While this is not exactly the 1356 calculated by SMA(BarsIndex[1],200)[0] in my example, it's close enough to add some validity to my hypothesis that the SMA(BarsIndex[1],200)[0] function is not looking at 200 daily bars in my strategy. If it were, the calculated SMA would be closer to 1470 which is almost 9% larger than 1351.

If I'm understanding you correctly, you're telling me that SMA(BarsIndex[1],200)[0] should give me the correct 200 day SMA value, just one day offsetted. As detailed above, that is not the behavior that that I see.

Regards,

David
dbw451

[NinjaTrader_JoshP]

David,

You need to compare the correct values. Consider this. The daily bars only update once per day. Chronologically this update happens at the end of the day.

For simplicity sake let us consider a 1 min and a 1 day series. Tick series would make no difference.

12/29/2008 4:00 PM 1min
12/29/2008 4:00 PM 1day
12/30/2008 9:30 AM 1min
12/30/2008 9:31 AM 1min
12/30/2008 9:32 AM 1min
....
12/30/2008 3:59 PM 1min
12/30/2008 4:00 PM 1min
12/30/2008 4:00 PM 1day

That is the chronological order in which bars update. Now consider the value when you try to do SMA(BarsArray[1], 200)[0] on 12/30. The daily chart has not updated because it is not the end of the day. You do not know the 12/30 SMA value. Instead when you access it you know the most recent daily update value and that is the 12/29's SMA. This is how it works and that is what happens from a chronological standpoint during backtesting. Since you cannot update the 1 day time frame intrabar there is no way to get 12/30 from a 1min time frame.

[dbw451]

This thread started with almost exactly what's being suggested. In the first post of this thread, I posted the code:

protected override void OnBarUpdate()
{
if (BarsInProgress == 1)
{
daySMA200 = SMA(Close,200)[0];
....
}

which is essentially the exact same thing as using:

SMA(BarsArray[1], 200)[0]

I cannot access the SMA 200 data because the code SMA(BarsArray[1], 200)[0] does not calculate the correct 200 day SMA values. Here is a specific example:

ES 500 tick data series
From Date: 01/29/08
To Date: 02/29/08

For February 29, 2008:
SMA(200) on the ^SP500 daily chart = 1469.80 (correct value)
SMA(BarsArray[1], 200)[0] in strategy = 1356 (incorrect value)

Attached is a snapshot the daily ^SP500 chart.

From this thread, I thought I understood that a 200 day SMA would not calculate correctly unless all the input series had 200 days of data loaded in the strategy.

I guess my mindset is different than 'most'. I don't think of fundemental market information like an SP500 200 day SMA, weekly T-Bill rates, T-Bond yields, CPI numbers, etc. as indicators calculated from my trading charts. They are long term market data used for biasing. I think of my trading chart data (i.e. tick data) as being totally different.

I simply want to reference long term bias numbers in my small timeframe strategies.

Regards,

David
dbw451

[NinjaTrader_JoshP]

David,

The issue is it does not make sense to process your daily data when your tick data is not present. It may not be an issue for your scenario, but in general most strategies calculate based off of all time frames used and as such the logic only makes sense when it is calculated from all the time frames. When you start the strategy when only one time frame is present it completely throws off the logic.

Also, I guess I do not understand why you cannot access your SMA 200 when you need it on the tick data. Just do SMA(BarsArray[1], 200)[0] and it will give you the SMA of the daily data as determined when your tick data starts "flowing".

[dbw451]

In my case, the highest granularity is the daily and my primary input series is a much smaller tick based series. The daily series only fires off once each time the day changes. The tick based series fires off 50+ times each day.

While this information is interesting, it's not really this thread's issue. The issue is the requirement that the time length of all input series (i.e. the Start Date and the End Date) in a strategy be the same. I am unable to calculate a long term daily indicator on a secondary input series without expanding the time length of the primary data series loaded in a strategy. The two problems with expanding the primary data series are:

1. Tick data is very large and NinjaTrader runs out of memory when large tick data sets are loaded.
2. Most historical data sources for futures tick data provide the data in 3 month contracts.

In my previous post, I made an enhancement suggestion that could enable NinjaTrader users to calculate long term indicators from a strategy that is based on a much smaller timeframe. A 200 day moving average is a very common indicator and is referenced by day-traders that use all sorts of intraday timeframes. There should be a way to calculate a 200 day moving average from historical daily data in a strategy without loading 200 days of tick data of which 170 days will not be used for anything. My enhancement suggestion could be nonsense because of ignorance, but hopefully the requirement is understood.

Regards,

David
dbw451