Why is Merge Back Adjusted apparently not working?

[timmbbo]

Hi there, I'm trying to test a futures instrument for a year, on Playback; however, no data apparently comes through after the first quarter.

I'm trying to test a futures instrument YM for 2021, on Playback. I start off at 03-21. The data is good until March, when it goes cold. As you can see from screen shot foo1, the chart no longer displays after March 19th 2021, though the Playback is at May 10th 2021.

From screen shot foo2, you can see that I do indeed have the replay data present on my system.

Lastly, from screen shot foo3, you can see I have Merge Back Data turned on.

Therefor, given all of this, I do not understand when I can not get data past one quarter during a year-long reply.

Thanks,

[backtester831]

Did you try using YM 06-21? This makes it sound like the latest contract expiration should be used for merging earlier dates: https://ninjatrader.com/support/help...rge_policy.htm

[timmbbo]

backteste831, thanks. The documentation says the "front" end contract should be used, which I interpret to mean the earliest one.

[NinjaTrader_Emily]

Hello timmbbo,

Thank you for your post.

backtester831 is correct that you should use the "latest" contract. With merge back adjusted as the merge policy, any futures months working backwards from the selected contract month will be merged into the contract. A front month is considered to be the most recent contract date. If you are wanting to playback through May 10th 2021, you should use the YM 06-21 contract that is closest to that date (meaning it was the most recent contract month at that time). For more details on what the term front month is referring to, please see the following publicly available link:

https://www.investopedia.com/terms/f/front-month-contract.asp

If you have any additional questions or concerns, please feel free to post them here.

[timmbbo]

NinjaTrader_Emily

I'm still having problems. When I start off with the front instrument, I still am getting incomplete data.

Here's a screen shot. I'm testing a 12-20 futures instrument, from 12/22/19 to 12/19/20. As you can see, I'm at March 10th 2020, and none of my strategies have activated. This isn't because of the strategy or missing data. If I test the 03-20 futures, and run from 12/22/19 to end of March 2020, then my strategies activate and run.

So, I've tried running the full year of 2020 with both the 03-20 futures and the 12-20 futures, with the Merge Back Adjusted turned on, and in neither case is are the strategies being feed data for the full 2020 year under a single run.

What's up? Thanks,

[NinjaTrader_Emily]

Hello timmbbo,

Thank you for your reply.

It seems your screenshot did not get posted. Please try to post it again. Additionally, when you are seeing that the strategies are not activated, please check the Strategies tab of the Control Center and provide a screenshot of what you see there.

I appreciate your patience and look forward to your reply.

[timmbbo]

Emily, here's the screen shot.

I can run the future across the four different runs of Q1, Q2, Q3, and then Q4, but I've never been able to do a single run that covers all four quarters at once. Thanks.

[NinjaTrader_Emily]

Hello timmbbo,

Thank you for your reply.

I would like you to test the Playback connection with only the SampleMaCrossover strategy running for the 12-20 contract from 12/22/19 to 12/19/20. If you get similar results, please go to Control Center > Tools > Historical Data. Click the + sign next to Market Replay and review the replay data downloaded for the 12-19, 03-20, 06-20, and 12-20 contracts during that time period.

I look forward to hearing the results.

[timmbbo]

Emily, I found the cause of the problem - I needed to copy the data into the last quarter's replay directory.


When I download the replay data, only the dates of the Q1 data is loaded into the 03-21 directory, only the dates of the Q2 data is loaded into the 06-21 directory, and so on for Q3 and Q4. So, when I started the replay with a Q1 date, but a Q4 chart, the replay mechanism was pulling the data out of the Q4 directory (which only had Q4 replay information inside of it.) Instead, when I copy all of the Q1, Q2, and Q3 replay data into the Q4 directory, then everything works.

With the "MergeReplayData" option turned on, I assumed it would pull the needed data out of each Q1, Q2, Q3, and Q4 replay data directories. Instead, it appears that the data is pulled only from the last directory. So, unbeknownst to me (and apparently undocumented), if you want to run a full year replay in a single run, you need to copy all of the replay data into the front replay data directory.

I hope this helps others as well.

[QuantKey_Bruce]

Emily, I found the cause of the problem - I needed to copy the data into the last quarter's replay directory.


When I download the replay data, only the dates of the Q1 data is loaded into the 03-21 directory, only the dates of the Q2 data is loaded into the 06-21 directory, and so on for Q3 and Q4. So, when I started the replay with a Q1 date, but a Q4 chart, the replay mechanism was pulling the data out of the Q4 directory (which only had Q4 replay information inside of it.) Instead, when I copy all of the Q1, Q2, and Q3 replay data into the Q4 directory, then everything works.

With the "MergeReplayData" option turned on, I assumed it would pull the needed data out of each Q1, Q2, Q3, and Q4 replay data directories. Instead, it appears that the data is pulled only from the last directory. So, unbeknownst to me (and apparently undocumented), if you want to run a full year replay in a single run, you need to copy all of the replay data into the front replay data directory.

I hope this helps others as well.

Wow. This should not be. NinjaTrader, were you aware of this? Can it be fixed?

[NinjaTrader_Emily]

Emily, I found the cause of the problem - I needed to copy the data into the last quarter's replay directory.


When I download the replay data, only the dates of the Q1 data is loaded into the 03-21 directory, only the dates of the Q2 data is loaded into the 06-21 directory, and so on for Q3 and Q4. So, when I started the replay with a Q1 date, but a Q4 chart, the replay mechanism was pulling the data out of the Q4 directory (which only had Q4 replay information inside of it.) Instead, when I copy all of the Q1, Q2, and Q3 replay data into the Q4 directory, then everything works.

With the "MergeReplayData" option turned on, I assumed it would pull the needed data out of each Q1, Q2, Q3, and Q4 replay data directories. Instead, it appears that the data is pulled only from the last directory. So, unbeknownst to me (and apparently undocumented), if you want to run a full year replay in a single run, you need to copy all of the replay data into the front replay data directory.

I hope this helps others as well.

Hello timmbbo,

Thank you for your reply.

What is your source for downloading replay data? NinjaTrader only offers 90 days worth of replay data for the most popular instruments. If you are downloading data from 2021, is it coming from a different source? Please provide screenshots demonstrating where you are copying data from and where you are copying it to for the "Q4 directory." Which "MergeReplayData" option are you referring to? Where is that option found in the platform? I would like additional clarification so I may try some tests in my own platform and see if I am able to replicate the same behavior you are describing.

I look forward to your reply.

[timmbbo]

> What is your source for downloading replay data?
The Intentional Trader, at MRDD | Market Replay Data Downloader - The Intentional Trader, a third-party provider of NT replay data that goes beyond 90 days.


> Please provide screenshots demonstrating where you are copying data from and where you are copying it to for the "Q4 directory."
Attached. Here's a screen shot of the data setup of me running a 3 year back test on the 6M futures. Each directory contains one quarter of replay data. Since I'm running a test from 03-21 to 03-23, I had to copy all of the replay data from those three years, into the 03-23 directory. I'm running this test on a 6M 03-23 chart.

> Which "MergeReplayData" option are you referring to?
Attached. Under Options -> Market Data -> Historical

Your thoughts?

[NinjaTrader_Emily]

Hello timmbbo,

Thank you for your patience.

After gaining a deeper understanding of your situation and doing additional digging, what you have described is the expected behavior. The following forum post has an explanation of how rollovers are not automatic in the Playback connection as it is similar to the real-world:

https://forum.ninjatrader.com/forum/ninjatrader-8/platform-technical-support-aa/109673-market-replay-contract-rollovers?p=839759#post839759

In real-time data, merge policy applies to load more historical data on a chart though that would not make a difference in how strategy positions behave. If you have a position in a strategy on one contract, then a rollover happens, you have to manually rollover the position. Whether a merge policy is applied or not, the position itself wouldn't rollover automatically. What you have done by pasting data into one directory is essentially the same workaround the user sledge suggests in the linked thread; they just created a folder for a "continuous contract" with contract months ##-## and that is the instrument they select for playback.

I would be glad to add your vote to SFT-2773 that requests for playback to support continuous contracts or apply the merge policy to realtime/streamed data in playback. Please let me know if this interests you and I will follow up accordingly.

I sincerely appreciate your time and patience.

[timmbbo]

Emily, thanks for the time and effort to get to the bottom of this.

Yes, please add my vote. Being able to test a strategy by replaying back data over a couple of years, in an easy way under a single run, is essential to being able to verify a strategy. This is such an elementary need, I'm a little surprised I'm the first one to bring it up.

Your efforts to get this into a future NT version is appreciated.

[NinjaTrader_Emily]

Emily, thanks for the time and effort to get to the bottom of this.

Yes, please add my vote. Being able to test a strategy by replaying back data over a couple of years, in an easy way under a single run, is essential to being able to verify a strategy. This is such an elementary need, I'm a little surprised I'm the first one to bring it up.

Your efforts to get this into a future NT version is appreciated.

Hello timmbbo,

Thanks for your patience.

Your vote has been added to SFT-2773. Please reference this internal tracking number when contacting Platform Support if you ever have questions regarding this feature request.

When a feature request is implemented, you'll find a description of the new feature in the release notes:

• https://ninjatrader.com/support/help...ease_notes.htm

Thank you for using NinjaTrader.