Where does tick data reside for multi-year playback?

[timmbbo]

Folks, I need clarification on where the data should exist for a multi-year Backtest or Optimization.

You'll see in a previous support thread about doing a multi-quarter playback I posted, I came to determine all of the playback data needs to reside within the last quarter's data, to do a full run. For example, if I running a backtest on ES from Jan1 2023 to Dec31 2023 within a single run, that means I need to copy of the ES 03-23, ES 06-23, and ES 09-23 playback data into the ES 12-23 data. I'm wondering if the Backtest/Optimization data schema is equally bent?

For my test, I have the April 2033 and Nov 23 data for GC only. (Screen shot foo6). That is, I have only the GC 06-23 and GC 12-23 directories with corresponding data under my /tick directory.

When I run a test on GC 06-23 from April 1 to April 30 2023 on 06-23 GC and run only April23 I get a total net profit of $2640 (screen shot foo1)
When I run a test on GC 12-23 from Nov 1 to Nov 30 2023 on 12-23 GC and run only Nov23, I get a next profit of $820 (screen shot foo2)

Now, when I run a test from April 1 to Nov 30, I should get a profit of $3460, right? Well, that's not the case. When I run both months under one run, I instead get a profit of $3740 (screen shot foo3)

If I remove the data from GC 06-23 and place the April and Nov 23 data into the 12-23 GC directory, and run both months under one run, I get a total of $4500 (screen shot foo4)

If I remove the data from the GC 12-23 and place April and Nov 23 data into 06-23 GC directory, and run both months under one run, I get a total of ($740) (screen shot foo5)

So, all of this leaves me confused:

- I'm using the exact same data, but my results are significantly different as to where the data is placed.
- If I want to run a backtest over several years in a single run, should each quarter's directory contain data only for that quarter? Or, just like when I run a multiple year playback, I have to shove all of the data into the last quarter's directory?
- What should my backtest data placement be under a multi-year time period under a single run, to make sure I have the most accurate results?

Thanks for your clarity.

[gaz0001]

I dont believe NinjaTrader has any way of doing multiple contracts anymore.

Previously the supported way was to use folder NQ ##-##.

Last time I had strange results and reached out to support who advised me this ##-## technique was no longer supported.

Interested to find out if there is a new method.

[NinjaTrader_Jesse]

Hello timmbbo,

The Playback and Strategy analyzer tools use different types of data. In a backtest you would be using Historical data, for the time period you are testing you would need to download historical data specifically. In playback you can use Playback data which is recorded realtime data, that is different than historical data.

When testing different contract dates that will change the data which the strategy is using for evaluating conditions, we should in general not expect similar results because the tests were not identical. If you want to know specifically what caused those differences you can add debugging into the strategy, we have a guide on how to go about doing that below.

For the questions you asked those would really be up to you and your goals, I cannot provide any recommendations on what specifically to test or what settings to use. The backtest/playback results are a direct result of how the strategy processed over that specific dataset. When comparing realtime vs historical there is an expectation that there will be differences which can be caused for quite a few different reasons. I have linked some resources below that go over comparing different types of tests and common differences that you will see. Regarding the data, you can import data for each contract and if you wanted to test over a long period of time you just have to select the current contract, the platform merges contract data into continuous data by default. For playback data you would need to select the correct contract and have downloaded data for that contract to play it back.

https://ninjatrader.com/support/helpGuides/nt8/discrepancies_real-time_vs_bac.htm

https://support.ninjatrader.com/s/article/Developer-Guide-Debugging-using-Print-and-TraceOrders?language=en_US

https://support.ninjatrader.com/s/article/Developer-Guide-Comparing-Strategy-Results?language=en_US

https://ninjatrader.com/support/helpGuides/nt8/merge_policy.htm

[timmbbo]

Jesse, I appreciate your response, but I need to understand this issue at a detailed level. So let's break this down:


> I cannot provide any recommendations on what specifically to test or what settings to use.
My goal is simple - to run an as accurate as possible backtest over multiple years within a single run!

Here's what I don't understand:

- I do backtest on Q2 and get result $X
- I do another backtest on Q4 and get result $Y

Now, I do one multi-quarter run on both Q2 and Q4 together. My result should be $X + $Y, right? If not, then how do I configure a backtest to do Q2 and Q4 under a single run to get as close to $X + $Y as possible?

Thanks,

[NinjaTrader_Jesse]

Hello timmbbo,

To run a long test you can select the current contract and then specify a start and end date. You can read about how the merge policy works in the link in my last reply.

When doing backtests on two different contracts those are two different sets of data, we should not expect identical results because the data was not exactly identical. We should only expect identical results in 1 situation where all settings and data was exactly the same between the two tests.

Now, I do one multi-quarter run on both Q2 and Q4 together. My result should be $X + $Y, right? If not, then how do I configure a backtest to do Q2 and Q4 under a single run to get as close to $X + $Y as possible?

No, these are separate tests which the strategy can evaluate differently between. If you run Q2 alone the strategy used only Q2 data to calculate conditions. If you use Q2 and Q4 the test now has additional data and offsets so the strategy will calculate a new result based on that specific dataset. If you now test only Q4 that won't match either of the other tests because that is again an individual dataset but without offsets applied. To know exactly why those results are different you would have to debug the strategy using prints to see how it differs in each use case. That is the link I provided in the last email that goes over comparing strategy results, that same type of test can be done for two historical tests rather than comparing realtime to historical.

[timmbbo]

Jesse, thanks again for your reply - it's helpful.

I understand your suggestion of using print statement to diagnose, but it doesn't scale well. If I have dozens of strategies that I want to run on multiple markets, I can't spend an hour on each one diagnosing inaccuracies through print statements. I need to develop one reliable BackTest/Optimization methodology that I can roll out to run hundreds or thousands of strategies to test through and find the winners.

You reference the Merge Policy, and the Offest treatment as the reason for my differences. When I run a single backtest over Q2/Q4, whether my Merge policy is set to Do Not Merge, Marge back adjusted, Merge non back adjusted, I get the exact same results. Should different Merge policies yield different results when running a Backtest?

[NinjaTrader_Jesse]

Hello timmbbo,

Besides using a print there is not really anything else which will give you any specific insight on the differences, asking on the forum is unlikely to be helpful because we can't debug the strategy for you to tell you why it did that. Prints are really the first step at finding out how your logic works in each use case and is a basic requirement of testing strategies in general. If you have dozens of strategies I would suggest to stop and test only 1 strategy until you are completely satisfied and know how it works before moving on to the next. If you are trying to test many strategies at once and don't know how they all logically evaluate in every use case you will just be setting yourself up for a much more complicated process of finding out what is happening.

If you don't have time to invest in developing and debugging your scripts we would suggest outsourcing that work to a third party developer who specializes in making strategies. If you would like further information on third party developers please let me know.

In your example the merge policy is likely being used in one of your tests or possibly multiple. Merging and back adjusting is the default setting so any time you include a date that exists in a previous contract from what is selected the data is merged. In your test that combines Q2 and Q4 that is merged data if you are using the default setting. Regardless of the policy you are still dealing with 3 unique datasets in those tests so we should not expect similar results.

[timmbbo]

Jesse, thanks. I want to wrap this up with one last question, to bring this into application. Say it's your goal to do an Optimization on SampleSMA on the ES, from 01-01-2020 to 12-31-2023. Your goal is a quality optimization producing profitable candidates with high likelihood of mimicking future behavior. The sequence of steps and parameters

Go Tools->Historical Data->Download, and download the ES data from 01-01-2020 to 12-31-2023
Set Tools->Options->Market Data to Merge Back Adjusted
Within the Strategy Optimizer,
- Set start date to 01-01-2020
- Set end date to 12-31-2023
- Set instrument to ES 12-23
- Set Type, Order fill resolution, and other parameters according to your tastes

Then, hit the run button.

Would this be correct? Thanks,

[NinjaTrader_Jesse]

Hello timmbbo,

The analyzer will automatically download data as long as you are connected so you can skip the first two steps. The merge policy only needs to be set once and then left alone. if you are changing the merge policy setting you do need to reload all historical data for the instrument being used in a chart before running tests.

Using the current contract for the ES with merge back adjusted will allow it to look back at the previous contracts data, you can see an image of how that works in the page I linked previously.