Different strategy results between NQ and MNQ?

[jaybedreamin]

Hey there,

Can someone more experienced than I help me out with this? I have developed a strategy that has been profitable on MNQ since 2019, when backtesting this strategy (the exact same code) against the full NQ, I get dramatically different results. Any idea why this is?

My understanding is that since the code is the exact same, the entry and exit criteria is the exact same, the profit factor, win % and other stats should be the same as well. Am I incorrect here? The only difference would be that losses and wins are bigger in size due to the multiplier of the full vs micro contracts.

To give a little insight into the strategy. It is a gap based strategy that adds additional contracts (pyramids) into winning trades, and does not add to losers.

What am I missing here? I have excluded commission and slippage to make the comparison easier.

[NinjaTrader_BrandonH]

Hello jaybedreamin,

Thanks for your note.

If we test the SampleMACrossOver strategy that comes default with NinjaTrader, we see about the same number of trades placed when comparing backtest results between NQ and MNQ. See the attached screenshots. Since a different number of trades were calculated, the overall backtest results slightly differ. Note that this strategy is set to use Calculate.OnBarClose.

In the screenshot you shared, we can see there were 1008 trades calculated on the NQ backtest and 1060 trades calculated on the MNQ backtest. Since the number of calculated trades differ, the results in the Strategy Analyzer differ.

Is your strategy set to use Calculate.OnPriceChange or OnEach Tick to execute trades intrabar?

Please review the help guide document on the differences on real-time vs backtest (historical).
http://ninjatrader.com/support/helpG...ime_vs_bac.htm

When in historical data, only the Open, High, Low, and Close will be available and there will be no intra-bar data. This means actions cannot happen intra-bar, fills cannot happen intra-bar. All prices and actions come from and occur when the bar closes as this is all the information that is known.

Because of this, OnBarUpdate will only update 'On bar close' as it does not have the intra-bar information necessary for 'On price change' or 'On each tick' and the script will not have the intra-bar information to accurately fill an order at the exact price and time.

Below is a link to the help guide on Calculate.
https://ninjatrader.com/support/help.../calculate.htm

Additional information about improving the accuracy of a backtest may be found in this NinjaTrader Forum post:
https://ninjatrader.com/support/foru...mance?t=102504

Ultimately, debugging prints should be added to the script to understand how the strategy is behaving. Below is a link to a forum post that demonstrates how to use prints to understand behavior.
https://ninjatrader.com/support/foru...121#post791121

Please let me know if I may assist further.

[jaybedreamin]

Hi Brandon,

Thanks for the reply. The strategy has the calculate setting to OnBarClose and runs on the 5 minute data series. It is the exact same code, entry and exit criteria so I am not sure why there is a ~60 trade difference. So strange. In theory, if it was profitable on MNQ it should also be profitable on NQ, and it is...just the difference in profitability is causing confusion on my end.

A poster on another forum suggested I compare the trades of each backtest individually to dive further into what's going on. Maybe I will try that, but was hoping there was a more obvious solution or something I was missing.

Any additional thoughts or suggestions that don't involve manually investigating each trade would help tremendously.

[QuantKey_Bruce]

Another possibility is that one or the other has some data missing. You could control for this by using the Historical Data tool to re-download the data for both symbols.

[NinjaTrader_BrandonH]

Hello jaybedreamin,

Thanks for your notes.

QuantKey_Bruce is right. You may have some missing historical data for one of the instruments that you are testing on.

You could redownload the historical data for both instruments on the Tools > Historical Data window. The historical data downloaded for these instruments could be compared in the Historical Data window to ensure that you have the same amount of historical data for each instrument.

Historical Data: https://ninjatrader.com/support/help...8/download.htm

Please let me know if you have further questions.

[jaybedreamin]

Thank you QuantKey_Bruce and Brandon,

Your suggestions helped me uncover 1 part of the issue. I was backtesting from 2010-2023. The micro NQ and ES contracts were not available until May 6th, 2019. So that explains the first issue with different results. The full size NQ backtest actually had 13 years of data, while the MNQ only had the last 4.

However, I tried to control for this by changing the backtest dates to 5/2019-12/2023. Still getting different results.

I then checked the historical data and it looks like both the NQ and MNQ symbols have minute data for each contract month from 2019 until now. I will go through them each again and check, but this is really scratching my brain now. Tried with a completely unrelated strategy and getting the same thing between the micros and minis.

To re-download the historical data, can I just input the dates 5/2019-3/2023 and it will re-download all of the data again for both contract types? If there is any missing it will just fill it in?

[jaybedreamin]

Found some interesting discrepancies.

Checked the historical data records for the same day, 5/7/2019 on the NQ and MNQ. It seems as though NinjaTrader has difference minute-based results for these days. MNQ has 1202 records, while NQ has 1365 records.

Can anyone explain the record differences, as well as the price differences? I believe this is what I've been digging for regarding the different results I'm getting during backtesting.

[jaybedreamin]

Further strange findings,

Went through the trade history of both the NQ and the MNQ backtests. Finding discrepancies in entry and exits prices. Look at the first trade that is selected, almost a 9 point different in the entry price. I imagine this phenomena across hundreds of trades will create significant differences, which would explain some of the oddness I am seeing.

Again, it is the exact same code. Entries and exit criteria are the same. The only thing I do is change the instrument and run the backtest again. Why would this happen?

[NinjaTrader_BrandonH]

Hello jaybedreamin,

Thanks for your notes.

While the underlying security is the same, the NQ and MNQ instruments are traded separately. Similarly, different contract months of the same symbol are traded separately. While the instruments tend to move in unison, the actual ticks and volume are completely different between these two instruments. Since the actual ticks and volume are different, you will see a difference in the number of historical data records when comparing NQ and MNQ historical data.

If you would like to know why the backtest results are different, you could write information to a file using a StreamWriter and then compare the information in the file.

StreamWriter reference sample: https://ninjatrader.com/support/help...o_write_to.htm

More information on this topic could be found on the forum thread linked below.
https://forum.ninjatrader.com/forum/...nce#post100192

Please let me know if I may further assist you.