Huge Performance Results difference between (Backtest) and (Sim or Playback)

[MohammedAmine]

Hello NT Team,

plz don't send me links to other forum posts that answered this same question, because I really read all of them, and applied everything it was written inside and still NO GOOD RESULTS AT ALL

I did everything u said in these posts yet, the backtest is (Great) yet in reality the strategy LOSES Money

at the beginning I thought that it was due to intra bar data unrecognizing by backtester, but the entry is made correctly in both backtest and playback, and I have a fix 4 ticks TP and 4 tick SL, yet the backtester is mentionning the trade as a winning trade, while in playback mode, we lose, then when I go and check the 1 range chart, and 1 tick chart, both of them CLEARLY show that the price didn't even come near the TP level, so why backtester consider it as a winning trade ? isn't it the same data that backtest and playback are reading from ?

Im using 15 range to backtest and execute on realtime, just in case this helps,

in the picrute below you see an example of 1 trade (among millions of trades) that has this problem, and I show you the difference in results between backtest and playback, and I also show you how the 1 range chart moved after Entry point, and I also show you how the 1 tick chart move to show you REAL PROOF that the price didn't even come near the TP level, so why I get it as a winning trade in backtester ? because like this the entire strategy results can be destroyed, and if we take assumptions of backtester to trust the strategy and strategy settings, we might BLOW REAL MONEY and this will get us into serious trouble, so for what serve the backtester if it is giving COMPLETELY WRONG INFORMATION ?

The video link : https://youtu.be/LYHiAGO0xCc

[NinjaTrader_ChrisL]

Hello, thanks for writing in.

Did you add a 1 tick series and submit your stop loss and profit target to this series? Assuming you have read our forum post on adding intrabar granularity to your script, this is the most important part of getting accurate fills in your backtests. Is your strategy using this overload to submit the protective orders and is this overload targeting your 1 tick series for the barsinProgresIndex?

ExitShortLimit(int barsInProgressIndex, bool isLiveUntilCancelled, int quantity, double limitPrice, string signalName, string fromEntrySignal)

Kind regards,

-ChrisL

[MohammedAmine]

Hey Chris,

all what I can say is Thank you very very much, your answer was really helpful, and Im sure it can help others, now we have the Backtest Looking almost EXACTLY as Realtime execution. Thank you once again

I have only 1 more question plz if you can help with it I'll really appreciate it, in the attached image, you see that almost all the trades are identical, and all the trades are having their (Mirror Trades) on the other side, EXCEPT for the one that happened at 3:34 PM on Realtime, it doesn't exist on backtest, please can you tell me why could such thing happen ?

because if it would be for 1 trade per day I wouldn't even care, but I have other backtests vs Realtime, we can endup with 10 trades difference, they exist on Realtime but don't exist in backtest, and this could have a significant gap on the final performance results between Backtest and Realtime, so can you give me a hint on how to fix it plz ?

Thank you

[NinjaTrader_ChrisL]

Hi, thanks for the follow up. I can not tell why this would happen with your custom strategy. You will need to first add Prints to your strategy to see if once scenario is hitting the entry signal and one is not e.g.

if(<entry condition>)
{
Print("Entry condition hit " + Time[0]);
EnterLong();
}

We have a guide on using prints and general debugging information here:
https://ninjatrader.com/support/help...ing_your_ninja script_cod.htm

Kind regards,

-ChrisL

[MohammedAmine]

Hello NinjaTrader_ChrisL ,

Now I noticed when the difference is happening, I get tons of different trades between Playback or Sim and Backtest in times of Super High Velocity and Volatility, usually US Market Open and Close, but the difference is really huge, for example I got only 8 trades on backtest, while I got 30 trades on playback, using the EXACT same everything, data, day, strategy, parameters, time period, time frames .... everything is the same, now I understand that dur to the high speed of candle forming, and (hypothetical latency) that Backtest or Playback could execute trades with could impact the entry or exit timing in each trade, but my question is :

Since they have the EXACT same data, even considering slippage and latency, why Realtime trades are nearly 4x more than Backtest, does backtest has special treatment to trades during times of Impulsive movements ? if so, how to prevent it ?

don't forget that I implemented all the advices you gave me, and on time of medium on low volatility / velocity the trades are almost identical on Backtest and Playback.

Thank you very much

[NinjaTrader_ChrisL]

Hello, thanks for the follow-up. We have a guide here on why your backtest results can differ, and you can also do a deep analysis of your strategy by adding prints within your entry condition to see if the strategy completely misses signals in one scenario versus another one.

e.g. It can be as simple as:

if(<entry condition>)
{
Print("Entry Signal " + Time[0]);
EnterLong();
}

To figure out why your strategy is different in a backtest vs real-time, you must analyze each and every data point that it is using to make a trade, not just the overall results. If you need assistance with your code, a programmer from the NinjaTrader Ecosystem may be hired for closer work with your custom strategies.

[MohammedAmine]

Hey NinjaTrader_ChrisL, Thank you for the answer

Look I found something weird while analyzing charts, look at the candles I pointed there, how can a trade close that up, if the market / price didn't come near that area AT ALL, there are 2 LONG trades there, you can see CLEARLY that the price didn't move up after entry, do you see any signal or indication that could help ?

The market was super fast there, as you can see plenty of candles were formed at the same second, but still, the market didn't go up, otherwise for sure it would have left few traces there (empty candles, or candles with 00000, but we don't see any indications to proove that the price visited those upper levels).

Thank you

[NinjaTrader_ChrisL]

Hi, please try resetting your cached data. Please try this and test the strategy again:

• Shut down NinjaTrader.
• Open the Documents > NinjaTrader 8 > db folder.
• Delete the sub-folder named 'cache'.
• Restart NinjaTrader and test.

Do you see this happening on the SampleMACrossover strategy as well?

I look forward to assisting.

[MohammedAmine]

ey NinjaTrader_ChrisL

Thanks for the permanent support, I really appreciate it , I removed the cache folder, and restarted NinjaTrader, and I run a Backtest and Playback Realtime trading using Sample MA Crossover strategy, and yes it give different results as well.

Take a look at the attached images, we see different results even though the crossover strategy works based on close not on tick, and we have as well a trade that won on Backtest and Lost on Playback, how could it be ?

Also look at the other image in which Playback mode is entering at "WHATEVER" Level, we don't even have a crossover there, so why the strategy enters or exits ?? I really didn't know why

Thanks

[NinjaTrader_ChrisL]

Hi, I am not getting this results, and I am no longer able to get this specific time for testing because it's too far back at this point. I'm going to recommend a complete re-installation of the platform.

1.Close all running applications.
2.Navigate to your Documents folder > Right-click on the 'NinjaTrader 8' folder > Rename > rename the folder to 'NinjaTrader 8 OLD'. DO NOT delete this folder.
3.Uninstall NinjaTrader within Windows Control Panel.
4.Navigate to C:\ > Program File (x86) > Delete the 'NinjaTrader 8' folder if it exists
5.Reboot your PC.
6. Once these steps are completed, download and install NT8 from the link below:

https://ninjatrader.com/GetStarted

If you have any custom scripts in the NinjaTrader 8 OLD folder in the bin/Custom indicator or strategy folders, you can move them over to the respective directory in the new NinjaTrader 8 folder.

[MohammedAmine]

Hey Chris,

I really apologize for asking over and over again, but this stuff doesn't seem to work man, but I think I found a small hint

we have missing footprint values on :

protected override void OnMarketData(MarketDataEventArgs e)

I tracked down strategy behavior as you recommended, and I found that on Backtest, the conditions for entry are met and confirmed, so the Print function actually prints out the correct message, but unfortunately the trade doesn't enter, why ?? I really don't know

while in realtime, the trade enters normal, and it shows the Print message, but it also gets executed

btw, I'm using Range candles on both Realtime and Backtest, "5 Range"

I hope the provided information can help to detect the problem and fix it

Thank you

[NinjaTrader_ChrisL]

Hi Mohammed, The platform does not store any form of Level 2 data unless it is in the form of Market Replay data. If you are backtesting on regular historical data then this would not provide level 2 data in a backtest.

[MohammedAmine]

ok right now i need solution plz, my strategy enters based on range candle POC, I use tick replay, and I use single tick data series, what i want is to replicate what would happen on realtime using backtest, what should i do to accomplish this, should i buy level II data ? if yes, then how ? for me i want to acheive this objective, please guide me how this would be accomplished using NT backtest.

Just to remind that we are using Market Replay which supposed to give us tick data as mentioned on your Doc. From this exact data we build the footprint. So, we have data on most of the candles but on some we do not, it is like "missing", why is that ?

Thanks

[NinjaTrader_ChrisL]

Hello, thanks for the follow-up. I have given you all the information on backtesting that I can possibly give, so I will not post this again. For backtesting a strategy that uses level 2 data, you must use the Playback connection as the strategy analyzer will not read level 2 data. For this missing data, can you be specific about the data that is missing so I can try it out on my PC?

[MohammedAmine]

Hey NinjaTrader_ChrisL,

Thanks for all your answers, and your desire to help, but Im reporting a problem that has not been solved so far, and I really hope that you can try your best to understand the REAL concern here.

here is the process I did :

1 - I run a backtest on NQ Market using NinjaTrader Continium data on day 12 jan 2023 using strategy analyzer, then i got a list of (hypothetical trades and performance results)
2 - I downloaded NinjaTrader Continium Market Replay Data of NQ the same day 12 jan 2023, and I connected to playback, and I run the strategy to trade on a 1x speed, then I got some Trades Performance for the same day
3 - Before doing the above tests, I run a "Live" strategy trading using the same market, same parameters, same settings on the same day, I run it on my computer that has 150ms latency from CQG execution IP address, and of course I got some performance as well.
4 - now to remove this "Latency" issue as much as I could, I bought a VPS that has 2ms latency from CQG IP address, and I run the same strategy, on the same day "LIVE" on simulation again using NinjaTrader Continium connection, and I got some performance for this same day (12 jan 2023)

Now please I need your focus to analyze the following results :

- the VPS and my computer which were executing at the same time on Sim (live) are having EXACTLY identical list of trades, with a slight difference in some trades performance (This is normal due to latency difference sometimes VPS was able to enter before my computer and close before it)

- but the results between VPS and Playback mode (which are supposed to be IDENTICAL as well because playback mode has (The Exact Same Data) from Ninja Continium as you say in your docs, and since it has no latency as well, so it should have ALMOST IDENTICAL list of trades, may be with some slight performance in some single trades, but still the overall list of trades should be EXACTLY the same, but guess what man, THEY ARE FREAKING COMPLETELY DIFFERENT, how could it be ? what is the world could make 2 connections that have the same data provider, same latency, same tick by tick data, same strategy, same day, same settings, same parameters, executed on the same market, at the same period of time to have this (UNLOGIC, UNREASONABLE, UNEXPLAINABLE DIFFERENCE, just WHY ???

- right now I'll not even talk about the difference in results between my computer live on Sim, and Playback mode, because of course the comparison is terrible, since even with VPS which supposed to have similar latency to Playback and results are different, so obviously my computer results as completely different as well.

- now talking about backtest, which is giving results from another PLANET, I really don't know for what it serves if it can NOT give us a good, accurate results to base our assumptions on, Mr. Chris, you said over and over again that we shouldn'tuse Backtest for strategies that use Level 2 data, then for what serve that Tick Replay checkbox that is in Backtest Strategy Analyzer ? one more thing, how am I gona get the Optimum parameters for my strategy if I don't run an Optimization process and get suggestions from it ? but if these suggestions are based on (NOT Accurate Results), then I'll use them on my strategy on live accounts and I will LOSE, so for what the optimization analyzer serve if it will give us suggestions based on (FAKE UNACCURATE RESULTS) ?

Plz I'm honestly asking real questions based on facts and based on Real data and results that I presented to you in details, now please help me with something REAL THAT WORKS, because everything I said makes complete sense, so I need answers that will help me SOLVE THE PROBLEM.

What I ask for is very simple, I want to be able to Backtest my strategy on an accurate tick by tick REAL data, on a long and various period of times to have a clear vision about what to expect as a REAL result when running it live, and I want to be able to Optimize it on various period using the same way, to get accurate settings for my parameters to have it work profitably, is it a strange request ? I think each and every strategy programmer is simply looking for this, simple, clear and straight forward request.

Thank you