Huge Performance Results difference between (Backtest) and (Sim or Playback)

[NinjaTrader_ChelseaB]

Hello MohammedAmine,

If the data is the same data set, and the conditions are evaluating exactly the same, then the orders will be submitted at the same time.
With 1-tick intra-bar granularity implemented, orders will be accurate to within 1 tick (without regarding slippage) as the individual ticks will be used for the historical fill engine.

The only way to determine what is causing differences, is to write the data and values used in conditions to file and review the output to see what is causing differences.

There is a video that demonstrates this on the forum post linked below.

https://ninjatrader.com/support/forum/forum/ninjatrader-8/strategy-development/100192-comparing-real-time-historical-and-replay-performance#post100192

If the conditions are not evaluating the same, this would imply the data is not the same.

Provide the output and we would be happy to assist with analyzing the data to find exactly what is causing differences.

[MohammedAmine]

Hey NinjaTrader_ChelseaB,

please give me your personal email, I can send you my script to let you check it in depth and find the problem, I can not publicly share it here

Thank you

[NinjaTrader_ChelseaB]

Hello MohammedAmine,

Instead of providing the script, provide the request output text files. The output should be labelled as demonstrated in the video and sample code from the forum page I linked.

As we cannot debug the script on your behalf, it may not be helpful to send the script.

That said, if you insist to do so instead of providing the requested information we will need to assist, you can email our support email scriptingsupport [at] ninjatrader [dot] com. That said, we will still require that you provide the requested output to continue assisting.

[MohammedAmine]

Hey NinjaTrader_ChelseaB ,

Thank you for your explanation and help man, I really appreciate it, I already print a log with every single detail about the trade, When conditions of trade are confirmed, I print all details that should be taking place, like price level in which the trade is supposed to execute, time ....

This is why, yes I insist to send you the script so that you can debug it and dig in it, may be you can find something that I was not able to find, or may be you know something that I don't know, I need to fix this problem, and see that all trades are identical (with slight difference due to slippage and latency) BUT NOT completely different things.

Why you said to contact support ? I didn't get it, btw, if this intervention needs payment, no problem tell me how much and let's proceed, but I need you to do it, since you have extensive knowledge on Backtesting NT strategies and how each test works.

What I want is :

Backtest, Playback, Sim and Real Account to show the SAME RESULTS, as long as they execute the "SAME EVERYTHING", please help me to achieve this, because this is how it should be, and this is how the Backtesting process will make more sense.

Thank you

[NinjaTrader_ChelseaB]

Hello MohammedAmine,

Where you have stated:
"yes I insist to send you the script so that you can debug it and dig in it"

Unfortunately, in the support department at NinjaTrader it is against our policy to create, debug, or modify, code or logic for our clients. Further, we do not provide C# programming education services or one on one educational support in our NinjaScript Support department. This is so that we can maintain a high level of service for all of our clients as well as our associates.

That said, through email or on the forum we are happy to answer any questions you may have about NinjaScript if you decide to code this yourself. We are also happy to assist with finding resources in our help guide as well as simple examples, and we are happy to assist with guiding you through the debugging process to assist you with understanding unexpected behavior.

You can also contact a professional NinjaScript Consultant who would be eager to create or modify this script at your request or assist you with your script. The NinjaTrader Ecosystem has affiliate contacts who provide educational as well as consulting services. Please let me know if you would like our NinjaTrader Ecosystem team follow up with you with a list of affiliate consultants who would be happy to create this script or any others at your request or provide one on one educational services.

To summarize, we would not be able to modify or write any code for you, and we cannot accept payment. If you would like someone to do this for you, you can contact a consultant.


Where you have stated:
" I print all details that should be taking place, like price level in which the trade is supposed to execute, time ...."

Please confirm, you have watched the video in the forum post, that you fully understand the message in the forum post and are understanding what causes differences, and that you have written to file using StreamWriter as demonstrated in the example script on that post.

If so, I am not seeing in this forum thread the post where you have provided the output .txt files that have the time and values used in the conditions with labels. This should look exactly like the output shown in the video.

May I have you confirm in what posts you have provided the two output text files? (one for live, one for historical, or whatever you are comparing)


Where you have stated:
"What I want is :

Backtest, Playback, Sim and Real Account to show the SAME RESULTS"

With 1-tick intra-bar granularity and testing over the exact same data set (meaning same start and end bar and all bars have the same OHLC values) the results will as similar as possible.

Live orders fill on the exchange based on market dynamics and a partner to exchange with. Historical order fills do not have this and instead simulate the fills based on the data provided. If the orders are placed to a 1 tick series, the order fills will accurate to within 1 tick.


If you want to know what is causing differences, and you would like assistance, provide the requested output files. We are happy to assist with comparing the output from the file run live, with the output file from backtest, and show you line by line (using diffchecker as shown in the video) what is different.

[MohammedAmine]

Hey NinjaTrader_ChelseaB ,

I completely understand everything you said, and I assure you I will go through the video, and output the data exactly following the structure that I will see there, but before doing so, please guide me through few steps to make sure I'm not doing something wrong.

Logically, what is the MAIN reason of having a Backtest or a Playback system ? I think we all agree that the MAIN reason for that is to MIMIC EXACTLY what happened in the past, Tick by Tick accurately, so that when we Backtest on it we get a 99% accurate results on "What WOULD have happened" if we put our strategy on LIVE by that time, using that settings ...

since we agree on that, the Backtest should be able to deliver accurate results, and the ONLY reasons that might interrupt that accuracy is the following :

- Having to adapt to a certain behavior of Backtesting Engine is NOT a good reason for me, because Backtest engine should be able to treat Tick by Tick data to give us accuracy, and NOT ask from us to not use accurate data to be able to Backtest it
- Incorrect code
- Slippage and Latency
- Different Tick by Tick Data from what happened in reality during that time, for that market (Data filtration, compression, smoothing ....)


now let's treat them one by one :

- Backtest is not able to treat intrabar granularity, this is not my concern, it should be able to treat it well, to give me accurate results, and I ask you please to show me how, because for sure exist a way to do it, and I assure you that I integrated single tick data serie to make sure I confirm entry conditions on 5 Range bar, and I enter on single tick bar.
- I can say that my code is good, but we will see about that when I send you requested output after I watch the video
- for Slippage and Latency, I will not get COMPLETELY DIFFERENT results, sometimes I get 40 trades per day difference, this would NEVER EVER happen because of latency and slippage, moreover as I said before, I use a server with 2ms latency, so I assure you its not the reason
- Now coming to the most important topic, the question is How to make sure I execute on the "SAME High Quality DATA" that is EXACTLY the same tick by tick data that REALLY happened in LIVE REALTIME ACCOUNT at that period of time ?

NinjaTrader Claims that NinjaTrader Continium is providing High Quality Unfiltered data, and I think you are taking it from Kinetick, well, I bought Kinetick data, and I can assure you that exist a huuuuuuuuuuge difference between the 2, but this is another topic we can discuss when we overcome the current problem.

now what are the steps that I do, that make me THINK that I execute all connection modes on the same data ?

1. First, I go to Documents > NinjaTrader 8 > db and I remove cache, I also remove all temp inside files (I dono why, but just to not use any old data)
2. Then I go to replay folder and I remove everything inside
3. then I go to tick folder and remove everything inside
4. all other folders are empty, minute, day ...
5. now I connect to NinjaTrader Continium
6. then I go to Tools > Import > Historical Data ...
7. I go to Load Tab > Download Scrolling Tab > I choose the intrument (NQ in my case)
8. I uncheck day and minute, and I check only tick, then I choose the last 2 weeks, and I check, Ask, Bid and Last (because I know that Kinetick keeps ONLY 180 days of Tick by Tick data) so this is why I don't download months or 1 Year of data
9. Then I click "Download" button
10. then I go to "Get Market Replay Data" Tab, I choose the same NQ intrument, and I download THE SAME 2 WEEKS, but this time I have NO choice except to download them day by day, and yes I choose 1 day at a time, and Click "Download" Button next to it
11. then I go to Strategy Analyzer, and run a Backtest using my strategy with some settings on it on this 2 weeks period that I just downloaded data for
12. I get some results and print screen them
13. then I disconnect from NinjaTrader Continium, and I connect to Playback Connection
14. I set the playback on "Market Replay"
15. I adapt dates to be exactly the 2 weeks that I just downloaded
16. I load a chart of NS (my intrument here)
17. I add strategy with EXACT SAME settings and EVERYTHING the same
18. I let it to run for the whole day

Now I run the EXACT same strategy and settings and everything for the same day, on live simulation mode

and I get EVERYTHING different from each other, can you please tell me, by following the above process, am I executing on the same data or not ? is it the right way to backtest or not ? am I supposed to get the same trades (may be with slight difference) or not ? I REALLY wish that I'm doing something wrong I really do, but what I DON'T wish is that the Backtest engine is not working good, or is not doing what it is supposed to do, and I really DON'T wish as well that NinjaTrader is giving us (bad quality data to test on) a data that is COMPLETELY different from what happened in Reality.

Please tell me that my process of backtest is wrong, and show me the right way to do a backtest that uses "The Same Data As Happened In Real LIVE Accounts", I'll be more than happy to see my strategy lose on backtest, and lose EXACTLY the same way on Real (at least I will say that my strategy is bad) but to be a winning strategy on ALL Backtesting and Playback and lose on reality it means that the data on backtest and playback (is faaaaaaaaaaaaaaaaaaaaaaar away from what happened in reality) to say the least.

Thank you man, and sorry for the long explanation

[NinjaTrader_ChelseaB]

Hello MohammedAmine,

"the MAIN reason for that is to MIMIC EXACTLY what happened in the past, Tick by Tick accurately"

Backtests with 1-tick intra-bar granularity are accruate within 1 tick. NinjaTrader is not able to simulate an entire exchange and market dynamics. Instead, the orders are filled with the historical data. Due to this, it wouldn't be possible to always be exact, however, the backtest and playback can be within 1 tick when it comes to order fills.

The backtest engine was designed by default to test without intra-bar granularity as this is much much faster at running the backtest and has a much lower performance impact and still gives a general sense of how the script will process data.

As detailed in the forum post on intra-bar granularity, the Order Fill Resolution is a way of enabling this without having to code intra-bar granularity in the script, but can only be used with the Calculate.OnBarClose setting as this cannot be used in conjunction with TickReplay.

If you would like to submit a feature request for intra-bar granularity to automatically be implemented, I would be happy to submit that request. As of now, the developer has to implement it internally if Order Fill Resolution is not used.

"Backtest is not able to treat intrabar granularity, this is not my concern, it should be able to treat it well, to give me accurate results, and I ask you please to show me how, because for sure exist a way to do it, and I assure you that I integrated single tick data serie to make sure I confirm entry conditions on 5 Range bar, and I enter on single tick bar."

Backtest is able to implement 1-tick intra-bar granularity. May I confirm you have read the instructions from the forum post on intra-bar granularity linked in the forum post on comparing results? Did you view the code of the SampleIntrabarBacktest_NT8 reference sample linked from that post?

Can you clarify what in the instructions you would like further information about?


"Now coming to the most important topic, the question is How to make sure I execute on the "SAME High Quality DATA" that is EXACTLY the same tick by tick data that REALLY happened in LIVE REALTIME ACCOUNT at that period of time ?"

My answer is the same for this. Implement 1-tick intra-bar granularity as directed.


"NinjaTrader Claims that NinjaTrader Continium is providing High Quality Unfiltered data, and I think you are taking it from Kinetick, well, I bought Kinetick data, and I can assure you that exist a huuuuuuuuuuge difference between the 2, but this is another topic we can discuss when we overcome the current problem."

Continuum is CQG not Kinetick. The historical and real-time data come from CQG.

Playback data is recorded by NinjaTrader from real-time for the Playback servers and is not guaranteed to come from any specific source.


"and I get EVERYTHING different from each other, can you please tell me, by following the above process, am I executing on the same data or not ?"

Once you have provided the output text files I have requested, we can assist you with seeing if the data is the same or not.

Until then, I would not be able to say.


"Please tell me that my process of backtest is wrong, and show me the right way to do a backtest that uses "The Same Data As Happened In Real LIVE Accounts""

Are you asking where the historical data is coming from?
If you are connected to NinjaTrader Continuum the historical data is coming from CQG and the real-time data is coming from CQG.
If you were connected to another data feed when you downloaded the data, the data came from the data feed you were connected to.
If you imported the historical data from text file, then the data came from wherever you obtained the text file.

[MohammedAmine]

Hello NinjaTrader_ChelseaB,

First of all I want to let you know that I fully read your forum post Link, and Watched the entire video, and downloaded the script that writes the files. I completely understand the process now, and I will try to do it tomorrow and send you, but let me tell you clearly, it is really NOT necessary in my case, and I'll explain why.

in the video, the guy compares the trades because he barely find a difference, he compares trades with very very very slight difference in performance, with less than 100 ms execution difference between trades, this is why, it makes sense to dig deeply and dive in to find the real cause of all these SLIGHT differences.

BUT in MY case, you will be shocked man, look at the attached screenshots bro, we are talking about TONS OF TRADES in difference, HUGE ENOURMOUS GIGANTIC difference that we can SEE CLEARLY with nude eyes just by looking at the trades performance report.

Please I invite you and I invite all readers of this thread to take a look at the attached reports, and compare them, TERRIBLE DIFFERENCE, and btw, after watching the video, I made sure that I download data correctly, and that I do Backtest correctly and that I do everything as it should be done, also I assure you that I implemented the single tick data seri correctly in code, so thanks to your video I made sure that my code is really really clean and correct.

We talk about 60 trades difference between playback and sim realtime, that's super crazy man, super crazy

Ps : on playback image, you can start checking trades from the yellow line, this is the time I started SIM, also for the VPS timing difference with my computer, it is normal because you compare CT to GMT+1

I think right now we look at facts, please after checking the attached pictures, let me know if you can see something that I can NOT see, and tell me if you still need the files.

Thank you

[NinjaTrader_ChelseaB]

Hello MohammedAmine,

My guess is that the data is different.

With the backtest is there data on the chart? Do you have code in the script to not process in State.Historical?

The output text files will tell all, even the questions above.

[MohammedAmine]

Hey NinjaTrader_ChelseaB,

1 - how could it be different data if I do EXACTLY as you do in youtube video you sent me, I follow the EXACT steps to do tests on playback and backtest

2 - yes there is data on backtest chart

3 - yes I have

        protected override void OnBarUpdate()
        {
            try
            {
#if REALTIME
                if (State == State.Historical) return;
#endif

                if (BarsInProgress == 0) ProcessPrimaryBars();
                if (BarsInProgress == 1) ProcessSecondaryBars();
            }
            catch (Exception ex)
            {
                Log("Error in RTStrategy.OnBarUpdate => " + ex.Message, LogLevel.Error);
            }
        }​

Thanks

[NinjaTrader_ChelseaB]

Hello MohammedAmine,

if (State == State.Historical) return;

This means do not process historical data. You wrote into the logic of your script for it to not be back-testable and to behave different in historical than in real-time.

The data can different, the output will tell us if it is. Notice in the video I'm using a custom trading hours template to limit the data that will load, so that I can ensure its the same and the real-time doesn't include historical or vice versa.

Also, it doesn't appear you need that try and catch. Try and catches are generally used to keep application from crashing due to an unhandled exception. That isn't going to happen in OnBarUpdate().

The preprocessor commands are not supported at all.

I look forward to receiving the output text files so that I may assist with analyzing the data.

[ntmehdi]

hi finally i test my strategy based on historical data

now can i run my automatic strategy on ninjatrader based on footprint imbalances or not?

if yes how can i have access to realtime data of cme market?
should i buy data feed seprated from ninjatrader or ninjatrader subscription is enough for real time cme data?

[NinjaTrader_ChelseaB]

Hello ntmehdi,

now can i run my automatic strategy on ninjatrader based on footprint imbalances or not?

In real-time, such as on a live connection or on Playback with Market Replay data, a script will have access to OnMarketData() and OnMarketDepth().

if yes how can i have access to realtime data of cme market?

Real-time data can be subscribed in the Client Portal.

On the NinjaTrader Client Portal, we have many features and tools you can leverage for:

• Accessing your statements, tax forms, current margin requirements and fee structure.
• Updating your data subscriptions
• Managing your funds via ACH or wire transfers
• Accessing account reports
• Resetting your Live Trading Password

For more info on everything you can do in our Client Portal check out our Help Center.

Please click here if you've forgotten your Password:

• NinjaTrader Client Portal Password Reset

should i buy data feed seprated from ninjatrader or ninjatrader subscription is enough for real time cme data?

NinjaTrader does provide data for CME Future products.