Strategy Testing: Different Results using SIM101 vs. Playback

[Dr Kerry]

I have been working on a strategy and testing it using both the real time data feed from the Ninja Continuum in my SIM101 account and also with the recorded data in playback. I let it trade for an entire session from 9:30 AM through 4 PM EDT using the SIM101 account. I also, in the evening, download the data for the session from Ninja and use this data to test the strategy in playback. I have noticed that sometimes the SIM101 results and the playback results for the same exact session with all of the exact same settings produce vastly different results in terms of P&L and many of the other parameters listed on the Trade Performance report such as the number of trades. Other times they correlate quite nicely. To some extent this seems to be dependent on the characteristics of the market on any given day.

What would be the primary reasons for discrepancies in the Trading Performance results of a strategy when using SIM101 vs. Playback given all settings and time frames are identical? Why does this seem to depend on the market characteristics for a given trading session?

Lastly, how can I assure myself, as best that I can, that either of these, SIM101 and Playback, are going to be reasonably accurate representations of the actual trading results with real money at stake?

Thank you for your inputs and suggestions. They will be very much appreciated.

Kerry

[NinjaTrader_Emily]

Hello Kerry,

Thank you for your post.

So I may accurately assist you, please answer all of the following questions:

• What version of NinjaTrader are you using? Please provide the entire version number. This can be found under Help -> About (Example: 8.?.?.?)
• Who are you connected to for data when testing in real-time on the Sim101 account?
• What instrument symbol (and expiry if applicable) have you selected? For example, ES 03-22, AAPL, EURUSD, etc.
• What interval is selected? For example, 5 minute, 1 day, 2000 volume, 4 Renko, etc.

I look forward to your reply.

[Dr Kerry]

Hello Emily,

My responses to your questions are below.

Thank you for your guidance.


What version of NinjaTrader are you using?
I am using version 8.1.1.6 64-bit

Who are you connected to for data when testing in real-time on the Sim101 account?
Ninja Continuum

What instrument symbol (and expiry if applicable) have you selected?
NQ 09-23

What interval is selected?
I am using 4 minute candles.

Thank you very much.

Regards,

Kerry

[NinjaTrader_Emily]

Hello Kerry,

Thank you for your reply.

First, I see that you're using an older version of NinjaTrader Desktop. To update NinjaTrader, please visit the link below for steps on how to update to the latest version of NinjaTrader Desktop.
Note: Updating NinjaTrader Desktop will not affect any custom configurations, workspaces, or add-ons.
https://support.ninjatrader.com/s/ar...language=en_US

Next, be sure that you are comparing the real-time strategy results from each of these tests, which should yield the most accurate comparison of results. This is because when running a strategy in simulation on real-time or Playback data, the fill price is based on the incoming market data and volume. If you are looking at historical results at all, this would be the results of a backtest run for that period of time that only uses OHLC data for fill prices and strategy actions that are processed on the close of each bar. You may view real-time strategy performance by right-clicking the chart where your strategy is applied, hovering over Strategy Performance > your strategy > Real-Time. For more info, see "Viewing strategy performance of a NinjaScript Strategy applied to a chart" on the following page:

• https://ninjatrader.com/support/help...t_strategy.htm

For more information regarding the discrepancies between real-time results (such as simulation in real-time or playback results) vs. Backtest (historical) results, please see the following page:

• https://ninjatrader.com/support/help...ime_vs_bac.htm
  • This page also contains some information about the differences in how an order fill is determined as well as the fill prices of orders between historical/backtest results, simulation/playback results, and real-time live brokerage trading. Understanding these differences is useful for knowing which results might be the most accurate representations of live trading.

If you have a scenario where the results between simulation trading on real-time data and then playback trading on that same day's data are different, you could compare the results from each strategy performance window to see where the discrepancies lie. Programmatically, you could enable Trace Orders within your strategy in State.SetDefaults and even add additional Print() statements throughout your strategy to understand its behavior. Then, you could compare the output from both scenarios and see what is different between the two. For more information on TraceOrders and using Prints() to debug your scripts:

• https://ninjatrader.com/support/help...aceorders2.htm
• https://ninjatrader.com/support/foru...ing#post791121

Please feel free to reach out with any additional questions or concerns.

[Dr Kerry]

Hello Kerry,

Thank you for your reply.

First, I see that you're using an older version of NinjaTrader Desktop. To update NinjaTrader, please visit the link below for steps on how to update to the latest version of NinjaTrader Desktop.
Note: Updating NinjaTrader Desktop will not affect any custom configurations, workspaces, or add-ons.
https://support.ninjatrader.com/s/ar...language=en_US

Next, be sure that you are comparing the real-time strategy results from each of these tests, which should yield the most accurate comparison of results. This is because when running a strategy in simulation on real-time or Playback data, the fill price is based on the incoming market data and volume. If you are looking at historical results at all, this would be the results of a backtest run for that period of time that only uses OHLC data for fill prices and strategy actions that are processed on the close of each bar. You may view real-time strategy performance by right-clicking the chart where your strategy is applied, hovering over Strategy Performance > your strategy > Real-Time. For more info, see "Viewing strategy performance of a NinjaScript Strategy applied to a chart" on the following page:

• https://ninjatrader.com/support/help...t_strategy.htm

For more information regarding the discrepancies between real-time results (such as simulation in real-time or playback results) vs. Backtest (historical) results, please see the following page:

• https://ninjatrader.com/support/help...ime_vs_bac.htm
  • This page also contains some information about the differences in how an order fill is determined as well as the fill prices of orders between historical/backtest results, simulation/playback results, and real-time live brokerage trading. Understanding these differences is useful for knowing which results might be the most accurate representations of live trading.

If you have a scenario where the results between simulation trading on real-time data and then playback trading on that same day's data are different, you could compare the results from each strategy performance window to see where the discrepancies lie. Programmatically, you could enable Trace Orders within your strategy in State.SetDefaults and even add additional Print() statements throughout your strategy to understand its behavior. Then, you could compare the output from both scenarios and see what is different between the two. For more information on TraceOrders and using Prints() to debug your scripts:

• https://ninjatrader.com/support/help...aceorders2.htm
• https://ninjatrader.com/support/foru...ing#post791121

Please feel free to reach out with any additional questions or concerns.

Emily,

Thank you very much for this very helpful and complete response. I very much appreciate all of the fine details that you included.

Just a quick follow up question. What version of C# is NinjaScript built upon?

I am asking this because I can't seem to get the range operator ".." to work with an array or Series<Double> in Ninja Script.

For example, writing Data[0..5] should return the first five elements of the array Data but NinjaScript does not seem to recognize the range operator. Would you know why this would be?

From what I have read this range operator was introduced in 2019 in C# version 8.

See https://zetcode.com/csharp/array/ (Scroll down to the "Array Slices" topic).

Thank you again!

Regards,

Kerry

[NinjaTrader_Emily]

Hello Kerry,

Thank you for your note.

NinjaScript is written in C# 5 targeting .NET 4.8. Due to the discrepancy in versions, the operator ".." is not expected to work in NinjaScript.

Please don't hesitate to reach out with any additional items we may assist you with.

[Dr Kerry]

Hello Kerry,

Thank you for your note.

NinjaScript is written in C# 5 targeting .NET 4.8. Due to the discrepancy in versions, the operator ".." is not expected to work in NinjaScript.

Please don't hesitate to reach out with any additional items we may assist you with.

Hello Emily,

Are there any plans or a schedule to update NinjaScript to the latest version of C#? There are certainly many more language features that are not currently available to us that should be.

Regards,

Kerry

[NinjaTrader_Emily]

Hello Emily,

Are there any plans or a schedule to update NinjaScript to the latest version of C#? There are certainly many more language features that are not currently available to us that should be.

Regards,

Kerry

These are features that the development team certainly considers, though I would not be able to provide any insight regarding current plans or schedules for when these features may be implemented. If it interests you, I would be glad to add your vote to an existing feature request to update the C# version used for NinjaScript so we may track additional interest. As with all feature requests, interest is tracked though we are not able to offer an ETA or promise of fulfillment. If implemented, it will be noted in the Release Notes page of the Help Guide here:

• https://ninjatrader.com/support/help...ease_notes.htm

Thank you for using NinjaTrader. Please let me know if I may be of further assistance.

[Dr Kerry]

These are features that the development team certainly considers, though I would not be able to provide any insight regarding current plans or schedules for when these features may be implemented. If it interests you, I would be glad to add your vote to an existing feature request to update the C# version used for NinjaScript so we may track additional interest. As with all feature requests, interest is tracked though we are not able to offer an ETA or promise of fulfillment. If implemented, it will be noted in the Release Notes page of the Help Guide here:

• https://ninjatrader.com/support/help...ease_notes.htm

Thank you for using NinjaTrader. Please let me know if I may be of further assistance.

Hello Emily,

Please do add my vote to request that NinjaScript be updated to the very latest version of C# and, furthermore, to continually keep the technology updated to the very latest version. The range function that I alluded to in the above post was introduced in C# version 8 back in 2019, four years ago, and we still can't even use that today when so many operations in NinjaScript depend upon efficiently manipulating either series or array elements. Instead, we are forced to use very slow and sloppy "for loops" to load array values one-at-a-time. That is a severe handicap. It is also ancient history in the high-tech world. The current version of C# is version 11. So NinjaScript hasn't been updated in all these years? Wow! That's almost unbelievable in this day and age of lightening fast tech life-cycles. Who wants to work with old/ancient technology? No me. I didn't realize that's what I was paying for when I bought the platform. I thought it was modern up-to-date software, not a dinosaur.

Hopefully those in control will get the message.

Thank you very much for your assistance.

Sincere regards,

Kerry

[jeronymite]

Dr Kerry I agree with you. You may wish to wade through this thread for more background and opinions: https://forum.ninjatrader.com/forum/...and-c-version/

On a different topic, but related to use of valuable C# features that make good sense, you may care to also read this thread: https://forum.ninjatrader.com/forum/...-not-supported

Thanks.

[Dr Kerry]

Dr Kerry I agree with you. You may wish to wade through this thread for more background and opinions: https://forum.ninjatrader.com/forum/...and-c-version/

On a different topic, but related to use of valuable C# features that make good sense, you may care to also read this thread: https://forum.ninjatrader.com/forum/...-not-supported

Thanks.

jeronymite Thank you for these links. Very helpful. I purchased the platform earlier this year naturally thinking that it is a modern well-maintained/up-to-date piece of software.

I am very familiar with the Wolfram language (a.k.a. Mathematica) from my professional R&D work. Mathematica is very rich in features for manipulating multidimensional arrays and lists and their elements in parallel. I am just now becoming somewhat proficient in C# and the development of trading strategies using NinjaScript.

Manipulating array elements in parallel is nothing new to me and it would seem inherently required for efficient scripting on this platform as so much depends upon calculations involving series and array elements. So I was quite shocked to learn that this is not possible to do in NinjaScript. I was even more shocked, and a bit angered, to learn that such functionality is not possible only because NinjaScript is based upon such an old version of C# that these simple, and quite necessary operations, are just not available in version 5 of C#. Yet such operational functionality has been available since 2019 when C# version 8 was released and included such things as the range operator ".." for manipulating groups of elements in an array together. Instead, I'm loading array elements one-at-time using "for loops" which is very slow and not practical for a real-time trading environment where time is always of the essence. Professionally, I almost never use "for loops" for anything these days other than as a very last resort. Such loops were quite popular in the 1980's when FORTRAN was the primary scientific programming language.

I really hope that someone at Ninja gets on the ball, decides to bite-the-bullet, and update the platform to the latest version of C# in very short order. From the posts I read on the link that you shared it seems that they are not in much of a hurray to do anything at this time. My guess is that they're afraid of somehow breaking the platform by updating it and therefore have made the conscious decision to do nothing for as long as possible. In the high-tech world such a do-nothing strategy will only work for so long until they are soundly surpassed by all of the competition.


Thanks again.

Regards,

Kerry

[NinjaTrader_BrandonH]

Hello Kerry,

Thanks for your notes.

I have added your vote to the feature request that Emily mentioned regarding updating the C# version used for NinjaScript. This request is being tracked under the number SFT-3124.

As with all feature requests, interest is tracked before implementation is considered, so we cannot offer an ETA or promise of fulfillment. If implemented, it will be noted on the Release Notes page of the Help Guide.

Release Notes — https://ninjatrader.com/support/help...ease_notes.htm

[wzgy0920]

These are features that the development team certainly considers, though I would not be able to provide any insight regarding current plans or schedules for when these features may be implemented. If it interests you, I would be glad to add your vote to an existing feature request to update the C# version used for NinjaScript so we may track additional interest. As with all feature requests, interest is tracked though we are not able to offer an ETA or promise of fulfillment. If implemented, it will be noted in the Release Notes page of the Help Guide here:

• https://ninjatrader.com/support/help...ease_notes.htm

Thank you for using NinjaTrader. Please let me know if I may be of further assistance.

Please add my vote also.

[NinjaTrader_ChelseaB]

Hello wzgy0920,

Thank you for your vote.

We've added this to SFT-3124.