bactesting vs historical (strategy performance)

[cmv727]

Why would results from a backtest in Strategy Analyzer be very different from Strategy Performance based on historical data? Aren't both based on the same historical data set?

If I attach a strategy to a chart, and then run Strategy Performance from that chart (using the Historical data option, I get wildly different results.

Which of the two sets of results is likely to more closely resemble real time results. (I DO understand that real time results must be different due to intra-bar issues, etc. But here the results are different using historical data).

Any help appreciated.

[NinjaTrader_ChelseaB]

Hello cmv727,

You can expect that a strategy running real-time (live brokerage account, live market simulation, Market Replay etc...) may produce different results than the performance results generated during a backtest depending on how much resolution the script is running with. Further, this difference may be more easily seen on certain Bars types (e.g. Point and Figure, Renko) than others due to their inherent nature in bar formation.

Below is a link to the Help Guide on 'Discrepancies: Real-Time vs Backtest'.
Help guide: Operations > Strategy Analyzer > Discrepancies: Real-Time vs Backtest

Adding intra-bar granularity can help with this.

Intra-bar granularity adds a second data series such as a 1 tick series so that the strategy has finer granularity in the historical data in between the OHLC of the primary series. By using the BarsInProgress index of the 1 tick series as the barsInProgressIndex parameter for each order method call, this allows for more accurate trades by supplying the correct price at the correct time for the order to fill with.

Below is a link to a support article that details adding 1-tick intra-bar granularity and includes a link to the official reference sample demonstration.
Developer guide: Improving backtest order fill accuracy with intrabar granularity

Further, below is a link to a support article that discusses what causes differences, how to use prints or writing to file to understand what is specifically causing differences in your script, and how to mitigate these differences.
Developer Guide: Comparing Strategy Results

[cmv727]

I understand the differences between backtesting vs. live and live simulation. But my question compared backtesting vs. strategy performance from a strategy on a chart that used HISTORICAL, not live data.

Aren't those two data sets the same, so shouldn't results be the same?

Thanks.

[NinjaTrader_ChelseaB]

Hello cmv727,

I would expect different data or different settings.

The output will tell us which.

[cmv727]

Is there a video or instructions that show how to add a tick series for a backtest in order to enable intrabar entries. I am unclear how this would be set up.

[NinjaTrader_ChelseaB]

Hello cmv727,

I do have a video I've made for another user you may find helpful.

https://drive.google.com/file/d/12F6TQD5VvJvYuLrb3G6uc3-C3TgPC3pA/view?usp=drive_link

[Edson Dias]

Hi there
I have a similar issue.
I expect that might be a little difference between the backtest and realtime, I but I my case, no operation are executed in backtest, but in realtime or playback runs ok
I'm using calculate at bar close
Is there any guide to address this?

[NinjaTrader_ChelseaB]

Hello Edson Dias,

If the expected trade(s) are not appearing, this would indicate that the condition to place the order is not evaluating as true and the order is not being submitted, or the order is being ignored for other reasons, or the order is being cancelled or rejected.

To understand why the script is behaving as it is, such as placing orders or not placing orders or drawing objects when expected, it is necessary to add prints to the script that print the values used for the logic of the script to understand how the script is evaluating.

In the strategy add prints (outside of any conditions) that print the date time of the bar and all values compared in every condition that places an order.
The prints should include the time of the bar and should print all values from all variables and all hard coded values in all conditions that must evaluate as true for this action to be triggered. It is very important to include a text label for each value and for each comparison operator in the print to understand what is being compared in the condition sets.
The debugging print output should clearly show what the condition is, what time the conditions are being compared, all values being compared, and how they are being compared.

Prints will appear in the NinjaScript Output window (New > NinjaScript Output window).

Further, enable TraceOrders which will let us know if any orders are being ignored and not being submitted when the condition to place the orders is evaluating as true.
After enabling TraceOrders remove the instance of the strategy from the Configured list in the Strategies window and add a new instance of the strategy from the Available list.
Last, print the order.ToString() at the top of the OnOrderUpdate() override, as this will show us when orders become working and are being filled.

I am happy to assist you with analyzing the output from the output window.

Run or backtest the script and when the output from the output window appears save this by right-clicking the output window and selecting Save As... -> give the output file a name and save -> then attach the output text file to your reply.

Below is a link to a support article that demonstrates using informative prints to understand behavior and includes a link to a video recorded using the Strategy Builder to add prints.

https://support.ninjatrader.com/s/article/Developer-Guide-Debugging-using-Print-and-TraceOrders

Let me know the date and time the behavior occurred or when you are expecting the behavior to occur.

Please let me know if I may further assist with analyzing the output or if you need any assistance creating a print or enabling TraceOrders.

[luketrades]

Hi,

I’ve encountered a serious issue in my backtesting that I can’t understand. Sometimes, trades are marked as winning in the backtest when they were actually losing trades.

In the attached image, you can see an example of a trade that is marked as "winning," but it was actually a losing trade.

Here’s some context: my trading strategy is based on order flow. The strategy looks for a CVD divergence with price, then waits for an initiative auction candle. If all the requirements are met, it enters on the POC retest of the initiative auction candle.

In this example, the strategy entered on the POC retest. However, on the next candle, while the price first moved to fill the buy limit, it then went on to hit the stop loss (which, in this case, is placed at the low of the initiative candle).

My question is: does the strategy analyzer ignore how price moves intra-bar? This is a concern because, according to the backtesting results, the strategy has an over 80% win rate and seems very profitable. However, I can’t trust these results if issues like this exist.

[NinjaTrader_ChelseaB]

Hello luketrades,

Welcome to the NinjaTrader forums!

In historical the intra-bar price movement is not processed unless you have implemented 1-tick intra-bar granularity.

Below is a link to a support article on 1-tick intra-bar granularity.

https://support.ninjatrader.com/s/article/Developer-Guide-Improving-backtest-order-fill-accuracy-with-intrabar-granularity

[luketrades]

I tried to implement by reading the documentation, but now it gives me an error that i never seen.

I used AddDataSeriers in State.Configure and BarsInProgress.

May I share the code ? For eventual correction

[NinjaTrader_ChelseaB]

Hello luketrades,

The method signature would be:
EnterShortLimit(int barsInProgressIndex, bool isLiveUntilCancelled, int quantity, double limitPrice, string signalName)

https://developer.ninjatrader.com/docs/desktop/entershortlimit

Try:
EnterShortLimit(1, true, numberOfContracts, entryPrice, "BearInitiation");

With the BarsPeriodType error, I would suspect that one of the default using statements have been removed.
Have you removed any default using statements at the top of the script?

Create a new strategy. Copy and paste the default using statements from the new strategy to your script.

[luketrades]

Hi i tried to use: EnterShortLimit(1, true, numberOfContracts, entryPrice, "BearInitiation"); and yes i removed some of the using statemes but i did like you told me and now the code is compiled correctly.

Now.. the problem is still there..

I leave the code in this link: https://codeshare.io/LjOJKK maybe you can look at it and find what's wrong.

I also tried to remove tick-replay and the same trade goes to stop but the entry is shifted to next candle.. by removing tick-reply every trade has a delay of one candle

[luketrades]

I forgot an other thing.. if i use true as a parameters any unfilled order will remain, while i wnat that if the order dosn't get filled on the next candle after my trigger should be cancelled, now.. this condition was already working properly without this parameter.. but if i remove it the error that i show you a few post ago will remain. What can i do?

[NinjaTrader_ChelseaB]

Hello luketrades,

Is TickReplay enabled?
Is Calculate set to OnPriceChange?


Set methods should be called before calling an entry method.

https://forum.ninjatrader.com/forum/ninjatrader-8/strategy-development/1307722-sell-order-stop-must-be-below-price#post1307731

Note, Set methods cannot implement 1-tick intra-bar granularity. Exit orders must be used to allow the exit orders to fill with the 1 tick series.

https://support.ninjatrader.com/s/article/Developer-Guide-Improving-backtest-order-fill-accuracy-with-intrabar-granularity

Use false as the isLiveUntilCancelled parameter for the order to automatically cancel.
EnterShortLimit(1, false, numberOfContracts, entryPrice, "BearInitiation");