Originally posted by NinjaTrader_ChelseaB
View Post
Citizens of the NinjaTrader Community,
A common question we hear from clients is 'why are results from backtest different from real-time or from market replay?'.
The main reason is due to a lack of intra-bar granularity.
See more about intra-bar granularity.
https://ninjatrader.com/support/forum/forum/ninjatrader-8/strategy-development/94098-isfirsttickofbar-vs-onbarclose-for-backtest-live?p=773377#post773377
By default when backtesting no intra-bar granularity is added. An added 1-tick series is necessary to increase fill accuracy to within 1 tick.
Further, TickReplay is necessary for intra-bar indicator values to update and recalculate for on each tick or on price change Calculate setting.
As a heads up, TickReplay cannot be used for accurate order fill prices and High Order Fill Resolution cannot be used in conjunction with TickReplay.
Please note, as a script is enabled live on a chart or on the Strategies tab of the Control Center, the historical data will be processed first before the strategy transitions to processing real-time data. As the historical data is processed, this is the same as performing a backtest and will provide the same historical results as backtesting in the Strategy Analyzer.
https://ninjatrader.com/support/foru...442#post782442
Small differences in fill time, slippage, as well as small differences in when ticks are received and which bar they are included with can cause differences in the fill prices which aggregate in the total performance.
For the tests to be even similar, it is necessary to test the exact same sets of data, without mixing and matching the data types. (Meaning it would not be wise for a comparison of real-time or playback to include historical data and the days tested must start and end on the same session)
Its best to test by starting the strategy and reloading all historical data before running the test in real-time or playback.
Once the basics of getting similar test environments (meaning same data set, and intra-bar granularity) have been satisfied, it becomes clearer how to spot exactly what has caused the differences in the order fills.
Switching to view the chart may show that the charts appear very very similar but there are still differences in the final performance.
Prints can be added to the script (or this information can be written to file if its a large amount of information) to further understand the behavior.
See the post below about using prints for debugging.
http://ninjatrader.com/support/forum...979#post510979
This applies to NinjaTrader 7 as well, except for one large limitation: Tick Replay is a new concept developed for NinjaTrader 8 that is not implemented in NinjaTrader 7.
The major difference with this is that indicators that need to update with CalculateOnBarClose as false will not be able to update with the secondary tick series. (The fill prices will use the secondary series, however, the indicators will not be updating tick for tick and will only update when the bar closes)
Due to this, to get comparable results in NinjaTrader 7 for any scripts that call indicators the script must have CalculateOnBarClose set to True for all environments, real-time, historical (backtest), and replay.
It can also be a challenge to know what information to print, and how to analyze the information once its printed. It was even a challenge for me putting this post together just getting the test data to be the same. (Printing the bar and tick information really helped with this)
Below is a link to a set of videos I've created that demonstrate how to put this all together and analyze a small backtest in all three environments.
Attached is the modified SMACrossOver script I've written to test.
A common question we hear from clients is 'why are results from backtest different from real-time or from market replay?'.
The main reason is due to a lack of intra-bar granularity.
See more about intra-bar granularity.
https://ninjatrader.com/support/forum/forum/ninjatrader-8/strategy-development/94098-isfirsttickofbar-vs-onbarclose-for-backtest-live?p=773377#post773377
By default when backtesting no intra-bar granularity is added. An added 1-tick series is necessary to increase fill accuracy to within 1 tick.
Further, TickReplay is necessary for intra-bar indicator values to update and recalculate for on each tick or on price change Calculate setting.
As a heads up, TickReplay cannot be used for accurate order fill prices and High Order Fill Resolution cannot be used in conjunction with TickReplay.
Please note, as a script is enabled live on a chart or on the Strategies tab of the Control Center, the historical data will be processed first before the strategy transitions to processing real-time data. As the historical data is processed, this is the same as performing a backtest and will provide the same historical results as backtesting in the Strategy Analyzer.
https://ninjatrader.com/support/foru...442#post782442
Small differences in fill time, slippage, as well as small differences in when ticks are received and which bar they are included with can cause differences in the fill prices which aggregate in the total performance.
For the tests to be even similar, it is necessary to test the exact same sets of data, without mixing and matching the data types. (Meaning it would not be wise for a comparison of real-time or playback to include historical data and the days tested must start and end on the same session)
Its best to test by starting the strategy and reloading all historical data before running the test in real-time or playback.
Once the basics of getting similar test environments (meaning same data set, and intra-bar granularity) have been satisfied, it becomes clearer how to spot exactly what has caused the differences in the order fills.
Switching to view the chart may show that the charts appear very very similar but there are still differences in the final performance.
Prints can be added to the script (or this information can be written to file if its a large amount of information) to further understand the behavior.
See the post below about using prints for debugging.
http://ninjatrader.com/support/forum...979#post510979
This applies to NinjaTrader 7 as well, except for one large limitation: Tick Replay is a new concept developed for NinjaTrader 8 that is not implemented in NinjaTrader 7.
The major difference with this is that indicators that need to update with CalculateOnBarClose as false will not be able to update with the secondary tick series. (The fill prices will use the secondary series, however, the indicators will not be updating tick for tick and will only update when the bar closes)
Due to this, to get comparable results in NinjaTrader 7 for any scripts that call indicators the script must have CalculateOnBarClose set to True for all environments, real-time, historical (backtest), and replay.
It can also be a challenge to know what information to print, and how to analyze the information once its printed. It was even a challenge for me putting this post together just getting the test data to be the same. (Printing the bar and tick information really helped with this)
Below is a link to a set of videos I've created that demonstrate how to put this all together and analyze a small backtest in all three environments.
Attached is the modified SMACrossOver script I've written to test.
Comment