Accurate Backtesting & Tick Replay

[imonlysleeping]

bruce i never used the word “perfect” and that is clearly not what i’m looking for, even in a general sense, which seems to be the only way you know how to answer specific questions - generally, and as if the person asking is 8 years old and just started trading yesterday.

thanks for trying, but your non answers have wasted my time and yours. and have only made ninja trader appear even less professional than they already do. your “helping” is honestly hurting at this point.

i’d prefer to hear from a ninjatrader cs person. perhaps emily would like to chime in? or anyone who knows AND CAN EFFECTIVELY COMMUNICATE anything specific about this god forsaken application.

i am happy to make the question more specific if that would help get a more specific answer.

thanks

[QuantKey_Bruce]

I had to go back and reread this thread to see what you're even talking about here. Tick replay isn't intended to replace the fill model with a higher frequency one - it allows you to see the price movements (bid, ask, and last) within the bar without adding a 1-tick series as a separate series to do this. You can use these intrabar updates to take actions e.g. place an order before the bar is over as well. It is not typical to do a tick replay in conjunction with market replay although you can. It's not clear to me what problem you are even having at this point - you seem to be asking us (the community and/or the platform software provider) to assure you that your strategy will work the same live, and we can't do that because we can't know except in generalities which is why you get generalizations in your answers - we also try to answer broadly so we can help other people who are searching for similar solutions and not just you. I don't speak for NinjaTrader - I was trying to help you, subject to the limitations on my time. If you're trying to reach NinjaTrader specifically, you can also email platform support [at] ninjatrader [dot] com - this is the community forum for questions that are public and open for community discussion. NinjaTrader personnel can and do respond here, and they will probably reply again to this when they are open and get to it (it is currently their weekend, of course).

[imonlysleeping]

Right, maybe because there have been a few redundant back-and-forths in a row and we are still at square one. Perhaps reading things more carefully in the first place can remedy this?

Regardless, i appreciate the continued efforts on your part and will try again to be as clear as possible:

I referred to post #10 where i described very generally the programming methods i have used, and the settings/options i have used to go along with it in an attempt to get accurate testing results for this custom strategy. i stated in post #10 that the results i have seen are exactly as expected. however, i have been downright puzzled by the contradictions within the reference manual and provided links in regards to different combinations of settings and options and programming methods available and their results accuracies and this has created a tinge of doubt to every result i see.

my primary desire in this thread was to just get a final confirmation from someone who is thoroughly familiar with and knows the ins and outs of ninjatrader and ninjascript, to give their confirmation that the expected results i am seeing, are, in fact, something trustworthy and that my method of acheiving those results is as sound and efficient as possible for my needs (barring all the general challenges like latency and order queue / liquidity we've already discussed at length, which to me are redundant as those would be more of a general disclaimer for anyone designing an automated trading system on any platform).

as i've stated numerous times, i am happy to clarify anything in more detail if that would help someone be able to give me a clear confirmation or non-confirmation.

thanks

[imonlysleeping]

In response to some of the specifics you gave in post #17:

1. Why would i or anyone go through the trouble of seeking out the ability to place an order intrabar (utilizing tick replay) at a 1-tick resolution, if the order is not going to have any accuracy to it? Makes zero sense. On the surface, it is a 100% contradiction of all logic. All it does is confuse people. Did NT have a lot of requests for the option to see inaccurate fills at a 1-tick resolution? That would be hard for me to believe. It's like going through all the trouble of building a Nascar and then driving it 30mph to the golf course on Sundays. Simply doesn't exist in the known universe, for good reason.

2. Why is it not typical to use tick replay and market replay in conjunction?

3. Wouldn't it be better to strictly limit available options to only those that provide accurate results, rather than screw around with all these extra options that don't? Who is utilizing intentionally inaccurate results for their strategy testing? It has zero use except to provide overwhelming confusion and frustration amongst potential new clients like myself.

It is my belief that NT should work on consolidating all of these half useful options into one or two that actually work well and can therefore provide less confusion and more initial ease of use and understanding. That's just my two cents.

I am honestly trying to understand this. Hope to hear from someone.

Thanks

[QuantKey_Bruce]

All of these options have specific use cases.

The specific use case for tick replay is when you need the intrabar detail e.g. you are doing something like calculating up ticks minus down ticks or trades at the ask volume minus trades at the bid volume, or perhaps you want to put in a limit order one tick above the last price precisely when RSI hits 80 or something like that on an intrabar basis.

Market replay can also do that, but market replay is all charts and all symbols at once, with controlled speed and under manual control, whereas tick replay is a single chart as fast as it is possible to compute with no visualization at all while it's doing it. A frequent use case of market replay is to practice trading by replaying a day's activity, or to test a system that involves several charts because perhaps the indicators communicate.

Tick replay is not intended to improve backtesting accuracy. This is explained in the documentation: https://ninjatrader.com/support/help...ick_replay.htm. You might use tick replay in conjunction with market replay if, for instance, you needed the HISTORICAL bars before the replay started to be tick replayed to get the order flow information calculated correctly so you have context and all the indicators are primed correctly up to the moment you started the replay. There might be other reasons. But, checking tick replay isn't going to change the way the market replay works once the market replay starts - it only pertains to the historical bars before the replay starts, so in a sense, they never happen at the same time and once you start the market replay, tick replay is over.

There are options to use high frequency data for testing. The default accuracy uses the OHLC bars of the series you are trading to process fills in an historical backtest like this https://ninjatrader.com/support/help...ical_fill_.htm. If you are doing a tick replay, it still operates this same way. The reason some people add a 1-tick series and place the orders on that (using the overloads of the order entry methods that allow you to specify which BarsArray to target the orders at) is that you can then get fills more accurately against that 1-tick series, similarly to using order fill resolution High and picking a 1-tick series there in the backtesting engine properties. But, you can't use High resolution if you have multiple data series in your strategy, so in that case, it makes sense to add a 1-tick series and place the orders against it yourself to get he most accurate possible simulation.

The reason there's no "one right" set of settings is that every arrangement is a compromise in one way or another. For instance, some people need to crank through thousands of iterations and there's no way to do it in a human lifespan with a 1-tick series and years of data, so they must have the ability to use OHLC bars to get a rough simulation quickly. Other people are only using market orders on bar open, so 1-tick series takes orders of magnitude longer and wouldnt't change the outcomes much if at all.

But, for limit order fill testing, a 1-tick series or High resolution with 1-tick selected in the backtesting engine if you just have one data series is the appropriate way to increase the accuracy over the default settings. Neither of these require a tick replay or a market replay, but they can be used with a tick replay or a market replay if you need those, and a market replay will process the simulation against the market data as it arrives similarly to how it would handle a live trader practicing against SIM101 in realtime.

[imonlysleeping]

Thanks very much for the detailed response to my post #19. Still hope to hear from someone in regards to my post number 18.

in the meantime, let me describe what i'm doing in a little more detail and see what you think and maybe we won't need to get into the prior post:

i'm currently using tick replay and a tick chart for a specific single futures instrument. utilizing IsFirstTickOfBar within onbarupdate() to find valid signals because i only need OHLC for that and potential valid signals are only tested for once at the close of each bar. if a signal is found, onbarupdate() is then checking every tick thereafter looking for specific entry conditions to be met which depend entirely on intrabar price movement of the bar immediately following the valid signal. if conditions are met, a limit entry order, setstoploss() and setprofittarget() are all immediately invoked on the same tick.

that's it.

i have confirmed expected behavior on a regular tick replay chart as well as on a market replay chart with tick replay, watching the trades take place and following the log as orders are placed, executed, etc. and everything looks good.

i'm just looking to find out if the strategy results that i am seeing are somehow inaccurate beyond the stated general latency and order queue issues we've already discussed.

[QuantKey_Bruce]

Just a quick reminder be sure to do your SetProfitTarget and SetStopLoss before you place your entry order not after. This could make a difference in outcomes during simulation or live trading, because if you the SetProfitTarget and SetStopLoss after, the previous trade's stop loss and profit target are still in effect and can cause you to be knocked out of the trade prematurely by the engine before the new SetProfitTarget and SetStopLoss are evaluated.

[QuantKey_Bruce]

I mention this because this is a common mistake - you may not have made it, but all of us have probably made it a bunch of times when we accidentally forgot about the sequence, so it's a good reminder. If you do them out of order (exits should be placed first for managed orders) the previous trade's SetStopLoss and SetProfitTarget exits are still in effect.

[imonlysleeping]

Bruce, thanks for that. Let me get some clarification because i already have read this same information in the reference material and wasn't sure how to handle it.

when you say place them before the entry order, do you mean literally just one line of code above the entry order?

or do you mean i need to place the setstoploss and setprofittarget orders outside of onbarupdate() altogether in OnStateChange?

the following is from the reference manual https://ninjatrader.com/support/help...and_onexec.htm

__________________________________________________ ______________________________________________

Tips (also see Overview):

•It is suggested to call this method from within the strategy OnStateChange() method if your stop loss price/offset is static

•You may call this method from within the strategy OnBarUpdate() method should you wish to dynamically change the stop loss price while in an open position

•Should you call this method to dynamically change the stop loss price in the strategy OnBarUpdate() method, you should always reset the stop loss price / offset value when your strategy is flat otherwise, the last price/offset value set will be used to generate your stop loss order on your next open position

•The signal name generated internally by this method is "Stop loss" which can be used with various methods such as BarsSinceExitExecution(), or other order concepts which rely on identifying a signal name

__________________________________________________ ______________________________________________

in the referenece material it states it is best to place these orders outside of onbarupdate and in OnStateChange so long as they don't need to be dynamically changed when already in a trade, but also used the wording "static", which is confusing because these values of course change every single trade.

i do not change the stop or target prices once the trade is entered, but i still couldn't understand how to set up the logic properly to be able to place those orders outside of onbarupdate because don't i have to call setstoploss and setprofittarget for every trade? or are they automatically invoked the moment any entry takes place even if they are called only once in OnStateChange?

currently, it is set up so that the stop price and target price for the upcoming trade are calculated / changed immediately at the first tick of the bar immediately after the signal is confirmed and prior to setstoploss or setprofittarget being called, and they are only changed if the signal is confirmed valid. so in essence they are reset anytime a new signal is found whether an actual entry ends up taking place or not. hopefully this makes sense and is the best way to handle it?

[QuantKey_Bruce]

Yes, above the entry order in the order of execution. What often happens, and I've done this more times than I can count, if I'm in a hurry and forget, is to place the entry order and then do a setstoploss and setprofittarget (assuming you're using that style of exit method - and it's the one you mentioned). But, let's say you have a stop loss that trails up to breakeven price after X ticks profit is reached. What can happen is when that happens during the trade, you SetStopLoss with a style of "ticks" and 0, or a style of "price" and the actual entry price with some offset. But, on the NEXT trade after the old one's already over, when the new entry order goes in, the engine can give you a SIM101 fill or a back-testing fill IMMEDIATELY when you place the new entry order (because you moved your stop to 0 ticks earlier) BEFORE it evaluates the line below which says to SetStopLoss to 20 ticks or whatever. So, to avoid this happening, it is a strong recommendation (really, a rule) that if you use the SetStopLoss and SetProfitTarget exit methodology, you should put these statements ABOVE the entry order in OnBarUpdate(). If you just think you're going to place them in OnPositionUpdate when the entry gets filled, remember that unless you preemptively moved them further away while you were still flat (before you placed the entry order, and after the previous trade exited), they're still set to whatever they were before and you can be knocked out of the trade the moment the new entry order goes in. So, if you keep all the exit orders in OnBarUpdate just put them ABOVE the entry order placement so the exits are known in advance when the entry order goes in. If you make this common mistake, you'll see exits that are all small losses right after you entered the trade, and when you investigate it, you will realize that the PREVIOUS trade's exits are being used to knock you out of the new trade before the new exits can be processed which is why you send the exits first.

[QuantKey_Bruce]

Also, if as you say, you do not change the stop or target prices ever after the trade is entered, just put your exits in OnBarUpdate and put them in BEFORE you place the entry order. This is all assuming you're using SetStopLoss and SetProfitTarget, and not ExitLongLimit etc. or unmanaged orders, each of which have their own considerations that are different.

[imonlysleeping]

yes, perfect. the entry and setstoploss and setprofittarget calls are all clumped together in onbarupdate currently. however, i have had it in the wrong order, having the entry order first. i will change it to have setstoploss and setprofittarget called first.

[QuantKey_Bruce]

Yes, if you rearrange to address this concern, you've done what's reasonable given what we know so far, and sticking with this style of order handling.

[imonlysleeping]

Yes, above the entry order in the order of execution. What often happens, and I've done this more times than I can count, if I'm in a hurry and forget, is to place the entry order and then do a setstoploss and setprofittarget (assuming you're using that style of exit method - and it's the one you mentioned). But, let's say you have a stop loss that trails up to breakeven price after X ticks profit is reached. What can happen is when that happens during the trade, you SetStopLoss with a style of "ticks" and 0, or a style of "price" and the actual entry price with some offset. But, on the NEXT trade after the old one's already over, when the new entry order goes in, the engine can give you a SIM101 fill or a back-testing fill IMMEDIATELY when you place the new entry order (because you moved your stop to 0 ticks earlier) BEFORE it evaluates the line below which says to SetStopLoss to 20 ticks or whatever. So, to avoid this happening, it is a strong recommendation (really, a rule) that if you use the SetStopLoss and SetProfitTarget exit methodology, you should put these statements ABOVE the entry order in OnBarUpdate(). If you just think you're going to place them in OnPositionUpdate when the entry gets filled, remember that unless you preemptively moved them further away while you were still flat (before you placed the entry order, and after the previous trade exited), they're still set to whatever they were before and you can be knocked out of the trade the moment the new entry order goes in. So, if you keep all the exit orders in OnBarUpdate just put them ABOVE the entry order placement so the exits are known in advance when the entry order goes in. If you make this common mistake, you'll see exits that are all small losses right after you entered the trade, and when you investigate it, you will realize that the PREVIOUS trade's exits are being used to knock you out of the new trade before the new exits can be processed which is why you send the exits first.

got it. thanks a bunch. i am setting the stop and target values preemptively, so should not have any issues with this, and have not so far.

[imonlysleeping]

so in terms of forward testing this... am i good using Market Replay with tick replay off? please refer back to my post number 21 if you need details on how the program is running