backtest replication

[NinjaTrader_JoshP]

ctrlbrk,

We have a misunderstanding. We are discussing the comparability between a real-time run and a replay run. In real-time you have delays set which for sure influences fills. In replay, no delays are used and as such produces different fills. Delays should not be used if you want to try and compare the two results.

[nikkk]

Not necessarily. Please go to Tools -> Options -> Simulator tab and ensure you don't have any delays set. These delays do influence your fills drastically from accelerated replay attempts.

what you mention is execution delay, which has some impact, agree. in this case the numer of trades should match. but it does not.
moreover, triggered trades (enter or exit, does not matter) differ.

what i am aiming at is that when i found a strategy which performs well with certain parameters and the result brings me certain confidence (given the size of sample and small PnL variation in some parameters range) i want to replicate that in real trade. so far, the real-time fake-executions do not give me that confidence...

[NinjaTrader_JoshP]

Delays do not just influence fill price. They influence much more. For instance, having an order not fill on a bar may incur the order to actually expire and cancel instead. You would not enter a trade in that instance and that completely influences all future trades since you may now be taking trades in other areas that you would not have if you were already in the first.

No matter what level of simulation you try and do, real trading is always different. Backtesting, forward testing, sim trading, paper trading, everything along those lines can only give you a glimpse at what could happen live. When you actually trade live things are often very different.

[nikkk]

Delays do not just influence fill price. They influence much more. For instance, having an order not fill on a bar may incur the order to actually expire and cancel instead. You would not enter a trade in that instance and that completely influences all future trades since you may now be taking trades in other areas that you would not have if you were already in the first.

No matter what level of simulation you try and do, real trading is always different. Backtesting, forward testing, sim trading, paper trading, everything along those lines can only give you a glimpse at what could happen live. When you actually trade live things are often very different.

This statement contradicts my experience.

I said at the beginning - with IB+excel and tick-based strategy I have achived 1-3% of PnL difference (exec delay and bid-ask taken into account and nearly ALL backtest trades match real trades).

My new idea which is based on the interval (bar) indicators requires new development, so i decided to use Ninja.

However, it seems that Ninja gives random (i.e. I cannot explain the difference) result w.r.t. expectations and you try to convince me that this is OK ???? well...

just to be constructive (I still have a hope to get some usefulness out of this system) - maybe this forum had some similar discussion in the past, so may be I can find some missing tricks I still have to implement in my strategy.

thanks

[NinjaTrader_JoshP]

nikkk,

At the end of the day just run SPY/SPY with no delays on recorded data with chart types that produce the same bars and you will have very similar if not identical results between real-time and replay.

Chart types that have the potential of creating different bars could be something like tick charts or volume charts since those are very dependent on what is the very first tick on the chart since all subsequent bars are built from that first one.

[nikkk]

nikkk,

At the end of the day just run SPY/SPY with no delays on recorded data with chart types that produce the same bars and you will have very similar if not identical results between real-time and replay.
.

Please, give more details on how to do SPY/SPY. Under Options/Account I see two "delay" parameters, have set them to zero but with no effect on the replay result. Definitely, this is not what you suggest (??)

[NinjaTrader_JoshP]

nikkk,

Your strategy is multi-instrument. You were comparing a real-time run on SPY/SPY against a replay run of ES/SPY. This is not comparable.

Set your delays to zero then run your strategy again in real-time as SPY/SPY and then in replay as SPY/SPY or as ES/SPY and ES/SPY both times.

[nikkk]

nikkk,

Your strategy is multi-instrument. You were comparing a real-time run on SPY/SPY against a replay run of ES/SPY. This is not comparable.

Set your delays to zero then run your strategy again in real-time as SPY/SPY and then in replay as SPY/SPY or as ES/SPY and ES/SPY both times.

my strategy is single instrument, directional betting.
in terms of delays and bar-sizes replay == real, i.e. I did two things:
1. I run strategy on 1-minute chart, CalculateOnBarClose=false, end of day I replay data in "brand new" 1-min bar chart with same option.
2. i set CalculateOnBarClose=true, run during a day and replay at evening with same option.

in both cases I get differences...

I will try to set all delays to zero, run strategy with that and see what happens.

will be back.
thanks for patience.

nik

[NinjaTrader_JoshP]

nikkk,

Sorry for the confusion. I got this case mixed up with another one.

Run 1min chart for your strategy while ensuring you have the exact same historical bars loaded too. Ensure you choose the exact same start date and actually make a confirmation check that your data starts at the same point.

Set delays to zero, try real-time, then try replay of that same instrument.

[nikkk]

I did today with zero-delays. repeated backtests with/without delays, CalculateOnBarClose=false/true, calculate in replay and backtest... all results are different.

Trades themselves do match to some extend, but not at the level I want (not within the bar even). I give up and quit. No more time can invest into this.

Good luck and Best regards,
Nik

[NinjaTrader_JoshP]

nikkk,

As stated, when using a strategy on the same chart will produce same results. If your bars build differently because your historical data is different, for instance, it will be different.

Best of luck to you too.

[nikkk]

Hello,

I came back to NT (was sorry for time spend) and finally managed to replicate "backtest" results with "replay". The difference is obvious one - trade at the "market". Now I should learn how to improve performance using limits.

Is it possible to implement some sort of limit traded strategy through NT??

Nik

[NinjaTrader_JoshP]

nikkk,

Not sure what you mean by "limit traded strategy". If you want to place limit orders you can place them. If you want market orders you can place those as well.

[nikkk]

sorry that I made myself not clear.

I want to be able to modify limit orders algorithmically. Inside OnOrderUpdate I can trace orders, but how to modify them through the program? Cannot find the reference, help shows only manual adjustments via GUI.

Another problem I predict is that my strategy is linked to the bar-chart, f.i. 5 min., therefore all Indicators will be calculated with that frequency (having CalculateOnBarClose=true). However, update of Limit price requires Indicators working with ticks. How to do that?

Thanks
Nik

[NinjaTrader_Bertrand]

Hi Nikkk, you want to work with the IOrder objects then - http://www.ninjatrader-support.com/H...V6/IOrder.html

To update your limits on each tick, you could set the strategy to CalculateOnBarClose to false and wrap statements to be evaluated 'on bar close' in If (FirstTickOfBar) - of course referencing then a bar more back to make up for calling this on the start of the next bar.