Backtesting and live (paper) trades are different

[lookOutBelow]

there is an application busy state, for me about 1 second not ten (could be difference in hardware) between my set up and yours.

What took 1 second? The building of the charts or compiling a strategy or something else? I've got a 4 core computer here and compiling seems to take a different amount of time everytime I click the button. The fastest it ever compiles is 4 seconds. Sometimes it takes as long as 10-15 or more!

Building the charts can take a long time too in some cases. I don't think I've timed it out. But I feel sometimes I'm waiting 30 seconds. Of course some of my strategies are producing 3000 trades over 6 months.


This is starting to drift away from my original post/problem. But I really think these things could be related somehow.

[NinjaTrader_Ray]

If you have a freeze (application busy) then there will be a delay in processing data and thus submitting orders and thus can impact the results of your strategy. For clarification, what I was saying is that you will not lose any events but they would be delayed.

This brings up a good point which is if you are doing any processor intensive tasks during during market hours, you run the risk of certain processes demanding the use of the CPU(s). It is best practise not to run live trading if you are doing computational sensitive work. I believe most system developers separate their development PC from their real-time strategy execution PC at best, they don't mix development with running real time strategies.

[lookOutBelow]

This brings up a good point which is if you are doing any processor intensive tasks during during market hours, you run the risk of certain processes demanding the use of the CPU(s).

Well, the only thing I do during market hours besides backtesting and futzing with strategies is read emails and some light web browsing. No movies or other CPU intensive things. Even during back testing, my cores are never fully used for whatever reason.

So, it sounds like, because of the App Busy/freeze situation when compiling and certain backtesting operations, this could be at least partially responsible for my screwy discrepancies.

That is somewhat comforting. I'll keep testing and not do any backtesting/compiling over the next few trading days to see if a pattern develops...

Thanks

[lookOutBelow]

I believe most system developers separate their development PC from their real-time strategy execution PC at best, they don't mix development with running real time strategies.

That would be nice. But Mirus does not allow 2 connections at once. Also, since NT doesn't allow two instances (I don't believe anyway) at once, I'm not sure what I could do about this.

edit: Are we even allowed to run NT on two different computers? Or do we have to buy two licenses for that??

[luxurious_04]

Also, is there other way to test a custom strategy aside from back testing in replay data. Or how we can backtest custom strategies without using Replay Data because backtest with replay will take too long and inaccurate. Is there other way to test a custom strategies that is more faster. If someone knows a better way then we will appreciate and always ready to listen. Thanks in advance..

[dave1992]

Although the application UI is no responsive during this time, you will not lose any events in your strategy, all will be processed.

This can happen very regularly depending on what you're doing. If you have lots of strategies, compile takes 5-10 secs of UI freeze. If you run backtest on multi instruments selecting a different set of results re-runs the test for that inst. UI freeze.

[NinjaTrader_Jason]

Please note that we support connectivity providers that provide data only as well. As per NinjaTrader 7, our preferred market data service is Kinetick. Real time service starts as low as $50 per month or completely free if all you need is end of day data. If you trade CME Globex products, you can have these exchange fees waived if you have a qualified brokerage account. This service is only available for NinjaTrader 7. Please visit www.kinetick.com for more detailed information.

We support the following data feeds as well.

• IQFeed
• eSignal
• Barchart

You cannot have two instances of NinjaTrader installed on the same PC of the same version. You can have both NinjaTrader 6.5 and 7 installed on the same PC - however you can only run one at a time. If you like to run two instances of NinjaTrader simultaneously, you will need two PC's.

It is against the EULA (end user license agreement) to trade live using the same license on two PC's simultaneously. But you can use it to trade live on one PC and only backtest (trade in simulation) on the other PC.

@luxurious_04, I assume you currently run your strategies in real-time while connected to the Market Replay connection. You can backtest strategies using the Strategy Analyzer. They will be calculated on historical data instead of real-time data you receive from the Market Replay connection. More information can be found at the link below.

http://www.ninjatrader.com/support/helpGuides/nt7/backtest_a_strategy.htm

[Camdo]

I too share LookOutBelow's frustration over the profit differences between backtest and real trades. I record results of same strategy in live, simulated, backtest after market close, backtest next day and get 4 different answers.
My conclusion is that the data is changing.
The help section titled "Historical and Realtime data" describes the conditions when data is loaded.
Real time data is reloaded with the providers historical data without the user being aware when it is happening. So live trades and live simulated tades are using different data than backtests. Backtests run after the session close can differ from backtests run the next day because the data is being reloaded from the provider. Historical data from the provider is not a replica of the data recieved live during the trading day.

I would like to see a feature in NT to prevent data being overwritten and to fill only missing bars rather than reloading an entire day.

[dave1992]

Couple of points - if data is being changed, presumably it's with data that's more accurate? Secondly, you can avoid any reload by not being connected to your provider when you run the backtest.

[NinjaTrader_Jason]

@Camdo, I will forward your suggestion to prevent data being overwritten and only fill missing bars to Development to put it on the list of future considerations of the software. Thank you for your feedback.

@dave1992, there can be minor discrepancies between real-time data and historical data with Zen-Fire in NinjaTrader 6.5.

Zen-Fire does not provide historical data. As a courtesy, we allow Zen-Fire users to make use of our historical data server. Please see the following forum thread.

http://www.ninjatrader-support2.com/vb/showthread.php?p=114191#post114191

As per post 7, data is aggregated differently as per the historical data server. The historical server is based off of NT 6.0 which agreggates tick data into minute bars based on 00:00:01 through 00:00:00. We changed this in NT 6.5 client so minute bars are created from ticks using 00:00:00 through 00:00:59. Thus, if you re-load data you can get subtle differences since there are two difference minute bar compression schemes used.

NinjaTrader 7 is not affected, since its historical data server uses the same compression scheme as the client.

[mart331]

Hello, I have another thought on this issue, having the same problem with majoor differences.
I do now test my strategies in replay
Can it be that the indicator values differ a lot? is their data filled in advance with replay and/or backtest?

[NinjaTrader_Jason]

Hello mart331,

When performing a backtest the indicator values are only known at the close of a bar. While running a strategy in real-time, indicator values are calculated intra bar (Calculate on bar close = False).

Could this be the cause of the discrepancy?

[mart331]

no, i use calculateonbarclose=true.
i think and test further.... For now i think all my backtesting and optimation efforts were useless

[mart331]

As far as I can see:

Backtest:
If your master chart is a 5 minute Kagi chart, but you trade (in strategy) on a 10 second chart (by Add ( second,10), you get enter/exit values from Kagi chart, NOT the 10 second chart values, and that are barCLOSE or barOPEN values, IS THIS THE CASE? (it should NOT)

[mart331]

As far as I can see:

Backtest:
If your master chart is a 5 minute Kagi chart, but you trade (in strategy) on a 10 second chart (by Add ( second,10), you get enter/exit values from Kagi chart, NOT the 10 second chart values, IS THIS THE CASE? (it should NOT)

I tried to workaround this, by adding 5 minute Kagi chart (AddKagi()) and 10 second master chart, but this also gives very different values compared to replay, a loss instead of 5000 euro profit in replay (6 weeks).

The problem can be the indicator. Because my entry/exit depends on 5 min Kagi, the indicator must use the Kagi data.
Now i use:
if (BarsInProgress == 1)
{
if (Indicator(...) == 1)
enterTrade()
}

I tried (dont know anymore in backtest or Replay) to use Indicator(barsArray[1]), but that gave the same results as in above example.

Tell me if I am doing wrong!