Backtesting and live (paper) trades are different

[Camdo]

I too share LookOutBelow's frustration over the profit differences between backtest and real trades. I record results of same strategy in live, simulated, backtest after market close, backtest next day and get 4 different answers.
My conclusion is that the data is changing.
The help section titled "Historical and Realtime data" describes the conditions when data is loaded.
Real time data is reloaded with the providers historical data without the user being aware when it is happening. So live trades and live simulated tades are using different data than backtests. Backtests run after the session close can differ from backtests run the next day because the data is being reloaded from the provider. Historical data from the provider is not a replica of the data recieved live during the trading day.

I would like to see a feature in NT to prevent data being overwritten and to fill only missing bars rather than reloading an entire day.

[NinjaTrader_Jason]

Please note that we support connectivity providers that provide data only as well. As per NinjaTrader 7, our preferred market data service is Kinetick. Real time service starts as low as $50 per month or completely free if all you need is end of day data. If you trade CME Globex products, you can have these exchange fees waived if you have a qualified brokerage account. This service is only available for NinjaTrader 7. Please visit www.kinetick.com for more detailed information.

We support the following data feeds as well.

• IQFeed
• eSignal
• Barchart

You cannot have two instances of NinjaTrader installed on the same PC of the same version. You can have both NinjaTrader 6.5 and 7 installed on the same PC - however you can only run one at a time. If you like to run two instances of NinjaTrader simultaneously, you will need two PC's.

It is against the EULA (end user license agreement) to trade live using the same license on two PC's simultaneously. But you can use it to trade live on one PC and only backtest (trade in simulation) on the other PC.

@luxurious_04, I assume you currently run your strategies in real-time while connected to the Market Replay connection. You can backtest strategies using the Strategy Analyzer. They will be calculated on historical data instead of real-time data you receive from the Market Replay connection. More information can be found at the link below.

http://www.ninjatrader.com/support/helpGuides/nt7/backtest_a_strategy.htm

[dave1992]

Although the application UI is no responsive during this time, you will not lose any events in your strategy, all will be processed.

This can happen very regularly depending on what you're doing. If you have lots of strategies, compile takes 5-10 secs of UI freeze. If you run backtest on multi instruments selecting a different set of results re-runs the test for that inst. UI freeze.

[luxurious_04]

Also, is there other way to test a custom strategy aside from back testing in replay data. Or how we can backtest custom strategies without using Replay Data because backtest with replay will take too long and inaccurate. Is there other way to test a custom strategies that is more faster. If someone knows a better way then we will appreciate and always ready to listen. Thanks in advance..

[lookOutBelow]

I believe most system developers separate their development PC from their real-time strategy execution PC at best, they don't mix development with running real time strategies.

That would be nice. But Mirus does not allow 2 connections at once. Also, since NT doesn't allow two instances (I don't believe anyway) at once, I'm not sure what I could do about this.

edit: Are we even allowed to run NT on two different computers? Or do we have to buy two licenses for that??

[lookOutBelow]

This brings up a good point which is if you are doing any processor intensive tasks during during market hours, you run the risk of certain processes demanding the use of the CPU(s).

Well, the only thing I do during market hours besides backtesting and futzing with strategies is read emails and some light web browsing. No movies or other CPU intensive things. Even during back testing, my cores are never fully used for whatever reason.

So, it sounds like, because of the App Busy/freeze situation when compiling and certain backtesting operations, this could be at least partially responsible for my screwy discrepancies.

That is somewhat comforting. I'll keep testing and not do any backtesting/compiling over the next few trading days to see if a pattern develops...

Thanks

[NinjaTrader_Ray]

If you have a freeze (application busy) then there will be a delay in processing data and thus submitting orders and thus can impact the results of your strategy. For clarification, what I was saying is that you will not lose any events but they would be delayed.

This brings up a good point which is if you are doing any processor intensive tasks during during market hours, you run the risk of certain processes demanding the use of the CPU(s). It is best practise not to run live trading if you are doing computational sensitive work. I believe most system developers separate their development PC from their real-time strategy execution PC at best, they don't mix development with running real time strategies.

[lookOutBelow]

there is an application busy state, for me about 1 second not ten (could be difference in hardware) between my set up and yours.

What took 1 second? The building of the charts or compiling a strategy or something else? I've got a 4 core computer here and compiling seems to take a different amount of time everytime I click the button. The fastest it ever compiles is 4 seconds. Sometimes it takes as long as 10-15 or more!

Building the charts can take a long time too in some cases. I don't think I've timed it out. But I feel sometimes I'm waiting 30 seconds. Of course some of my strategies are producing 3000 trades over 6 months.


This is starting to drift away from my original post/problem. But I really think these things could be related somehow.

[lookOutBelow]

I ran a test with many DOMs, T&S, MA with 40 markets on it, backtested SampleMA with your settings with a longer data set, March 1 through October 1st and for sure there is an application busy state, for me about 1 second not ten (could be difference in hardware) between my set up and yours. This is expected since as you click on the Chart Tab, loading 6+ month of 150 tick bars plus all associated trade markers takes processing time, all this is done on the main thread where all UI is being handled. Although the application UI is no responsive during this time, you will not lose any events in your strategy, all will be processed.

Hmm. So, even if there is a long delay for charts to build or something to be compiled, and during this period the GUI is freezing, buy and sells will still happen in a strategy at the exact time they would normally happen if the GUI wasn't frozen?

Okay, then I'm at a loss for why this (seems to) significantly affected the accuracy of live trading vs backtest. I guess it is coincidence. But it seems awfully peculiar that the day I decided to not do any compiling or backtesting that this whole discrepancy issue was mostly a non-issue.

[NinjaTrader_Ray]

I ran a test with many DOMs, T&S, MA with 40 markets on it, backtested SampleMA with your settings with a longer data set, March 1 through October 1st and for sure there is an application busy state, for me about 1 second not ten (could be difference in hardware) between my set up and yours. This is expected since as you click on the Chart Tab, loading 6+ month of 150 tick bars plus all associated trade markers takes processing time, all this is done on the main thread where all UI is being handled. Although the application UI is no responsive during this time, you will not lose any events in your strategy, all will be processed.

[lookOutBelow]

BTW, another easy way to freeze it is to delete an indicator or strategy.

[lookOutBelow]

In addition, could you perhaps include an example of how you edit your strategy code if possible or provide more information on this? That way we can better replicate this step on our end as well.

Well, I do it the standard way. Tools > edit ninjascript > strategy. Don't even have to change the code. Just hit the compile button. Everything freezes until it is done compiling.

[lookOutBelow]

Could you please send us your strategy, so we can test it on our end. I have tried to reproduce it using the SampleMACrossOver strategy, but so far I was unable to do so.

I can reproduce this using SampleMACrossOver. Look at my screenshot to get the settings and try to match it. After running the backtest. Click on the chart. During the time it takes to build/display the charts (took around 10 seconds or more), all quotes, and everything else on NT freezes. Make sure you have a couple of DOMS open and maybe TS to see it clearly.

[NinjaTrader_Jason]

Hello lookOutBelow,

Could you please send us your strategy, so we can test it on our end. I have tried to reproduce it using the SampleMACrossOver strategy, but so far I was unable to do so.

Please send it to support[at]ninjatrader[dot]com and reference this forum thread. You can export strategies as per the instructions at the link below.

http://www.ninjatrader.com/support/helpGuides/nt7/export.htm

In addition, could you perhaps include an example of how you edit your strategy code if possible or provide more information on this? That way we can better replicate this step on our end as well.

[lookOutBelow]

NT freezes when you do certain operations

Okay. So, I realize something with NT that may not be the root cause of this problem stated in the original post. But in the very least, it is a serious issue that affects performance. And it certainly affects running strategies.

Lucky for everyone, it is easily reproduce-able.


Yesterday, I decided to let NT run throughout the day and not touch it. Didn't edit any strategies, didn't compile any code, didn't do anything. Guess what? The results were much closer than previous days where I had wild discrepancies. Live/sim and backtesting almost lined up as you would expect.

With that in mind, this morning, I did a couple of simple tests and I saw that NT actually FREEZES when you do certain operations.

Open up ES in a DOM and a time and sales. Also open a Market Analyzer window and put several instruments in there. You can test this yourself.

Now, open a strategy analyzer and run a backtest over a long period of time, say 6 months. Now, after you get your results, hit your "Chart" tab. The time it takes NT to calculate a chart , it actually freezes either the data coming into NT or the program freezes for whatever reason.

Now, go edit one of your strategy code. Now compile it. Everything freezes again!


Now, if you are back testing and editing code like I've been doing all day long, these freezes are going to screw up entries and exits. This may be the issue!!! Or maybe not. Whatever the case, this is a problem itself.