Trades not adding up

[frostengine]

Should market replay for a day and Strategy Analyer->Backtest for that same day give the exact same trades? Perhaps entries/exist slippage may be different? but same trade signals should be generated correct?

[NinjaTrader_JoshP]

No, Market Replay is forward testing while Strategy Analyzer is backtesting. Forward testing and backtesting are fundamentally different to almost the same degree of real-time vs backtesting.

As far as signals go it really depends on your strategy. If you use CalculateOnBarClose = false then for sure it will be different because backtesting is always done with CalculateOnBarClose = true.

[frostengine]

Yea I am aware of the fundamental differences. And I do use CalculateOnBarClose=true in my strategies. Mostly all I care about is will the actual trades placed be the same. Such as if market replay over a set of days generates 10 trades, should I expect that backtester will also produce 10 trades...

Just trying to get a handle on all the differences so I can find out what in my results I can trust and what I can not. Slippage, slightly different entries etc are fine but when whole trades go missing it makes me not be able to trust anything that is generated... So trying to establish what the boundries are...

[NinjaTrader_JoshP]

Signals should be the same provided the exact same data is loaded. Depending on your strategy, backtesting for 30 days may not necessarily produce the same results on the last 10 days as a Market Replay just running on the last 10 days.

[frostengine]

Josh,

Can you elaborate a little more on that principle? Such as what type of things your strategy could cause it to produce those different results?

[NinjaTrader_JoshP]

Anything that causes you to have different data sets. Your start and end dates need to be EXACTLY the same.

A Market Replay starting yesterday will load 15 days back from that point. A Market Replay starting from today will load 15 days back from today. Strategies are calculated based off of those loaded histories. You can quickly see how this rolling 15 days back can cause differences in data and therefore differences in calculations and ultimately differences when compared to backtest results.

Historical data is a VERY critical factor in influencing your strategy calculations. You need to ensure each Market Replay run would have the same set of data as if it were backtesting. That means 10 days ago should load 0 days back. 9 days ago should load 1 day back. 8 days ago's replay should load 2 days back, etc. You set these through Tools->Options->Data.

In fact I don't recommend you even compare backtests to forward testing. They are just not comparable. You should do BOTH backtesting and forward testing separately and understand ALL the results given to you. They offer you a range of possible results and you should always take them with a grain of salt. At the end of the day you also have to realize that real-life again will be VERY different from any testing you do. That is why you need to fully accept the range of results you get and be willing to take that risk based on the range before you go live.

[frostengine]

I just noticed one potential issue I am having is the fact I do not have all the recorded data I thought I had.... For past 3 weeks or so I have been running market analyzer everyday with a bunch of symbols. I have store real time market data checked under options->data. So, I figured all the data it gathered on these instruments would be saved.. that appears to not be the case...

It looks like only the instruments I had actively running strategies stored data, or something... Is that the case? Does running market analyzer against a symbol not guarantee data for it will be saved?

[frostengine]

I just finished checking data for some of the symbols I have been running strategies on everyday for a few weeks. I have market replay data for them that entire day but when I view the data outside of market replay it appears I am missing sections..

This explains my problem with backtesting and market replay not lining up with each other. For backtesting a LOT of data seems to be missing all of the place..... I am doing this test from not the same computer it was captured on... Will do the same check tonight on the main computer...

If that computer indeed has missing data what is the explanation. The basic question being: I have market replay data for that entire day, viewing a chart of the data in market replay looks fine.. View the same day outside of market replay using the historical data stored and there are missing segments.. for instance on 5/11 i ahve data the entire day in market replay. When I open a chart for just that day i have no data until 1;00 that afternoon...

I am using interactive brokers for data feed and tick charts so that rules out any sort of strange backfill or something....

[NinjaTrader_JoshP]

frostengine,

If you feel your historical data set may have gaps I suggest you to try to reload the data. Reloading the data may load up a different set of data than what you had recorded in Market Replay and that could influence results though.

Tools->Historical Data->Reload

[frostengine]

Josh,

As I pointed out in my last message I am using Interactive Brokers which you can NOT reload historical tick data. What could possibly be causing this missing data? As has already been pointed out I can not reload historical tick data so that removes any possibility that I messed up the data by having it reload or anything. Therefore the historical data for those instruments should only be what was recorded in REAL time. Which obviously data was recoded for that period of time because it shows up in my market replay. Which leaves only one option I can imagine which is a bug in NinjaTrader not saving all the live data properly. Am I missing something? How can I work around this....

[NinjaTrader_JoshP]

You likely triggered a reload in real-time with your charts which could introduce gaps since you do not receive historical data so when it tries to grab some there is none and then there is a gap.

In either case, the suggestion still stands. You should not try to compare backtests to market replay for being identical. They will most likely produce different results and you should evaluate both as potential outcomes of your strategy.

[frostengine]

Hmm so your saying if I load up a chart in the middle of the day that it would try to reload, couldn't and produce a gap? The gap in question is from 8;30 AM to 1:00.. all of that data was wipped out. So what your saying is if I opened a chart on this symbol at 1:00 that would have done it?

[NinjaTrader_JoshP]

That is the only potential explanation I have right now since no one else has ever reported missing data from a database while recording. As far as if that was actually the case I could not say for your particular scenario. You could try keeping an eye out and seeing what happens when you lose data.

[frostengine]

Is it possible for market replay to change your historical data? I am trying to use a second instrument in my trading... so i included 1 minute data of SSO so that I could use IB to backfill that data. So i backfilled some 1 minute SSO data since apparently i had gaps ALL over the place in the tick data..

I then ran some backtests.. then i connected a market replay and ran the strategy against that day.. then i disconnected market replay and ran the normal backtest again.. and to my surprise gaps were back.. its like the backfill from IB I did got erased just by running market replay....

[NinjaTrader_JoshP]

No, Market Replay does not influence anything in your database.