MarketReplay versus Strategy Analyzer Result Discrepancy

[rdavido]

Recently, I have been testing multiple strategies in NT8. The strategy is a neural network based strategy. In this example, I am running it for CL 07-17. I have attached the strategy and accompanying indicator it references. I have XLS files from the actual trades but am unable to upload them here.

My results are mixed. In strategy analyzer, using optimization, I receive these results:


When I run this strategy for the period in MarketReplay, the fills are completely different (lag behind my almost a bar it seems) and my P/L is mixed.



Could someone take a look at my code and see if there's a glaring reason why the results are so different?

Thanks

[rdavido]

Here's a list of trades:

[Strategy Analyzer - Optimized]



[Market Replay - Based on SA Optimization]

[NinjaTrader_ChelseaB]

Hello rdavido,

Historical data and real-time data (Market Replay) are fundamentally different.

Please review the following:

http://ninjatrader.com/support/forum/showthread.php?p=491297#post491297

May I ensure that intra-bar granularity has been added?

May I ensure that no historical data is being included with the Playback test?


To dig into what is causing the different results I would recommend starting with the data. Add prints to ensure that the very first bar (real-time only) in playback matches the first bar of backtest in the strategy analyzer.

Historical data is recorded for your specific data feed connection. Market Replay is recorded by NinjaTrader from a single source. Each data feed is slightly different with timestamps of ticks as well as qualifying vs non qualifying trades.

Who are you connected to for historical data?
Are you recording your own Market Replay data from this source or is this being downloaded from NinjaTrader's Market Replay server?

Add prints to ensure that the open, high, low, close, volume, and time are the same.

[rdavido]

I'm preparing to add prints. Would you recommend adding the Print to my strategy or the Indicator itself?

		if (IsFirstTickOfBar)
	    {
	        double FirstClose = Close[0];
			Print (FirstClose);
	    }

Will this suffice?

[rdavido]

Additionally, I have a secondary data period setup in my indicator

AddDataSeries(Data.BarsPeriodType.Minute, barPeriod);

My strategy calls back to the indicator and I have a custom variable to declare barPeriod, however, I do not know what to print to ensure I am getting the correct range.

Furthermore, I have a calculated double from my indicator "l3_0" that I would like to reference within my strategy, how could I do this?

[NinjaTrader_ChelseaB]

Hello rdavido,

I would recommend adding the prints to the strategy (first).

The goal would be to see what is different.

The first print in both playback and historical need to the be the same bar (shown by time).

The playback would not include any historical data so that this builds with the same data that was tested historically.

On each bar if the times match up, I would recommend printing the values of all variables in conditions that place orders outside of any conditions so that we can understand what was different (either data or order fills).