Forward run Sim101, then Strategy Analyser - both same dataset, but different results

[James Beale]

Today, 17th December, I ran a live session on the Sim101 account, until it ran out of currency. Then I ran a back test using Strategy Analyser on the same data.

But I got very different results.

- The Sim101 account shows a negative cumulative profit of -629.30
- Strategy Analyser shows a positive cumulative profit of 1550

The attached .xls spreadsheet files tell the story. I've also attached the strategy file in case it helps.

I'm at a loss to know why this could be so. Could you enlighten me, please?

[NinjaTrader_CalH]

Hello James,

Thank you for your post.

When you ran the strategy on the live, what did you set CalculateOnBarClose to?

Please note that you are going to see differences between live results vs a backtest as outlined in the link below -
http://www.ninjatrader.com/support/h...ime_vs_bac.htm

[James Beale]

Cal

Thank you for your reply. It was set to false. I'll try again with true.

[James Beale]

Cal

I have run a further test over the last several days. I attach graphs and csv outputs from the live run on Sim101, and the back test on Strategy Analyser.

"CalculateOnBarClose" is set to "true", as earlier advised.

I find that the results from the Strategy Analyser are wholly different from the Sim101 live run results, and therefore seem completely unreliable.

Please advise.

Thanks.

P.S. The 6th and final file follows in the next post...

[James Beale]

...6th and final file herewith.

[koganam]

Today, 17th December, I ran a live session on the Sim101 account, until it ran out of currency. Then I ran a back test using Strategy Analyser on the same data.

But I got very different results.

- The Sim101 account shows a negative cumulative profit of -629.30
- Strategy Analyser shows a positive cumulative profit of 1550

The attached .xls spreadsheet files tell the story. I've also attached the strategy file in case it helps.

I'm at a loss to know why this could be so. Could you enlighten me, please?

On what time frames are you running the tests?

[James Beale]

My test is 1 minute time frame.

[koganam]

My test is 1 minute time frame.

On a 1-minute bar basis, your results would seem to indicate that your tests are over different data sets. Retest, making sure that the start and end times of our tests match EXACTLY, to the minute, in the Strategy Analyzer and Sim101 tests, for the time periods of the tests.

[James Beale]

Hi there koganam

Thank you so much for your reply. Its much appreciated.

If you look at the two 1 minute chart .jpg images uploaded earlier, you'll see that they do indeed show identical times, that is 24th December 2014, 05:30hrs to 08:00 hrs.

The Sim101 chart is a live run ("forward test"), and the other is the Strategy Analyser ("back test") chart.

Even though the times are exactly the same, the trades are very different. I can't understand why the data saved by the "forward test" using Sim101, seems so different in the Strategy Tester

Indeed, I took the precaution to run a "forward test" of other time frames at the same time, and the two attached charts to show the 10 minute time frame. Again, running the Strategy Analyser over the same data time frame produces different results.

This is what I'm unable to explain, but am hoping that someone in NinjaTrader technical can.

[koganam]

Hi there koganam

Thank you so much for your reply. Its much appreciated.

If you look at the two 1 minute chart .jpg images uploaded earlier, you'll see that they do indeed show identical times, that is 24th December 2014, 05:30hrs to 08:00 hrs.

The Sim101 chart is a live run ("forward test"), and the other is the Strategy Analyser ("back test") chart.

Even though the times are exactly the same, the trades are very different. I can't understand why the data saved by the "forward test" using Sim101, seems so different in the Strategy Tester

Indeed, I took the precaution to run a "forward test" of other time frames at the same time, and the two attached charts to show the 10 minute time frame. Again, running the Strategy Analyser over the same data time frame produces different results.

This is what I'm unable to explain, but am hoping that someone in NinjaTrader technical can.

What are your "Data" options? Specifically, the ones that I have pointed at in ochre. Your pictures show considerable differences in data presentation.

I notice that these are currencies. Are you sure that you are using the same data source for the different tests?

Maybe you want to run the same tests on a stock or futures: with central exchanges, we are unlikely to have any data variances there.

[James Beale]

Hi koganam

Thank you again for your kind post. I'm grateful for your help.

I've attached a screen shot of my "Options / Data" set up. As you see, apart from not being tenth pip for currencies, I match yours where you indicated.

The $JPN225_CFDJPY is not, in fact, a currency. It is a CFD on FXCM's service, and supposedly mirrors the Japanese NIKKEI 225 stock market index.

And, yes, I agree with you. My data appears to differ between the Sim101 and the Strategy Analyser runs, despite my "Data" options being correctly set.

All of which I'm unable to explain.

[NinjaTrader_Jesse]

Hello,

I looked through the trades document you posted in #4 for the sim101. Are you sure you have the COBC setting to true? I see the trades have been happening not on the minute, here is an example time stamp from that document: 11:33:56

If it is on true the time stamps should all line up on the minute being that you are using 1 minute data. Can you please look into this to ensure that the setting is set to true?

Also in back testing vs forward testing with forex and cfd's, in a backtest the orders would be filled using the last price where forward tests would be filled using the ask and bid so there can be differences between results so this is also something to keep in mind.

Please check the the COBC setting and let me know if the results change.

I look forward to being of further assistance.

[James Beale]

No. The COBC is set to "true". See attached. The problem lies elsewhere. I look forward to finding out where.

[NinjaTrader_Jesse]

Hello,

Thank you for the reply.

These are strange results assuming the strategy is running on CaluclateOnBarClose = true and it is having trades intrabar when there is no way for it to trade intrabar using this setting or the code provided.

I do see the strategy in the image is a different strategy than the one originally posted. Have you added intrabar logic to the strategy that we don't know about being that I am looking at the NBarsUpDnTP that was originally posted and you are testing NBarsUpDnTPV2?

Please clarify that we are talking about the same script.

After verifying that we are talking about the same script can you please re run the test using COBC = true and post the new results? We should be seeing a trade happening only on the end of each minute bar.

The outcome using the first posted script on CalculateOnBarClose = true. After re running the test you should see that trades happen only on each minute and not intrabar like the results you posted.

We could also expect some differences in the backtest vs live run being that the backtest will be filled using the last price where the live will be using the ask and bid so the spread will effect this and should be kept in mind when comparing results.

I look forward to being of further assistance.

[James Beale]

These are strange results assuming the strategy is running on CaluclateOnBarClose = true and it is having trades intrabar when there is no way for it to trade intrabar using this setting or the code provided.

Yes, indeed!

I do see the strategy in the image is a different strategy than the one originally posted. Have you added intrabar logic to the strategy that we don't know about being that I am looking at the NBarsUpDnTP that was originally posted and you are testing NBarsUpDnTPV2?

Please clarify that we are talking about the same script.

Yes, we are talking about the same script. I attach NBarsUpDnTPv2.cs for your information. You'll see they are both attempting to place identical trades.

After verifying that we are talking about the same script can you please re run the test using COBC = true and post the new results? We should be seeing a trade happening only on the end of each minute bar.

"COBC=true" is true for both.

The outcome using the first posted script on CalculateOnBarClose = true. After re running the test you should see that trades happen only on each minute and not intrabar like the results you posted.

No. Sorry.

We could also expect some differences in the backtest vs live run being that the backtest will be filled using the last price where the live will be using the ask and bid so the spread will effect this and should be kept in mind when comparing results.

Agreed, but the differences are far too marked. Even the chart data isn't the same, despite the "Save chart data as historical" being checked. Perhaps this is the nub of the problem?