Market Replay depends on Days To Load

[ATI user]

It is very difficult to test a strategy using Market Replay for the following reasons:

1. you can replay the same code on the same day from the same time and get different results based on:
a. how many days are loaded into chart
b. which day you replayed prior to the current test

2. replay does not come close to live trading results even though each tick is recorded live for replay...i.e. not downloaded from ninja replay server

For example, I have NQ data recorded from 13 Aug to present. I open a replay session for 16 aug and load the strat....replay and get poor results....I change the replay data to 13 aug and replay that...then go back to 16 aug and get great results as expected. I seems there is a bug in replay relative to historical and replay data...i.e. by changing date back to 13th the replay/historical data is reset?

Also, running the same code live in sim101 and recording that data for replay and then running same code through replay101, the results are not even close.

Problem with testing a strat is that replay/simulator bears little relation to live trading results.

[NinjaTrader_PatrickH]

Hello ATI User,

Thank you for your post.

For your first two questions here please advise if you have the From and To fields of the Replay Control window set to specific dates and you are changing these dates or using the Replay Slider or Go To... option to go to a specific time within that range.
Or are you changing the From field to the date you wish?

I would ask the same questions on the NQ data item, what are the From and To fields set to? Do you change the From field or use another method to go to the 16th?

You need to ensure the same amount of data is used when comparing real-time to Market Replay, because the strategy will calculate differently if started on one bar versus another.

I look forward to assisting you further.

[sledge]

Does your strategy rely on previous days data (like 3-4 days worth?).

What kind of time frame and what type of bars are you using?

Is this a multitime-frame or just NQ strategy?

It is very difficult to test a strategy using Market Replay for the following reasons:

1. you can replay the same code on the same day from the same time and get different results based on:
a. how many days are loaded into chart
b. which day you replayed prior to the current test

2. replay does not come close to live trading results even though each tick is recorded live for replay...i.e. not downloaded from ninja replay server

For example, I have NQ data recorded from 13 Aug to present. I open a replay session for 16 aug and load the strat....replay and get poor results....I change the replay data to 13 aug and replay that...then go back to 16 aug and get great results as expected. I seems there is a bug in replay relative to historical and replay data...i.e. by changing date back to 13th the replay/historical data is reset?

Also, running the same code live in sim101 and recording that data for replay and then running same code through replay101, the results are not even close.

Problem with testing a strat is that replay/simulator bears little relation to live trading results.

[sledge]

Does your strategy rely on previous days data (like 3-4 days worth?).

What kind of time frame and what type of bars are you using?

Is this a multitime-frame or just NQ strategy?

Ok , I MARKET REPLAYED my ES "4 range" bars which is an intraday strategy and doesn't rely on previous day (only current intraday).

Basically, what I saw, is that the 1st day, no trade was taken. (I need to review this strategy, maybe I missed setting up some variables???).

But each day after, was the same entry/exit. (Sample was small. 3,2,4,3 days length were tested, until I saw the pattern of missing the first day, and then the next day matches and the day after matches)

I screenshot the chart that failed, and Market Replayed starting 1 day earlier, the charts look identical (except for the missing trade on the day of start chart).

Range Bars are tick based, and I remember some posts here about NT standardizing how they start. (ie, my tick chart would look the same as your tick chart, no matter where are start times were).

Before I debug my strategy, I'm going to replay and test 3 minute charts and see if i lose another $1,500 with this strategy. (Hey, 2/14/12-2/17/12 was tough ).

[ATI user]

For your first two questions here please advise if you have the From and To fields of the Replay Control window set to specific dates and you are changing these dates or using the Replay Slider or Go To... option to go to a specific time within that range.
Or are you changing the From field to the date you wish?

I would ask the same questions on the NQ data item, what are the From and To fields set to? Do you change the From field or use another method to go to the 16th?

You need to ensure the same amount of data is used when comparing real-time to Market Replay, because the strategy will calculate differently if started on one bar versus another.

in all cases From and To are first set to the same date then GoTo 9:30

all charts are 1 min with a Session template for 9:30 AM to 4:00 PM...and so should always start at 9:30 exactly....the code starts on FirstBarOfSession

trading live is the same session template on 1 min chart...ninja records for replay while trading live and therefore should replay exactly the same.

[NinjaTrader_PatrickH]

Hello ATI user,

Thank you for that information.

Please send a note to support[at]ninajtrader[dot]com with 'ATTN: Patrick' in the subject line and a reference to this thread in the body of the e-mail: http://www.ninjatrader.com/support/f...ad.php?t=52291

Please attach the Market Replay files you are using during your test, you will find these files in the following directory on your PC: (My) Documents\NinjaTrader 7\db\data.
The files will be listed in a folder dated for their respective date ("20120907"), the files will be NTM and NT2 file types.

Please also include any steps we can take on our end to reproduce the same behavior you are experiencing on your end.

I look forward to assisting you further on this matter.

[ATI user]

strangely if I replay going backwards in time it replays properly

I have NQ and other data recorded for replay from 13 aug thru 12 sep.

previously I had been replaying 3 or 4 big days then starting at 13 aug I replayed all remaining days going forward...this required changing the DaysToLoad 6 or 8 times in order to get the strategy to make trades

starting with From and To both set at 12 sep and then using GoTo 930am, with 22 days loaded, then load/enable strat worked fine....then disable strat, change To from 12 sep to 11 sep, GoTo 930, load and enable strat also worked fine...this procedure worked all the way backwards in time to the first day 13 aug

it would still appear there is a relationship between historical and replay data that Market Replay can not consistently resolve... unless using a procedure as above....or only replaying one session with coincidentally the correct number of days loaded

I have been using replay extensively for 8 years and it has never been consistent with live results. Today the new NQ contract started and I am recording all data on 1 min charts with 22 days loaded. I will then replay on 1 min chart with 22 days loaded and see if live and replay are exactly the same... will update post here with results

[ATI user]

the problem is not the number of days loaded but rather going forward or backward in dates

changing the days loaded resets something in replay .... going backward the reset is automatic and the days loaded does not need to be changed... going forward does not do this reset and therefore often requires changing days loaded

[NinjaTrader_Bertrand]

Hello ATI user, could you please send in the files Patrick requested so we could look into reproducing the issue you ran into?

Would this happen for the SampleMaCrossOver strategy for exampe as well, or would it need a MultiSeries strategy to trigger the issue?

I would expect the results to come close, but not be exactly the same in replay - this is for example due to potentially difference source data used and live executions not being exactly the same as the simulated ones.

[ATI user]

I would expect the results to come close, but not be exactly the same in replay - this is for example due to potentially difference source data used and live executions not being exactly the same as the simulated ones.

I appreciate that live and replay will only 'come close'...this is expected and acceptable

what is not acceptable is running the same exact code with the same exact data with the same exact atms on a 1 min chart with a 930 session start, which should therefore start on the exact same tick for all data series EVERY TIME, and getting vastly different results depending on days loaded or which date was replayed last or whatever...

for example, one replay of YM for 21 Sep produces a PnL Hi of $2,183... a few days later the replay PnL Hi is $48...the data did not change...the code did not change...the atms did not change.... how replay loads/processes the data must have changed

it does not matter that there is a difference between live and replay.... as far as developing/testing goes
it DOES matter however if replay is inconsistent....i.e. how can you develop code when the replay changes without code changes?....you do not know whether your code change caused the different results or replay....so you go back to the original code and can not get the original results.... impossible

this problem is based on a multi-series strategy.... so I am in the process of developing a stripped down strat based on ninja sample and will forward it plus the data when I get it to reproduce the problem.... this because I am unable to proceed further developing my own code.

[NinjaTrader_Bertrand]

ATI user, thanks for the clarifications - I agree with your thoughts and will appreciate testing the sample you're looking to forward. I'm using Market Replay extensively here as well almost daily and have not yet run into such inconsistencies you report. For any MultiSeries script it would be important to keep in mind that the BarsRequired would have to satisfied for all series involved before OBU() is getting called, so depending on this setting if not enough data is loaded, it could lead to unexpected results.

[ATI user]

For any MultiSeries script it would be important to keep in mind that the BarsRequired would have to satisfied for all series involved before OBU() is getting called, so depending on this setting if not enough data is loaded, it could lead to unexpected results.

how can this be an issue when the same data and same code are replayed?

i.e. both replays on 1 min 9:30 chart with 33 days loaded...code has not changed....how can results change?

[NinjaTrader_Bertrand]

Ok, in this case I would need to reproduce here to investigate - I would not expect a change as well. Are any custom resources used in the strategy?

[ATI user]

custom resource?
just adds instruments...all in 1 min timeframe...and does calcs in primary series
I suspect the problem here is that replay can not consistently align/sync the various series to the same tick each time...I can see it loading series next to the 'Connected - Replay' in green.... more active sometimes than other times....I think replay gets corrupted at times...through trial and error this problem seems to be reduced by replay from most current date backwards...i.e. it seems that replay gets the data in sync for the 33 days for the whole 6 weeks being replayed, then going backwards does not corrupt the data....going forward definitely corrupts the data in so far as not all data series are on the same tick for 930 open until you change the days loaded in which case it gets data back in sync

[NinjaTrader_Bertrand]

A custom resource would for example be a data base or txt / signal files you would like to access as well. So to isolate this more out, would you see issues with your data and settings with our SampleMultiTimeFrame as well?