Market Replay

[NinjaTrader_JoshP]

It would be nice if my charts could revert back to the last time series I had on my charts before watching a replay. It is tedious work to constantly change the chart settings to reflect the current day every time I switch between replays and live.

[NinjaTrader_Ray]

This has been resolved for 6.5 due out in fall.

[NinjaTrader_JoshP]

I think it would be beneficial to have NinjaTrader automatically know not to download any historical data when it is recording. On numerous occasions I have lost connection to my datafeed and when it automatically reconnects it wants to download historical data and that destroys the recording. Breaks in the datafeeds should just produce gaps in the charts instead of having a whole day's worth of recording wasted.

[NinjaTrader_Dierk]

Downloading historical data for charts will not affect the data recorded for market replay, since this is 2 different pools.

[NinjaTrader_JoshP]

Hmm. If losing connection isn't what is breaking my replays than I'm not sure what is. I start up my NinjaTrader to record data around 12AM or 1AM and just leave it running till market close. My "good" files all end up having start times around 4AM (premarket open PST) and go till 1PM (market close PST). My "bad" replays all seem to start at 5PM the previous day and end at seemingly random times. When I scroll through the replay all the data is jumbled up and I can't make sense of it. Any ideas on why these replays are turning sour? My success rate on replays is about 33% right now.

[NinjaTrader_Dierk]

Recording replay data stops are there are bad ticks coming in which are reported in the logs.

[NinjaTrader_JoshP]

Would these be considered bad ticks? Even if they stopped the recording for the few select stocks in question it doesn't explain why all recorded instruments are messed up.

7/11/2007 1:50:23 AM|1|2|OpenTick: Primary connection=Connecting, Price feed=Connecting
7/11/2007 1:50:26 AM|1|2|OpenTick: Primary connection=Connected, Price feed=Connected
7/11/2007 1:50:27 AM|3|4|Market data error for instrument 'AAPL Default': No data (1003)
7/11/2007 1:50:29 AM|3|4|Market data error for instrument '^COMP Nasdaq': No data (1003)
7/11/2007 1:50:29 AM|3|4|Market data error for instrument 'AKAM Default': No data (1003)
7/11/2007 1:50:29 AM|3|4|Market data error for instrument 'BIDU Default': No data (1003)
7/11/2007 1:50:29 AM|3|4|Market data error for instrument 'BIIB Default': No data (1003)
7/11/2007 1:50:29 AM|3|4|Market data error for instrument 'CROX Default': No data (1003)
7/11/2007 1:50:29 AM|3|4|Market data error for instrument 'CTSH Default': No data (1003)
7/11/2007 1:50:29 AM|3|4|Market data error for instrument 'DRIV Default': No data (1003)
7/11/2007 1:50:29 AM|3|4|Market data error for instrument 'ESRX Default': No data (1003)
7/11/2007 1:50:29 AM|3|4|Market data error for instrument 'FMCN Nasdaq': No data (1003)
7/11/2007 1:50:29 AM|3|4|Market data error for instrument 'GOOG Default': No data (1003)
7/11/2007 1:50:29 AM|3|4|Market data error for instrument 'JOYG Default': No data (1003)
7/11/2007 1:50:29 AM|3|4|Market data error for instrument 'KLAC Default': No data (1003)
7/11/2007 1:50:29 AM|3|4|Market data error for instrument 'LRCX Default': No data (1003)
7/11/2007 1:50:29 AM|3|4|Market data error for instrument 'NIHD Default': No data (1003)
7/11/2007 1:50:29 AM|3|4|Market data error for instrument 'NTRI Default': No data (1003)
7/11/2007 1:50:29 AM|3|4|Market data error for instrument 'PCAR Default': No data (1003)
7/11/2007 1:50:29 AM|3|4|Market data error for instrument 'QCOM Default': No data (1003)
7/11/2007 1:50:29 AM|3|4|Market data error for instrument 'QQQQ Default': No data (1003)
7/11/2007 1:50:29 AM|3|4|Market data error for instrument 'RIMM Default': No data (1003)
7/11/2007 1:50:29 AM|3|4|Market data error for instrument 'SHLD Default': No data (1003)
7/11/2007 1:50:29 AM|3|4|Market data error for instrument 'SMH Default': No data (1003)
7/11/2007 1:50:29 AM|3|4|Market data error for instrument 'UAUA Default': No data (1003)
7/11/2007 1:50:29 AM|3|4|Market data error for instrument 'VSEA Default': No data (1003)
7/11/2007 1:50:29 AM|3|4|Market data error for instrument 'WYNN Default': No data (1003)
7/11/2007 4:01:03 AM|3|4|MarketData (Bid) for 'GOOG' at 7/11/2007 4:01:01 AM exceeds max price gap (529.9/55.55)
7/11/2007 4:31:33 AM|3|4|MarketData (Ask) for 'ESRX' at 7/11/2007 4:31:31 AM exceeds max price gap (54.93/2000)
7/11/2007 4:31:33 AM|3|4|MarketData (Ask) for 'ESRX' at 7/11/2007 4:31:31 AM exceeds max price gap (54.93/2000)
7/11/2007 9:17:45 AM|1|4|Verifying license at primary server...
7/11/2007 9:17:46 AM|2|4|Session Break (Version 6.0.1000.3)
7/11/2007 9:17:46 AM|1|4|Current license: Type=SimulationOnly State=Verified StartDate=12/31/2004 EndDate=11/30/2099 Collective2 Dtn ESignal External OpenTick Replay Simulator Yahoo Gain FFastFill Hosted InteractiveBrokers MBTrading Patsystems TrackData TradingTechnologies AdvancedStrategyManagement AutomatedTrading BasicEntry Charting SuperDom
7/11/2007 9:18:08 AM|1|4|Automated trading disabled
7/11/2007 9:18:12 AM|1|2|OpenTick: Primary connection=Connecting, Price feed=Connecting
7/11/2007 9:18:14 AM|1|2|OpenTick: Primary connection=Connected, Price feed=Connected
7/11/2007 9:18:19 AM|3|4|MarketData (Ask) for 'ESRX' at 7/11/2007 9:18:18 AM exceeds max price gap (52.51/2000)
7/11/2007 9:18:25 AM|3|4|MarketData (Bid) for 'GOOG' at 7/11/2007 9:18:22 AM exceeds max price gap (542.61/55.55)
7/11/2007 9:18:25 AM|3|4|MarketData (Bid) for 'GOOG' at 7/11/2007 9:18:22 AM exceeds max price gap (542.61/55.55)
7/11/2007 9:18:25 AM|3|4|MarketData (Bid) for 'GOOG' at 7/11/2007 9:18:22 AM exceeds max price gap (542.61/55.55)

Notice how those events happen after a disconnect and the subsequent reconnect though.

I would also like to note that the image in my previous post is suppose to be a market replay of 7/11 not 7/10 as stated in the date. The recording was started as stated in the logs here at 7/11 1:50AM.

[NinjaTrader_Dierk]

Right.

These bad ticks ultimately will stop the recording for the affected instrument:
7/11/2007 9:18:25 AM|3|4|MarketData (Bid) for 'GOOG' at 7/11/2007 9:18:22 AM exceeds max price gap (542.61/55.55)

[NinjaTrader_JoshP]

Yes, but what I am having problems with is what is causing my 7/11 replay data to want to tell me the start time on ALL my instruments are 7/10 5PM instead of the usual 7/11 4AM.

[NinjaTrader_Dierk]

Not sure why there is confusion:
- apparently you connected to your provider at 5PM (7/10?) and left it running
- you experienced temporary losses of connection, but you did not (!) manually disconnect from your provider -> replay recording keeps going
- after the connection comes back, there were some bad ticks which halted the recording for the affected instruments

[NinjaTrader_JoshP]

Ok I think we are misunderstanding each other. Here are the steps I take each day.

1AM turn on NinjaTrader -> connect
4AM Pre-market data starts getting recorded
6:30AM Market opens; recording continues
1PM Market closes
2PM turn off NinjaTrader
rest of day I turn NinjaTrader on and off to watch replays/develop etc
repeat for every new day

Basically I start up my recording session for the new day at 1-2AM on the new day and not 5PM of the previous day. It should also be noted that all my messed up Market Replay data decided to have a start time of exactly 5:00:00PM of the previous day. Too precious for it to be my manual doing.

[NinjaTrader_Dierk]

Sorry, I don't know what would cause that. The recording starts as soon as you connect (1AM) and not 4AM (provided you have the recording option set).
I suggest monitoring and see if you can detect a pattern.

Also: make sure you don't change timezone settings on your PC back and forth. I would suspect this could throw off stuff.

[NinjaTrader_JoshP]

From my experience even though I connect at 1AM there are no ticks till 4AM and the instrument start time comes in at the first tick so thats why I say it "starts" at 4AM.

Do you think there are any activities I could possibly be doing in NinjaTrader during my "rest of day I turn NinjaTrader on and off to watch replays/develop etc" that could be damaging the replay data? I was thinking maybe watching replays was throwing it off, but it was already clarified earlier that I did not have to uncheck the "Run market replay recorder" button before watching replays.

I don't change my PCs timezones ever so that shouldn't be an issue. Hmm. I will try my best to pinpoint an exact activity for us as this seems to be a bug.

[NinjaTrader_Dierk]

Sorry, I'm out of ideas.

[NinjaTrader_JoshP]

Ok Dierk. I believe I have found out the source of this problem.

I did as I always do. Start recording at 1AM. A few hours after market close I closed my data connection to stop the recording. I went into Market Replay Connection to test if my data recorded properly and it did (note: no disconnects from datafeed today). 4AM start times and end times are set at whenever the latest tick for the instrument was in after-market hours. So far so good so I saved out my replay data folder to preserve the good data. Then I proceeded into reconnecting to my datafeed. My "Run market replay recorder" is still checked and also be aware that after-market is still in operation right now. My charts all wanted to reload historical data. It might be worth noting that all the data for the current day now no longer shows volume in the millions as it does from the real-time volume bug we sorted out earlier. The volume seems accurate now after the charts reloaded. I now disconnected from my datafeed without doing anything else and reopened the Market Replay Connection. Lo' and behold the replay now says 7/11/2007 5:00:00PM start time for all my instruments instead of the prior 7/12/2007 4:00:00AM.

Only thing I can think of is loading historical data is not as independent of market recording as we previously thought. Perhaps it has something to do with replacing the erroneous data volume that NinjaTrader remembers. Could it also have something to do with reopening the recording feed during after-market hours and the subsequent recording of ticks during historical data loading that is messing things up?