Export Tick Data from NT8 Replay Data File, to .csv

[Nate G]

Hi All,
Just asking to see if anyone has done this, tried this, heard of this, etc...
I want 3+ years worth of tick data for the ES or MES in a .csv file.
I have found tick data for sale online, but the data only goes back 180 days.
I'm thinking of taking NT8 replay data (of which I have 10+ years), and writing a script that plays through each bar, tick by tick, and outputs OHLCV into a .csv. Boom: there's my 3 years of tick data.
The only challenge I see is taking the data from one contract and meshing it with the data of the next contract. Not a show-stopper, but could definitely become tedious.

Has anyone already done this?
If, so, are they willing to share/trade this code?
Reading the above, does anyone see any show-stoppers I should be aware of before I start coding this? Any hints or bits of advice?

Thanks for reading this far!

Nathan.

[NinjaTrader_ChelseaB]

Hello Nate G,

The only way I would be aware of doing this is to run the playback and write the values to file with StreamWriter.

Below is a link to the reference sample.

https://ninjatrader.com/support/helpGuides/nt8/using_streamwriter_to_write_to.htm

[Nate G]

Hi ChelseaB,
You pretty much nailed it! Thank you! I'll go down this rabbit hole and see what I come up with. I'm pretty confident I can get this doing what I want. One more question: All my replay data files are broken out by their respective contract expiration dates. Is there anything inside NT8's replay data world that links all the contracts together in one continuous stream of ES/MES data? Something like ES ##-##? Just trying to find a way to help me avoid having to manually stitch all of these separate contracts date ranges together.

Nathan.

[NinjaTrader_ChelseaB]

Hello Nathan,

If you used BarsRequests to pull data for specific contract months, this might be possible. It would take some fancy coding to know at what time to start streaming specific contract months.

BarsRequests would be necessary as calling AddDataSeries() would only allow OnBarUpdate() to run if all added series are in session and have data.

The examples below may be helpful.

https://ninjatrader.com/support/forum/forum/historical-beta-archive/version-8-beta/87988-barsrequest-question#post1206656

https://ninjatraderecosystem.com/user-app-share-download/rolloverindications-2/

[Nate G]

Hi Chelsea,
Happy New Year
So the Stream Writer sample you provided is basically exactly what I needed, thank you for that. I just formatted the Date and Time output to match what I need inside AmiBroker, and that's it.

Just to make sure I'm climbing the right ladder here, I'd like to describe what I'm doing and get your input:
Let's consider two instruments: MES 03-2020 and MES 06-2020. At some point daily volume on the 03-2020 fades and 06-2020 volume picks up. The date on which MES 06-2020 volume exceeds MES 03-2020 is March 13, 2020. I deduce this from the file size of the .nrd replay file I'm looking at.

So I run a replay from December 13, 2019 to March 13, 2020. This gets me roughly 88 days of trading data (about a quarter of the year), so to me this makes sense. This outputs my OHLCV to a .txt file. I checked the data and it looks to be what is expected.

Next I do the same thing for the MES 06-2020 contract. Again I've got close to 85 days of data, I get another quarter. If I don' touch the text file, it just adds it to the end it it keeps going without issue.

My intention is to look at the data that my strategy would see in real life: when volume on one contract fades off, the new one picks up, so that is the data I am choosing to output.

My question is: Should I harvest all data during those phase-in/phase-out weeks? Trades are still occuring on the MES 03-2020 after that March 13 date. Should that data be considered? Or does it make more sense to roll over the contract (like I'm doing) because when my strategy is working, that's what it will see? I think I should go with the latter because I am more concerned with what my strategy will actually see, not necessarily the MOST accurate portrayal of all the transactions out there.

Please let me know your thoughts. I'm curious if there's anything I'm missing. Thank you.

Nathan.

[NinjaTrader_ChelseaB]

Hello Nathan,

I wouldn't be able to say if you should collect the data.

You could, in the logic of the script, compare the volume between two contract months and use the contract month that has higher volume.