market replay and real time not same. kind of a problem.

[rocketstock]

So I ran the same strategy on market play and real time on today's market. Order was placed in market replay and not in real time. Also the indicators are off (for more details see values on right). See two pictures.

How reliable is market replay?
Are there any tips to manage this difference while developing strategies?
Is there any use at all in using market replay?
This is kind of a huge problem. :-)

[NinjaTrader_ChelseaB]

Hello rocketstock,

Playback with Market Replay data is very accurate, but note this pumps data out in 1 second intervals (which can cause small differences in when orders are filled).

Below is a link to a forum post that demonstrates how to find what is causing differences between real-time and playback.

https://ninjatrader.com/support/forum/forum/ninjatrader-8/strategy-development/100192-comparing-real-time-historical-and-replay-performance#post100192

[rocketstock]

Hi Chelsea, could you elaborate more on 1 sec? My experience is that its not accurate at all. In previous picture the market close data of indicators values are completely different. So that's when both market replay and real time scripts are stopped and not running. The values are still different. makes a huge difference in crossovers, slopes etc. Attached is another screenshot from today. Real time placed short order while market replay placed long order in the same time span. That's really off.

How can one remedy this? should we just recode entire startegy on real time? Wait for weeks for data capture for a longer time frame?

[rocketstock]

Hi Chelsea,

I really need some help on this. Months of hard work. I did some more debugging.

1. When I run playback from 3rd to 8th Feb. The playback and real time are different as shown in previous threads. That is when it gets to 7th with different trades on real time and playback.
2. However when I run playback from 7th to 8th the screen loads with trades already populated on 7th. Second pic. Which matches real time.
3. These trades were on 7th.
4. Not sure why.
5. Any settings I am missing for playback or should look out for. Like on bar close, on price close etc. Currently it's on bar close.

Thanks.

[rocketstock]

I restarted real time again this evening. This chart came preloaded with a whole new set of trades for same period as above. Third version. This one is a snapshot of same trades that came pre-populated on real time startegy this evening for period above. Terribly confused.

[rocketstock]

Hey Chelsea,

Sorry, I might have screwed up. Not sure. I assumed the sim with live data was exact as real market. Could that be the reason? Read on your docs that sim adds some simulated data?

However if SIM101 with live data is not the reason for above problems then I still have a problem. :-)

Is market replay more accurate than sim101 on live data?

Thanks,

[demarcog]

rocketstock,

I wanted to thank you for all the hard work, a someone who has literally worked "man years" on resolving one if not the most important function of any product like NT8.

I am not a C# developer although spent many years coding and backdating, then to ALWAYS regardless of platform tried to code around the discrepancies between live market back tested results.

However I've worked for 30 years in IT as a Trading System Quantitative Analyst, not a "real quant". IMHO profitable trading system development is as much "common sense' more art than science.

It's a massive problem for all NT8 users, only some haven't quite realized how financial devastating it will become. There is already too much "psychological" or gray areas a back test should not be one of those. Firms are very quick to "hypothetical" results and chalk it up to what most of us would call slippage. The "slippage" myself and others are trying to resolve goes many many orders of magnitude greater then this.

Renko or Unireko in particular is quickly pointed out by NT8 as having some underlying flaw that renders it nearly impossible to back test accurately

It's explained that a bar can be redrawn which makes it impossible to back test as if to say unirenko is not quite a valid chart type. I trade with a room with dozens of Renko or Unirenko charts and once drawn it doesn't seem to ever redraw itself as they bars we look at together in real time and after trading and the charts are always the same.

If it Unirenko were an invalid chart type with people trading all day everyday they why is it NT8 can keep perfect to the exact penny the P&L of every trade.

It defies logic to claim users can accurately trade Unirenko implying NT8 can successfully process every aspect of trading down to the penny, commission rate, exchange fees etc, without any problem we know NT8 can processes Unirenko tick by tick.

But its' the back tester that has a problem, maybe they should convert the Unirenko back tester from the real time trading engine because at that level all is understood.

I'm not so sure Market replay is highly accurate. The problem you can only run 1 strategy on 1 symbol at a time and that is very time consuming.

For NT8 to know how to process Unirenko the code/knowledge has to be there and maybe the problem is making the connection between the two.

I would truly help many of us from becoming another trading statistic, and I'm on my last leg right now, by simply applying the code/resources that already exists in the code to work while back testing.

Thank you very much for keeping the issue alive as we both realize it's a make or break issue and those of us unhappy with back testing Unirenko should as paying customers are entitled and all the Support staff will and have done everything then can and nothing would make them happier.

Contact me anytime rocketstock I'd love to just talk shop with you. Two hinds are better then one a bndest of luck, PM me anytime, they removed my email address.

Glen Demarco

[rocketstock]

Hi Glen thanks for your reply. I did read some of your posts and obviously you know lot about all this. I am at a beginner level with NT. All I am trying to figure out is why is there such a sizeable discrepancy in my indicators basically shifting and their values are so different that trades are going completely wrong in sim101. Between market replay and sims101. These could be newbie concerns but I have spent entire day today on reading up on this and no solution yet. If the indicators are shifting below in my pictures then it should be proportional shift to price above because they are being calculated on two things price and volume. If that was a latency issue everything should move together. Doesn't make sense. Their actual values is a whole other problem.

I want to start with the most basic questions I cant find on this forum.

1. Which is more accurate 1. market replay or 2. sims with real live data? By more accurate I mean which one will give a closer picture of live trading.
2. Which data is more accurate in terms of numeric values that in turn form numeric values of various indicators. Because clearly both replay and sims are showing different data for my indicators.
3. Whichever one it is, how much is the deviation when one goes to live trading? I don't care about slippage that much, because my trades are not rapid firing like HFT. I stay in for over 45 min to few hours. So a few points here and there is fine due to latency. However I cant have a long trade in market replay and the same time frame have a short trade in sims. :-) That's funny but its happening because of different indicator values I know for certain.
4. My strategies don't care if the data received + calculation is 3 seconds late. As far as I get the right data for my strategies, I can fire a long or short trade even if i miss the best entry 3 seconds ago. I just need the right accurate market data. Understand that could decrease profits a lot but I'm playing direction and stay in for a while. Or I exit quick.


You are correct, I'm extremely concerned at this point in time about the strategies not working well enough in live trading. The whole premise is based on market replay.

I also cant seem to find what real market data is being changed by sims algorithm. Still looking. All I can find is this link NinjaTrader 8.

Is it just delay or actual market data is not correct in sims or replay. Reason being in picture above the sims account indicator values (on right side of picture) at market close are different from market replay account at market close. its clearly not just delay if the values of indicators on the right are different when both strategy scripts are at rest and not running. Why the difference in values on same indicators on same trading day close?

Happy to discuss more. Hopefully awesome NT team can shed more light too. Terribly confused which picture is correct. Or are both even correct. ha

[rocketstock]

found this really old thread on this. I am not concerned about delay as much as my indicator calculations are getting out of sync with the price in sims. Both should be delayed equally in sims I imagine. price and indicators should move in tandem. Plus different values of indicators (right tab) on market replay and sims is baffling. Even after markets close and no strategies are running.

Ninjatrader realistic fills? - NinjaTrader Support Forum

[NinjaTrader_ChelseaB]

Hello rocketstock,

Market replay is real-time data recorded from a live market and then played back in playback.

A sim order placed when connected to a real-time data from a live market, is also real data from a live market.

Indicator values would only be different if the data source, or data type is different. Many trades will mix historical and real-time data which will alter the results.
This is discussed in the forum post with a video on comparing, which uses a custom trading hours template to prevent historical data from loading and altering the results of the tests.

Please provide the output written to files as demonstrated in the forum post and I will be happy to assist with analyzing the output.

[rocketstock]

Hi Chelsea,

Thank you so much. This problem is the most worrying so far :-) I am still learning NT so bear with me. I am nearly done with strategies and this monkey wrench threw me off.

1. What do you mean by "Many trades will mix historical and real-time data which will alter the results"? Which one is adding data? Market replay or Sims? If I'm running sims what should be my confidence level?

2. Which is more accurate data to consider for live trading. Market replay or Sims? Keeping delay aside.

3. Your reference to video is this? https://www.youtube.com/watch?v=2iM5BPpcUBg

4. Please see attached files. At market close why are indicator values different on right when scripts are not running? One is market play other is sims. Please see bottom 3 indicators and their values when all is set and done.

5. Hard for me to generate output files for this. Not sure how to do it (if you want to guide). Last night sims connection was down. I'll also look at custom hours template.


Thanks,

[NinjaTrader_ChelseaB]

Hello rocketstock,

In the post I have provided I have demonstrated how to write information about the data and conditions to file. The video shows interpreting that output.

See the link to the forum post below about how historical data will load before real-time data.

https://ninjatrader.com/support/forum/forum/ninjatrader-8/strategy-development/95921-auto-trail-code-for-strat-that-is-backtestable#post782442

The pictures do not show why anything is happening. You want to evaluate why the script is evaluating, then you need to debug the script and output the information the script is using. This is what is demonstrated in the video.

The data from real-time is as accurate as the connected provider. Market replay data is a recording of real-time data from.

These have the same accuracy as far data. Historical data, however, is processed differently.

The data is pushed to the market replay, in order, in one second intervals.

Either is filling with the simulation engine which uses price and volume to determine a fill. This is a simulation whereas a brokerage order would fill with market dynamics and someone on the other side of the trade.

[rocketstock]

Hi Chelsea,

I'll do into details and set up that output to figure out "why". This is the most important thing on my list else everything goes wrong. I will work on this and get back to you. Need some elaborate time. Thank you.

[rezik]

The data is pushed to the market replay, in order, in one second intervals.

NinjaTrader_ChelseaB is there any way around the 1 second interval? Reg key? Config file setting? Something to allow the NT application to push to market replay at the proper time frames?

Why the 1 second? I've seen a thread suggesting performance optimization...but, for those people that have fast machines I'd be willing to risk to see how it behaves. Can we do that? If not, what does the roadmap hold for replay in coming iterations? Are there any future sprints planned to address this? Do we as a community have any visibility into the roadmap of replay? If not, how do we get a feature request in?

Thanks!

[NinjaTrader_ChelseaB]

Hello rezik,

There would not be a workaround or a way to modify the internal playback engine.

I will submit a feature request on your behalf for the playback to push data faster than 1 second. Once I have a tracking ID for this request I will post in this thread for future reference.