My experience- market playback sims & live for anyone starting out. plain horrible.

[rocketstock]

1. After having spent 2 months. Countless hours using playback to develop very complex strategies, I have realized that playback is completely useless.
2. My strategies are very sensitive to crossovers, slopes and values of indicators. They are basically useless now on live data.
3. Have to start from scratch again.
4. No one from NT support informed me despite asking neither could I get a straight answer from them in past three days. You can ask the questions 10 different ways and there's no knowledgeable answer. Circles after circles. They are more worried about organizing this forum postings and their internal support process than about helping customers with real money on the line. I mean what is the end goal of this forum to make customers successful or organize their process of posting replies. The process is more important than the objective of the process. Ridiculous.
5. Whether the platform is nice or nor we will find out. I have to start out again.
6. Playback and Sim101 are complete junk. Do not use for startegy building.
7. I am building everything again on live data now.
8. Just terribly peeved off. All NT had to do was post somewhere that playback and sim strategies would not work exactly on live trading. Perhaps a pop-up menu and I wouldn't have wasted 2 months with very long days and nights. Ya cant just say well there might be slight differences in chart data. Well those slight differences cause major difference on live trading orders. I had bull order on playback while live trading placed a short order for same period. That's not slight difference.
9. On top of that one NT support person tells something cannot be done and another tells shows me screenshots how it's done. But yes lets organize this posting board as highest priority.
10. Completely frustrating.

[NinjaTrader_Jim]

Hello rocketstock,

Thanks for your post.

Development can be frustrating, and I see that your frustrated, but Playback and Sim are useful tools for testing strategies. They can give us more true-to-market results than can be achieved with backtesting historical data. Playback is is also helpful to repeat occurrences that happened live so you can take debugging steps.

Our Strategy Development Process guide gives a good path for developing strategies:

https://ninjatrader.com/support/help...nt_process.htm

What exact challenges are you facing when testing your strategies with Playback and Sim?

We look forward to hearing from you.

[rocketstock]

Hi Jim,

Thanks for your kind words. The thing is my open and close conditions are dealing with second degree slopes. That gets multiplied because I have over 5-6 indicators with various time frames which leads to over 20 conditions for open and close each. A slight variation in data magnifies the situation. So what happens is that the indicator conditions are completely off with live data. Sometimes the slope of slopes are crossing over 45 minutes later with live data as opposed to playback. On a 3 min chart. That's pretty wide. I have to collect my thoughts on this problem. Get some data and screenshots for you. Currently I plan to start from scratch again. I will seek your help once I make a new plan with live data. Have something meaningful for you. The challenge now is with live data I have only so much sample. I can't run it for a longer time frame to cater to various market conditions that I did get to experience in playback. Unless there is some guidance you have for playback. Cuz I have lost hope with it. Any settings etc? I am also thinking about recording live data but your docs say its the same as playback data. Dont know honestly. Needs a good startegy to develop the strategy :-) If you have any thoughts or documents or experience you would like to share, it would be most appreciated. Have a great day.

[NinjaTrader_Jim]

Hello rocketstock,

Our futures Market Replay data is recorded from CQG and it is an exact snapshot of how the data was received. If you have a different data source this could result in some variances between recorded Market Replay data from CQG and your own, but if you record the data yourself it will indeed be the exact same as what the platform saw when that data was live.

Differences in data will cause differences in results, but this may not be your root issue. You can also open charts to examine data to spot differences. Testing a small set of data to see if there are differences, and then examining the bars would be one way to see if data is a difference.

What you can expect to be consistent would be data based signals that are generated when the strategy is enabled on the same set of data, with the same settings. Enabling the strategy at the same time with the same data loaded on the chart is important so the historical portion matches and the real-time portion matches is important as well.

You could not expect consistent results regarding actions based on order fills, because Sim adds random delays between switching order states, so you can end up with different fill prices than in Playback. In this case, Sim tries to act more like a live account where orders actions take time to reach the broker/exchange and come back to NinjaTrader. Playback tries to be consistent with each test, so no delay is added there.

If you are approaching this again, try to focus on making sure the same amount of data is loaded on the chart before you enable the strategy, and the strategy is enabled the same time it was enabled when trading live. Then specifically examine data based events that control your entries. The data based triggers should be consistent then because we are looking at the same data, with the same code and same settings, and the order fill factors differences are not involved.

[rocketstock]

Thanks Jim

1. I am using CQG and have a new brokerage account with NT.

2. When I used sim I used it with live data.

3. Good to know live recording will be exact as when I saw it with live data.

4. Like your idea about testing data for small differences. I think I'll try that. I just feel overwhelmed with all the conditions I have based on playback based on 2 months of work. However I'll try that first. Tweak and see if things work.

5. BTW I use 3 min chart but stay in trade for 45 min to a few hours.

Questions

"You could not expect consistent results regarding actions based on order fills, because Sim adds random delays between switching order states, so you can end up with different fill prices than in Playback. In this case, Sim tries to act more like a live account where orders actions take time to reach the broker/exchange and come back to NinjaTrader. Playback tries to be consistent with each test, so no delay is added there."

1. I used sim with live data. Is above still applicable? Or am I not understanding sim account correctly? Does it add delay with live data too?
2. Does sim (with or without live data) only add delay or does is massage any data?


If you are approaching this again, try to focus on making sure the same amount of data is loaded on the chart before you enable the strategy, and the strategy is enabled the same time it was enabled when trading live. Then specifically examine data based events that control your entries. The data based triggers should be consistent then because we are looking at the same data, with the same code and same settings, and the order fill factors differences are not involved.

3. I think I understand you on this point but not completely. which options are you referring to? playback or sims with live or my data recording?

So if I was take a tweaking approach instead of building from scratch. What would you recommend?

1. Record live data and compare to existing scripts on live data?
2. Record live data and compare to both existing scripts and playback?
3. Dont use playback. use sims with live only? Or use playback too? Just looking for the best combo that mimics real life conditions. Some minor 1 or 2 second delay in fill is fine for my strategies.
4. I dont do any backtesting.

[NinjaTrader_Jim]

Hello rocketstock,

Sim101 accounts will use live realtime data to simulate order fills. The data is not massaged, but the order never leaves your PC. NinjaTrader adds some random delayed between switching order states, (Submitted, Accepted, Working, Filled) to simulate latency that would be seen on a live order submission to a live account. This in turn would result in slippage being added to the order fills.

Playback uses the same recorded data to simulate order fills, but there are no delays added in Playback that simulate order latency. This is intentional, so when you repeat the same test in Playback multiple times, you get the same Playback results.

Delays are not added to live orders. The order naturally takes time to reach the broker/exchange.

Regarding #3, my comments were if you were to perform the same tests in Playback as you did in Sim. Your own recorded market replay data would always be exactly what you saw live as you saw it, but I do not immediately suspect data variances between downloaded and recorded Market Replay are relevant for your case.

Most often, variances between sim and playback are that the tests are not set up the same, and that results involving order fills will always show variance.

If you focus specifically on when data based signals trigger in OnBarUpdate, this will be consistent as long as you keep the factors of code + settings + data the same. Keeping the data the same involves making sure the chart was loaded with the same amount of data [and the same data] before enabling the strategy, and the strategy was enabled at the exact same time.

Past this, make your tests simple and focus on small parts at a time. The key to solving any problem is to simplify it so it can be studied without being overwhelming.

I have attached some videos I recorded for another client regarding how to make consistent Playback tests. The same rules for setting up consistent Playback vs. Sim tests would apply, but we should note that the order fills will not happen at the same prices, due to the delays added with Sim. However, what is data based and triggered in OnBarUpdate (I.E. moving average crossovers) would be consistent.

Part1: https://drive.google.com/file/d/1L1m...w?usp=drivesdk

Part2: https://drive.google.com/file/d/1cqa...w?usp=drivesdk

Part3: https://drive.google.com/file/d/1Ohw...w?usp=drivesdk

[rocketstock]

Thank you. I will study the information and get back. Appreciate it a lot.