If Playback Mode is the most accurate way to back test, then we are in trouble

[demarcog]

Hi,

Is there anyway using Ninja Trader of running a back test that bears any semblance to an accurate report?

We all know the extensive list of reasons why the Strategy Analyzer is inaccurate. OHLC data is such that an accurate fill price cannot be provided.

The we are told to use Order Fill Resolution of High specifying Tick and that will be close to accurate. as now tick data is used to calculate fill prices.

I have posted examples where the results are such that strategy that looses $20K in the back test and makes $10k in Playback.

Then the "closest" possible way of testing is to run playback mode. Although only one day and one symbol at a time can be downloaded suppose someone actually downloads 90 days of data for 20 symbols or 1800 individual downloads just to get 3 months of data.

At least then the results will be accurate, or near accurate?

Not even then!

Here is a Playback results that individually made about $20K and the Account page show loosing $5,000!

I absolutely given up and came to the conclusion that it's impossible in NT8 to get an accurate back test anyway at all.

It's beyond my comprehension that that most basic, simplest too that every other platforms has and can be done in excel in about 30 minutes with the trades data is either incapable or refuses to provide an accurate back tester.

People are paying a lot of money, ALOT of money for commissions and the product one would think the least the product would provide is an accurate back tester.

I and others earn their living, and support themselves and families trading and without an accurate back tester NT makes is almost impossible to gauge the profitability.

I think they should drop everything else anyone is doing, all hands on deck and fix the fragging strategy analyzers and playback recorder.

Thank you.

glen

[NinjaTrader_Jim]

Hello demacrog,

Thanks for your post.

I recall a similar post of yours where I had explained the differences between historical processing and mimicking realtime processing in depth.

That thread can be found here - https://ninjatrader.com/support/foru...45#post1139645

This comes to a trade off:

• If you want absolute accuracy in a backtest, it will be an extremely resource intensive operation requiring full tick data and level 2 data to completely recreate market movement as it has transpired. We have this, and it is using Market Replay in the Playback Connection
• If you want to get an idea for strategy performance over extended periods of time, it would be a trade off for the amount of data involved and an an extremely resource intensive backtest. We have this with the Strategy Analyzer.

If you want to have orders filled with intrabar granularity in a backtest, use High Order Fill Resolution, or submit orders to a single tick data series.

If you want to have historical data processed processed OnEachTick or OnPriceChange, use Tick Replay. (This should be combined with submitting orders to a single tick data series since Tick Replay is not compatible with High Order Fill Resolution.)

In our NinjaTrader 7 Help Guide, we outline the process for developing a strategy and where these tools can be used in an efficient manner. I.E. Design the strategy, backtest and optimize, realtime test (or use Market Replay to mimic realtime,) then test on a live account.

https://ninjatrader.com/support/helpGuides/nt7/the_strategy_development_process.htm

We look forward to assisting.

[demarcog]

Jim,

H[*]If you want absolute accuracy in a backtest, it will be an extremely resource intensive operation requiring full tick data and level 2 data to completely recreate market movement as it has transpired. We have this, and it is using Market Replay in the Playback Connection[

The point I was making and have made in prior posts is I have done exactly what you suggested.

The screen I posted shows was run in Playback Mode for the same run, one tab show a profit of $20 and one a loss of $6000, one is incorrect.


I have followed to the letter each of the methods described and posted the inconsistent results as does this post which demonstrated a single Playback Reports show a profit of $20K in the strategy tab and the Account tab shows a loss of $6000. This is the most accurate method of testing? Now if I run these strategies with real money at the end of that week what will be the approximate result up $20k or down $6k, because that is the choice I am faced with.

I understand everything you have said and appreciate the help and link.

Do you agree that this is an accuracy and someone in development should fix it code problem causing it?



thank you

glen

[NinjaTrader_Jim]

Hello demacrog,

If you see different results in Playback runs it is likely due to the strategy not being enabled with the same data, same settings, and starting the strategy at the same time. This is important to keep in mind with Playback because the data processed before the strategy is enabled is historical processing, and if we also want realtime processing to match, both strategies need to transition to realtime processing at the exact same time.

Please see the demonstration below where this is kept in mind, and the results match.

https://drive.google.com/file/d/1YVGhPeLHE7fCDqnJGMXIaIuAQDOuETQT/view?usp=drivesdk

Please perform the same test with SampleMACrossover as I have to ensure the factors above are kept in mind.

We look forward to assisting.

[demarcog]

Hi Jim,

As always thanks for the response and references.

Your response assumes the strategy and account tab were for independent runs.

There was only one Playback mode run.

I started all the strategies enabled at the beginning of the data, it stopped at the end of the data, in the Strategy Tab up $20k, Account Tab down $5k.

What more can I say obviously one number has to be wrong.

It would be great if the back testing defects I'm taking a lot of time to document are fixed in some future release but not acknowledging them won't get us any closer to that goal.

Thanks again I do appreciate your time and effort and know this is not your area of responsibility and unfortunate and apologize as I feel like I'm being unfair to you but just reporting what's happening.

Glen

[NinjaTrader_Jim]

Hello demacrog,

So I am clear, you mean that if you enable multiple strategies with Playback and start them all at the beginning of the session the following happens?

1. All strategies start with $0 Realized PnL in the Strategies tab
2. After playback completes the Realized PnL in the Strategies tab does not add up to the Realized PnL of the Accounts tab?

Demonstration - https://drive.google.com/file/d/1UIr...w?usp=drivesdk

If this is not what you are referring to, I cannot make any more guesses and I will need clear context to test the same thing you are seeing.

Please demonstrate using the SampleMACrossover strategy, with steps to reproduce the issue so I can take the same steps on my end and give my analysis. Videos like I have created are helpful to convey a specific test.

If this cannot be demonstrated with the SampleMACrossover strategy, please demonstrate the issue with a barebones test strategy and attach that simple strategy with steps needed to reproduce the behavior (or a short video like mine) that you see as incorrect.

Exporting as source code - https://ninjatrader.com/support/help...tAsSourceFiles

[demarcog]

Jim,

Yes that is correct exactly. I would have never taken up so much of your time with this had that not been the case.

The issue is intermittent and I will try and reproduce it as best I can. The point was this occurred and I've experienced similar issues with Playback Mode and given this is the most accurate way we have of testing it is an issue.

I appreciate the help, it's actually not a major problem and I've seen it happen before especially when a few dozen strategies are run.

There are other back testing issues which are more concerning and I will be documenting and submitting as soon as I can.

Thank you very much for the help, I was trying to be a clear as possible and posted the screen shots and thought it apparent that they screen shots were for the same run.

glen

[demarcog]

Now I am back to the original message telling me insufficient case in the account when there is $10,000,000 in cash available

I sent 4 sets of traces and logs regarding this problem how are things going on that end?

thanks for the help,

glen

[demarcog]

HI,

Can't do any sim trading because of these error. So my Playback Mode in inoperative and now my SIM trading is inoperative.
Anybody have any idea how to resolve this problem I have the default risk settings....

Any progress on the troubleshooting on your end I'm sending the log and traces again.

thanks
glen

[NinjaTrader_Jim]

Hello Glen,

I have received your log and trace files over email.

With the rejection on RTY, I do not see any successful order submissions on this instrument.

2021-05-21 11:20:38:180|1|32|Order='1d663821a1a7407abeb9cb4a1 da1f777/Sim101' Name='Short1a' New state='Rejected' Instrument='RTY 06-21' Action='Sell short' Limit price=0 Stop price=0 Quantity=1 Type='Market' Time in force=GTC Oco='' Filled=0 Fill price=0 Error='Unable to submit order' Native error='Order '1d663821a1a7407abeb9cb4a1da1f777' can't be submitted: No risk settings defined for this instrument.'

The error means no risk settings are defined for this instrument and would be seen when a Risk template is applied to the account, but no risk settings for that instrument are defined in the template. The Playback account inherits from the Sim101 account, so if you want to either remove the risk template, or change the risk template applied, you can disconnect from Playback, connect to another connection, and then right click on the Sim101 account in the Accounts tab of the Control Center and select Edit Account to change the risk template or to set it to None.

If you want to edit Risk templates, this is done from the Control Center's Tools > Risk window.

As for the CashValue related error, I see the error message in your screenshot, and the Accounts tab is in view, but I do not understand why you have the CashValue column on the Accounts tab condensed where we cannot read the value there. The error means that the Cash Value as seen in the Accounts tab of the Control Center is less than the Minimum Cash Value set in the account's properties. In the case of the Playback account, it is inherited from Sim101, so you could again disconnect from Playback, connect to another connection, right click on the Sim101 account, and select Edit Account to change the Minimum Cash Value of the account.

Remember that Sim101 trades persist in the NinjaTrader database and if you expire the Sim101 funds, you can right click on the Sim101 account, and select Edit Account where you can click the reset button to reset the Sim account and restore funds.

The Playback account will reset when rewinding or disconnecting/reconnecting.

Please let me know if you have any additional questions.

[demarcog]

Jim,

Thank you Jim I appreciate the response.

All these errors occurred in SIM as I mentioned in my first sentence, not Playback. My SIM is effectively disabled all day today very much like the Playback Mode was, same symptoms, same errors only much more painful in SIM.

So all the information you provided about Playback is good stuff to know and I thank you and appreciate the relationship between Playback and SIM but I'm not using Playback.

The cash value was $500,000. You are asking me why the columns are compressed to appear as they are appearing?

That's how the account tab appeared on my screen, perhaps that's another problem that someone should be working on.

Why would I go through the 5 minutes it would take to compress or change those columns what you see I what I got and it not pretty to me either.

Maybe moving from a high resolution to my laptop screen as I use the big monitor for charts might have thrown the columns off I have no idea but I didn't resize anything.

I get around all of these problems by deleting the data base, resetting the SIM account, restarting NT and it runs for a while.

What I was pointing out was this never happened in SIM before.

I can maybe live without Playback mode but can't live without SIM mode. That was the main message I was trying to communicate.

Your response is with all due respect and I could be wrong but usually implies it's something I am doing.

When I had to diagnose code I was responsible for I would look at the stack trace, see what module is involved, pull up the source code for the modules that failing or issuing an erroneous message. Just curious but have you done that yet as eventually we are going to end up there.

I appreciate the info on Risk templates but I really am not using or concerned with risk templates I manage my own risk separately.

I only added the risk template because the I was just trying to get around the error message I sent you.

If I was causing the problem wouldn't it fail consistently?

Would deleting and redefining the database correct the problem if I then run the exact same script without an issue? I don't think so.

I'm not doing anything with the SIM account except responding to erroneous messages I'm getting .

My goal is simply tot Playback Mode and now Sim Mode to work which is not happening now. I wish it was something I was doing then I could fix it.

I ran Playback and SIM mode for Months with rarely if ever an error.

Can I reinstall the older release as this one is not for me so far.

thank you

glen

[NinjaTrader_Jim]

Hello Glen,

Your screenshots show you are connected to Playback and my response was directed in terms of the screenshots given.

So we are on the same page, we are talking about 2 issues here:

• Order can't be submitted: No risk settings defined for this instrument.
• Account below minimum cash value

The matter between Sim and Playback will be the same here, only that Playback inherits what we give Sim, and the Playback account gets refreshed whenever connect or rewind.

If you are seeing the No risk setting defined for instrument error, you need to check the Sim account properties and check the template that is applied to the sim account. The error means that the risk template does not have an instrument applied to the risk template, so that instrument would either need to be added to the risk template you are using, or you would need to set the risk template to None.

If you are seeing the Account below minimum cash value, you would need to check the Sim account properties again, and compare the Minimum Cash Value seen in the Sim account properties window to the Cash Value column.

I can't illustrate a platform side issue with the information provided. I just see errors when you were using Playback and that the Cash Value message is not in view. To me, it just appears that the account ran out of funds, which can be reset with the reset button, but I cannot confirm this since the Cash Value column is not in view.

Below is a video showing what should be checked and what we would need to see if you have a strategy that is reproducing these errors when the Sim account has enough funds and has a risk template with the instrument defined.

Demo - https://drive.google.com/file/d/1Gki...w?usp=drivesdk

After comparing Cash Value in the Accounts tab to the Minimum Cash Value set in the Account setting window, and you see the Cash Value column is greater than your Minimum Cash Value setting, please report back here with the screenshots requested. Please provide a screenshot of the account settings and show that you do have the instrument configured in your Risk template. This would establish if we are looking at an account setting issue or something else.

If you find your custom strategy is doing something to show this behavior, when these values would not allow the error, please reset your database file, and test to reproduce the issue, trying to get your strategy to hit the issue faster by taking larger trades, or by decreasing the Cash Value of the Sim account. From there, you would have a scenario that can reproduce the issue. That testing scenario can be used when reducing the strategy (by commenting and uncommenting functionality) to get down to what is causing the issue.

If it is not clear what is causing the issue is once you have reduced the code to what exactly is causing these symptoms, please implement that specific functionality in an example script and attach it here for our review. The script will need to compile and reproduce the issue with some steps for us to provide any comment.

If the strategy uses 3rd party indicators that you cannot distribute in an export, please modify your strategy logic so it does not use these indicators when reducing.

Exporting as source code - https://ninjatrader.com/support/help...tAsSourceFiles

Renaming the database file to clear Sim accounts and historical trade data.

• Shutdown NinjaTrader and Go to the Start Menu
• Select My Documents--> NinjaTrader 8--> db--> "NinjaTrader.sdf" file (may only say 'NinjaTrader' depending on your Windows File Explorer settings)
• Right click on NinjaTrader.sdf and select "Rename" *Name it "OLDNinjaTrader.sdf"
• Then restart the software and NinjaTrader will create a fresh database file to use
• Unfortunately the following items stored in the old database will be lost - Trading Hours templates, Instrument Lists / Custom Instruments, historical trade execution data, saved Strategy Analyzer backtest results

We look forward to assisting.

[demarcog]

Jim,

Try and understand my situation when this is occurring. I'm trading multiple positions on multiple platforms in real time and was also trying to get the documentation you need to work on the the problem.

The point I was trying to make with those messages attachments wasn't the context (Playback) of when they occurred but the content was identical to what I was then seeing on the screen. I just used the images I captured last night to send you. In hindsight I apologize it probably confused the you know what out of you.

I'm looking for the least restrictive parameters in the SIM account. I had the minimum cash value set to 0 and Risk to none when those errors stated occurring. I just want the strategies to finish running are they the correct setting to do that?


Big picture you showed me how to resolve the problem and that's all I care about right now. Thank you very much for the video it was very informative.


I really appreciate you taking the time and effort to help me understand and get past these issues.



So thanks again. When it happens again if it happens. I will take get you all the screen shots you need but let's hope that doesn't happen.



Have a great weekend.

glen

[demarcog]

Hi Jim,

I ran it again, see attached and it looks like when it's closing at EOD it also zeroes out the accounts. I will Look into disabling that features.

I real life trading I want everything closed at at 16:40 EST to avoid overnight margin. I though I was implementing that in the strategies options setups looks like it went too far.

thanks glen

[demarcog]

Hi Jim,


I'm running a new Playback right now and you can see each component on the screen.

In the Strategies Tab see all the strategies that are running and the realized and or unrealized for each symbol.

You can see the chart for NQ which is Long 1 contract up $4095.50 for account Playback.

Shouldn't this be reflected somewhere in the strategies tab?

Any Idea what I can do to get an accurate Playback,I really need to know if these strategies are profitable of not?

Thank you,

glen