We need a real fix for corrupt chart data

[jcrza]

https://i.gyazo.com/8383457fc3482b99b8cb600d90d4a80d.png

This just popped up for me. On a typical day I notice odd gaps in my charts and have to reload data with "Reload all historical data" or do what that pictured popup says.

If you can detect this and give me a popup why can't you just automatically fix the data by loading it from the server?

It should be impossible for me to load a chart with bad data on it, period. When I load a chart of /ES data to view it NT8 should checksum the chart data and make sure it's exactly what your server has.

This isn't acceptable. I'm trading charts that don't reflect reality. The gaps I see aren't real, I have to reload every instrument on every data-type to make sure I have the correct data every morning. It's a major flaw in NT8 and I'd be astounded if at least one of your customers hasn't annihilated their account with a bad trade because their chart showed them something that wasn't real.

[NinjaTrader_EricB]

Hello jcrza

Thank you for your post,

I will certainly forward your note to our Development team for future feature consideration.

Let us know if we may assist further.

[NinjaTrader_EricB]

Thank you for your post,

So we may look into what is occurring further, please write into to PlatformSupport[AT]NinjaTrader[DOT]Com along with your log and trace files and a link to this forum post.

Please follow the steps below to manually attach your log and trace files to your response so I may investigate this matter further.

* Open your NinjaTrader folder under Documents.
* Right click on the 'log' and 'trace' folders and select Send To> Compressed (zipped) Folder.
* Send the 2 compressed folders as attachments to this email.
* Once complete, you can delete these compressed folders.

[RobotSyndicate]

I second this. Corrupted data in the charts and in strategy analyzer is a constant battle. I literally have a sticky note on my monitor that reminds me to not trust ninjatrader and to triple check and reload the chart data every single day.

[afcmaff]

This is still an issue in 2022 (build 8.0.25.0). After loading ES historical data, I purchased from Tick Data LLC, which includes the raw unfiltered ticks, I realized (after verifying several times) that the Tick Data file has accurate historical pricing, whereas the data I originally had was nearly 100 basis points off. When running extensive back tests, there are still issues (seemingly phantom data, or a database corruption issue) whereby the strategy's trade exit of type "Exit on Session Close" is in a completely different price range than the back test chart data - very strange behavior that makes it extremely difficult to get clean back tests. I believe this has something to do with how the db sorts the historical data... I use ES ##-## for loading and testing certified/verified historical data as a continuous auto-rolled contract, whereby my live charts are obviously using DTN (Kinetick) loaded from the server with the current front month.

NOTE: the Tick Data LLC historical data was properly formatted using the NT provided "NinjaTrader.twj" file with TickWrite and named "ES ##-##.Last.txt" as it is last-trade-based tick data.

The picture example is from the Strategy Analyzer showing a scaled-in trade placed on May 2, 2006 -- the Exit is at the verified historical ES close price of 1318.50, yet the chart is plotting the price data incorrectly somehow and displaying bars in the 1160 region (that's quite a way off).

NinjaTrader_EricB Are there any known issues with the db for loading / back-testing historical data? The data in the Historical Data files for ES ##-## is correct, but the Strategy Analyzer keeps charting and painting bars incorrectly.

[NinjaTrader_Kate]

Hello afcmaff,

Thank you for your reply.

First, I would suggest clearing the cache and then reloading the chart as there may be a bar caching issue occurring if you previously had other tick data loading.

To delete the cache, follow the steps below.

• Shut down NinjaTrader.
• Open the Documents > NinjaTrader 8 > db folder.
• Delete the sub-folder named 'cache'.
• Restart NinjaTrader and test.

Do you see the same in the Strategy Analyzer after deleting the cache and re-running a backtest?

Thanks in advance; I look forward to assisting you further.

[afcmaff]

As a follow-up to my previous post -- I did notice the Historical Data Files for ES ##-## were accurate for the 'Tick' data, but the 'Minute' data showed the incorrect prices. So, the issue can be more narrowly focused on NT's conversion algorithms for generating minute/day data from tick data during import with the option to "Generate 'Minute' bars from imported data" and "Generate 'Day' bars from imported tick or minute data"

[afcmaff]

Thanks NinjaTrader_Kate …I discovered the issue was likely the dataset format (reimporting now, so if all is good, I won’t need to repost anything). I noticed that I forgot to add the Adjustment method to the continuous contract (adjustment was set to ‘None’) so all the imported data was simply spliced together which is no bueno for back-testing. My only suggestion would be to update the TickDataLLC.zip “NinjaTrader.twj” job file to auto set the adjustment to either ‘Ratio, Backward’ or ‘Difference, Backward’ instead of ‘None’ for continuous contract (it does select Auto Roll method which is good), but this is extremely low priority since it can be easily toggled in TickWrite7.1 — with properly adjusted price data, back testing should reflect actual price development with historical accuracy.

Cheers

[afcmaff]

As a final follow-up to my previous posts - there were still discrepancies between the tick-by-tick trade data (continuous file import) and the minute bars that NT8 created from that data within the ES ##-## instrument placeholder... I think perhaps there's a reason the NinjaTrader.twj job file for TickWrite does not have an adjustment method selected as it may need the original tick prices for its own calculations. There is some mechanism NT8 seems to have that auto-adjusts the imported data (this may likely be contributed to the platform setting in "Tools > Options > Market Data > Merge Back Adjusted" which might be applying the merge calculation internally and automatically on all data including imported data when forming bars, etc.)

I found that by individually (and painfully) importing the raw unadjusted contract-by-contract tick data, each into their respective contract objects (i.e. "ES 03-05.Last.txt", "ES 06-05.Last.txt", ... ) vs. a continuous file within the "ES ##-##" object; the prices for both NT8-generated minute bars and the original tick-by-tick trade data was correctly aligned. I noticed NT8 did not have instrument objects for ES older than 03-05 (I realize I could make these, but it wasn't necessary).

Bottomline - the only way I found to get the most accurate and successful import was to not use a continuous file and instead individually import each contract (unadjusted raw trade data) into their respective pre-defined instrument objects, which NT8 was able to successfully generate correct minute-bars from. Hope this helps others when importing large datasets for back testing.