RC2 bug Historical Data load (minutes)

[DaveE]

No one responded to this problem I reported last Monday

Now its been moved to archive (locked).

If no one from NT wants to respond, could some other user post here if they find a similar problem using other connections than IB?

I think there is a problem with how minute data is currently accessed in these ncd files (presumably Ninja Cryptic? Database"). Before loading new information from the data provider (here IB) NT seems to do a block delete of existing information in the area it thinks is going to be loaded, then does the load.
However this whole idea of doing block deletes is asking for trouble. In the problem reported last Monday NT has made a wrong decision about the end DateTime of the block to be deleted, and deletes an extra date, then the load comes along and only inserts the requested date (resulting in a large data gap).

On this page it says

Note: If you already have historical data for an instrument, please be sure to only select a date range in which your data provider offers historical data. If you choose a range older than what your data provider offers you may lose any data you had stored on those dates in that range outside of what your data provider offers

This implies that NT is doing a block delete for the requested dates, and only later seeing if it can get all the data from the provider. That is completely unacceptable.
Why on earth dont you use a reliable database (such as SQL Server) for storing the Minute and Day prices? Then when a "download" is required you would never have to use DELETE you only need to do something like this:
For each 'bar' received from provider:
If DateTime exists UPDATE... else INSERT...
No DELETE is required because there are a fixed set of minute DateTimes available on any date, and a fixed set of 'Date only' values for any year.
Obviously situation is different for Tick data, so you may want to keep ncd for that.
I am just getting frustrated by NT not being able to provide simple Minute data without operations introducing data gaps. I think your proprietary 'ncd' system is not currently fit for purpose for Minute data.

[NinjaTrader_ChelseaB]

Hello DaveE,

I'm understanding that you receiving gaps on your chart.

To confirm, you have right-clicked the chart and selected 'Reload All Historical Data' and this has not corrected the chart, is this correct?

Are you currently recording data for historical or market replay?
(Tools -> Options... -> Real-time -> 'Enable market recording for playback' and 'Record live data as historical)

If not, please send an email to platformsupport [at] ninjatrader [dot] com for further assistance.

[DaveE]

Hello Chelsea,

I don't think you followed the link to the original post (click here)
There I provided clear steps for reproducing the problem using "Historical Data" form.
I have since verified that the status of 'Record live data as historical' has no effect on this problem.
The problem gets even worse using 'Reload All Historical Data' from a chart. I can produce steps to reproduce that problem also, but I want someone to try the steps for "Historical Data" form first (since its the simplest set of steps).

[NinjaTrader_PatrickG]

I see that you've posted a few times with similar information on historical data behavior. We appreciate the feedback. The beta forums were archived but we can still view your other inquiries.

For the time being, the behavior you've reported would be expected due to NinjaTrader's current logic. We are tracking requests to make improvements to this logic, however, there are no changes planned at this time.

I will be adding your information to the existing feature request: SFT-1371

[DaveE]

NT8.0.1 Problem 2 Chart “reload all historical data”

Since the original post I have upgraded to NT8.0.1, but the data integrity problems are still there.

Problem 2: Chart “reload all historical data” causes all future minute data to be deleted.
Steps to reproduce the problem:
1) Pick an instrument where you don’t mind data getting messed up (I picked USDMXN).
2) Ensure you have a full set of Ask Minute data for USDMXN in Historical Data (eg use “Historical Data” form “Download” facility to get USDMXN Minute Ask, start date = 21 Oct 2016 end date = 15 Nov 2016.
Verify you have the full set of rows (use Edit tab). Close “Historical Data” form (this is necessary because information displayed in it is not refreshed if changes take place outside this form).
3) Now open a new Chart USDMXN Ask, Minute, 1. Time Frame: Days, 1, end date=24 Oct 2016. Note that we already have info in Historical Data for this period.
4) Use “reload all historical data”.
5) Open Historical Data form to check Historical/USDMXN/Ask/Minute/…

Expected result:
NT should repopulate Historical data (USDMXN Ask Minute) just for the period covered in the chart (single session ending 24 Oct 2016), and refresh the chart.

Actual result:
NT does repopulate Historical Data (USDMXN Ask Minute) for the period covered in the chart but it also (serious bug) deletes all Historical Data (USDMXN Ask Minute) for the time after the chart, (except for a single minute ending 0:01 for 25 Oct 2016! It does leave alone Historical Data earlier than the period covered by the chart.
What I suspect NT currently does is this:
a) Block delete all Historical Data (USDMXN Ask Minute) from the start of the period covered by the chart (to the end of the latest data). This is madness!
b) Repopulate Historical Data (USDMXN Ask Minute) just for the period covered by the chart, plus bizarrely the first minute of the next date.
I see IB Gateway log has this:
src:Ask e:20161028-23:00:00 len:6 d step:1 min
which indicates that NT has requested 6 days of minute data ending 23:00 on 28 Oct 2016. But this goes further ahead than the period in the chart, and the information that is further ahead gets discarded (except for one ridiculous minute).

The status of ‘Record Live as Historical’ option has no effect on this problem.

It almost looks as though steps a) and b) were written by different developers who never talked to each other. I see no sign of any QA or Testing being done on this.

I have now demonstrated two ways (Historical Data form & Chart “reload all historical data”) where standard NT operations that are supposed to load minute data actually delete some or all of the customer's future data. NT needs to recognize that this is not going to be considered "expected behaviour" by any reasonable person. It does not obey any reasonable "logic" these are serious bugs and they must be fixed.

[NinjaTrader_PatrickG]

This 'reload all historical data' behavior is currently expected. I will add your feedback to the feature request to change this behavior.

[tyrin_snow]

NT8.0.1 Problem 2 Chart “reload all historical data”

Since the original post I have upgraded to NT8.0.1, but the data integrity problems are still there.

Problem 2: Chart “reload all historical data” causes all future minute data to be deleted.
Steps to reproduce the problem:
1) Pick an instrument where you don’t mind data getting messed up (I picked USDMXN).
2) Ensure you have a full set of Ask Minute data for USDMXN in Historical Data (eg use “Historical Data” form “Download” facility to get USDMXN Minute Ask, start date = 21 Oct 2016 end date = 15 Nov 2016.
Verify you have the full set of rows (use Edit tab). Close “Historical Data” form (this is necessary because information displayed in it is not refreshed if changes take place outside this form).
3) Now open a new Chart USDMXN Ask, Minute, 1. Time Frame: Days, 1, end date=24 Oct 2016. Note that we already have info in Historical Data for this period.
4) Use “reload all historical data”.
5) Open Historical Data form to check Historical/USDMXN/Ask/Minute/…

Expected result:
NT should repopulate Historical data (USDMXN Ask Minute) just for the period covered in the chart (single session ending 24 Oct 2016), and refresh the chart.

Actual result:
NT does repopulate Historical Data (USDMXN Ask Minute) for the period covered in the chart but it also (serious bug) deletes all Historical Data (USDMXN Ask Minute) for the time after the chart, (except for a single minute ending 0:01 for 25 Oct 2016! It does leave alone Historical Data earlier than the period covered by the chart.
What I suspect NT currently does is this:
a) Block delete all Historical Data (USDMXN Ask Minute) from the start of the period covered by the chart (to the end of the latest data). This is madness!
b) Repopulate Historical Data (USDMXN Ask Minute) just for the period covered by the chart, plus bizarrely the first minute of the next date.
I see IB Gateway log has this:
src:Ask e:20161028-23:00:00 len:6 d step:1 min
which indicates that NT has requested 6 days of minute data ending 23:00 on 28 Oct 2016. But this goes further ahead than the period in the chart, and the information that is further ahead gets discarded (except for one ridiculous minute).

The status of ‘Record Live as Historical’ option has no effect on this problem.

It almost looks as though steps a) and b) were written by different developers who never talked to each other. I see no sign of any QA or Testing being done on this.

omegle tv I have now demonstrated two ways (Historical Data form & Chart “reload all historical data”) where standard NT operations that are supposed to load chat roulette minute data actually delete some or all of the customer's future data. NT needs to recognize that this is not going to be considered "expected behaviour" by any reasonable person. It does not obey any reasonable "logic" these are serious bugs and they must be fixed.

This 'reload all historical data' behavior is currently expected.

[NinjaTrader_Jason]

Hello tyrin_snow,

I'm not sure how to interpret your post. Could you please clarify?

Please note that this forum thread is almost 5 years old and handles version 8.0.1.0. We are currently on version 8.0.24.3. Please see the release notes at the link below.

https://ninjatrader.com/support/helpGuides/nt8/NT%20HelpGuide%20English.html?release_notes.htm