Secondary series history missing history when running in real time

[michelm]

Hi,

Had a client running live today using an indicator I wrote.

The client hit f5 and noticed the indicator values were changed to incorrect values after the chart updated.

Took a while but I tracked it down and wrote a simple indicator that reproduces the issue.

1. No history. The client is running realtime, and no history was saved for the current day nor the previous day.
2. Primary series is 5-minute candles, secondary series is one-minute candles.

I'm using the playback connection with a start date of 02/05/2020 and an end date of 02/06/2020. ES 03-20 contract. Once again, no history, the only data I have for the two days are the downloaded replay files.

Set up the chart with a data series end date of 02/06/20 and days to load set to 2. Chart starts off completely empty since there is no history to backfill from.

Start the replay, I use max playback speed. Pause the playback around ~9:40 on 02/06/2020 Hit f5. At this point my indicator plotted at incorrect price levels.

On the refresh, NT only provides the internal minute bars starting at 02/06/2020 12:01:am. So the minute bars from 02/05/2020 are not sent to the indicator when refreshing with f5. The missing one-minute bars from 02/05/2020 cause my indicator to calculate incorrect values.

So here is an example of logging when running the playback. Getting the secondary series one-minute bars for both 02/05/2020 and 02/06/2020 as expected:

Raw obu: 2/5/2020 11:55:00 PM
Prim bars called: 2/5/2020 11:55:00 PM Current Bar: 271

Raw obu: 2/5/2020 11:55:00 PM
2nd series (min) bars called: 2/5/2020 11:55:00 PM Current Bar: 1359 Prim CB: 271

Raw obu: 2/5/2020 11:56:00 PM
2nd series (min) bars called: 2/5/2020 11:56:00 PM Current Bar: 1360 Prim CB: 271

Raw obu: 2/5/2020 11:57:00 PM
2nd series (min) bars called: 2/5/2020 11:57:00 PM Current Bar: 1361 Prim CB: 271

Raw obu: 2/5/2020 11:58:00 PM
2nd series (min) bars called: 2/5/2020 11:58:00 PM Current Bar: 1362 Prim CB: 271

Raw obu: 2/5/2020 11:59:00 PM
2nd series (min) bars called: 2/5/2020 11:59:00 PM Current Bar: 1363 Prim CB: 271

Raw obu: 2/6/2020 12:00:00 AM
Prim bars called: 2/6/2020 12:00:00 AM Current Bar: 272

Raw obu: 2/6/2020 12:00:00 AM
2nd series (min) bars called: 2/6/2020 12:00:00 AM Current Bar: 1364 Prim CB: 272

Raw obu: 2/6/2020 12:01:00 AM
2nd series (min) bars called: 2/6/2020 12:01:00 AM Current Bar: 1365 Prim CB: 272

Raw obu: 2/6/2020 12:02:00 AM
2nd series (min) bars called: 2/6/2020 12:02:00 AM Current Bar: 1366 Prim CB: 272



This is the log after pausing the replay and pressing f5, note I don't receive any of the secondary bars for 02/05/2020. It's not until 02/06/2020 that the secondary one-minute bars start coming in. Seems unexpected. My indicator uses the prior day's data as part of it's calculation, and the missing data is throwing it off.

Raw obu: 2/5/2020 11:50:00 PM
Prim bars called: 2/5/2020 11:50:00 PM Current Bar: 270

Raw obu: 2/5/2020 11:55:00 PM
Prim bars called: 2/5/2020 11:55:00 PM Current Bar: 271

Raw obu: 2/6/2020 12:00:00 AM
Prim bars called: 2/6/2020 12:00:00 AM Current Bar: 272

Raw obu: 2/6/2020 12:01:00 AM
2nd series (min) bars called: 2/6/2020 12:01:00 AM Current Bar: 0 Prim CB: 272

Raw obu: 2/6/2020 12:02:00 AM
2nd series (min) bars called: 2/6/2020 12:02:00 AM Current Bar: 1 Prim CB: 272

Raw obu: 2/6/2020 12:03:00 AM
2nd series (min) bars called: 2/6/2020 12:03:00 AM Current Bar: 2 Prim CB: 272


If there is history available for 02/05/2020 and 02/06/2020 the issue goes away.

Here is a very simple indicator that adds the secondary data series and does the logging:

//This namespace holds Indicators in this folder and is required. Do not change it.
namespace NinjaTrader.NinjaScript.Indicators
{
public class RtNoHistoryTest : Indicator
{
protected override void OnStateChange()
{
if (State == State.SetDefaults)
{
Description = @"Simple indicator to reproduce refreshing a multitimeframe chart that does not have any backing historu";
Name = "RtNoHistoryTest";
Calculate = Calculate.OnBarClose;
IsOverlay = true;
DisplayInDataBox = true;
DrawOnPricePanel = true;
DrawHorizontalGridLines = true;
DrawVerticalGridLines = true;
PaintPriceMarkers = true;
ScaleJustification = NinjaTrader.Gui.Chart.ScaleJustification.Right;
//Disable this property if your indicator requires custom values that cumulate with each new market data event.
//See Help Guide for additional information.
IsSuspendedWhileInactive = false;
}
else if (State == State.Configure)
{
// Force calc back to onbarclose if user tries to change
Calculate = Calculate.OnBarClose;

// Add one minute data series
AddDataSeries(BarsPeriodType.Minute, 1);
}
}

protected override void OnBarUpdate()
{
Print("Raw obu: " + Time[0]);
if (BarsInProgress == 1)
{
Print("2nd series (min) bars called: " + Time[0] + " Current Bar: " + CurrentBars[1] + " Prim CB: " + CurrentBars[0]);
}

if (BarsInProgress == 0)
{
Print("Prim bars called: " + Time[0] + " Current Bar: " + CurrentBars[0]);
}
Print("");
}

}
}

[NinjaTrader_Jesse]

Hello michelm,

I just want to write a quick post to let you know I am looking into this, once I have more details I will post back here.

[michelm]

Thank you Jesse.

I hope my post was clear, not sure it was....

I'll try to sum it up:

When refreshing a single instrument(with no backing history) multi-session multi-timeframe realtime chart using f5, secondary data series are only provided for the latest chart session, secondary data updates(OnBarClose) are not provided for previous sessions that are present on the chart. Primary bars work fine, I get OnBarUpdars for all primary bars across all chart sessions.

Idle speculation: I imagine that the primary bars are held in some kind of cache, but the secondary higher resolution bars are not....

[NinjaTrader_Jesse]

Hello michelm,

I just wanted to provide a follow up after testing this situation.

Currently I have been unable to see a difference in reporting after pressing F5 In my tests. I used a DIFF tool to compare the output and they matched in each test.

The steps I used for testing were as follows, can you review and make sure I am not missing a step that you are doing?

1. Start with platform closed
2. Cleared all existing data/cache
3. Open platform
4. Downloaded 02/05/2020 and 02/06/2020 market reply
5. Connect to playback
6. Created a chart with the end date of 2/6 and 2 days lookback
7. Chart appeared with no historical data
8. Applied the indicator
9. Played forward at max speed until 2/6 after 9:40 am
10. Paused
11. Copied output and cleared window
12. F5
13. Copied output and made comparison.

From what I can see the output is matching in this test, if you can provide any other insight from your end that would be helpful. I could also suggest clearing your data/cache to see if that helps or makes a difference in the results.

I look forward to being of further assistance.

[michelm]

Hi,

Can you confirm that you don't have any historical data saved (Tools: Historical data) on the 5th and 6th for the instrument that you are running the test with?

[NinjaTrader_Jesse]

Hello michelm,

Correct, I have also turned off the option to request historical data from the server/ don't have a data connection set up. I am essentially using a new/clean install for the test without anything downloaded.

I have been unable to reproduce the output you had shown using those guidelines.

What version are you and the client using?


I look forward to being of further assistance.

[michelm]

my version: 8.0.19.1

[NinjaTrader_Jesse]

Hello michelm,

You may want to try updating to the current release and clear the old data/cache to see if that still happens. I see there were some changes to playback in the releases after 19.1 so it is possible that will help.

I look forward to being of further assistance.

[michelm]

OK, I'm not in a position to try it right now. I'll get back to ASAP and post the results here. Thank you for your help with this, I do appreciate it.

[michelm]

Hi,

I updated to the latest version(8.0.20) and deleted all the files in the db cache directory. I still get the issue, when I pause the playback sometime during the second day(6th) and refresh the chart with f5, my OnBarUpdate does not receive and of the secondary 1 min bars from the 5th. The secondary bars don't start coming until the 6th at 12:01

Raw obu: 2/5/2020 11:50:00 PM3350.5
hi: 3350.5 low: 3289.25
Prim bars called: 2/5/2020 11:50:00 PM Current Bar: 270

Raw obu: 2/5/2020 11:55:00 PM3352
hi: 3352 low: 3289.25
Prim bars called: 2/5/2020 11:55:00 PM Current Bar: 271

Raw obu: 2/6/2020 12:00:00 AM3352.75
hi: 3352.75 low: 3289.25
Prim bars called: 2/6/2020 12:00:00 AM Current Bar: 272

Raw obu: 2/6/2020 12:01:00 AM3352.25
hi: 3352.75 low: 3289.25
2nd series (min) bars called: 2/6/2020 12:01:00 AM Current Bar: 0 Prim CB: 272

Raw obu: 2/6/2020 12:02:00 AM3353
hi: 3353 low: 3289.25
2nd series (min) bars called: 2/6/2020 12:02:00 AM Current Bar: 1 Prim CB: 272

[NinjaTrader_Jesse]

Hello michelm,

Thank you for updating.

Can you confirm the steps I posted were the steps that you are following?

Are you using any addons during playback or do you have a lot of third party items imported? It may be helpful to do a clean install test to make sure that works. That could help to avoid a longer troubleshooting process in case something external is a factor. This type of test is essentially just renaming your current user folder so that it is not being used, re install NinjaTrader and then test. I will post the full steps below.

Also just to confirm are you using the same script that was posted here? I understand this seems like an odd question however if you have this print inside a different script with other logic than what is in the sample that may be part of the problem.

(edited)
To do a clean user folder test:

• Go to Help > License Key and copy and paste your License Key into a text document
• Exit the platform
• Go to (My) Documents > rename the 'NinjaTrader 8' folder to 'NinjaTrader 8.Prev'.
• Then re-install from the following link: http://ninjatrader.com/GetStarted
• Test using the new clean user folder

To revert after the test

1. Exit the platform
2. Go to (My) Documents > rename the 'NinjaTrader 8' folder to 'NinjaTrader 8.Testing'.
3. Go to (My) Documents > rename the 'NinjaTrader 8.Prev' folder to 'NinjaTrader 8'.

I look forward to being of further assistance.

[michelm]

Hi,

I will run the clean test.

This is an NT 8 indicator, I don't think you running it under NT7, but your clean instructions ref NT7, just wanted to mention that

[NinjaTrader_ChelseaB]

Hello michelm,

I think was just an overlook.
The instructions for NinjaTrader 8 are basically the same. I've updated Jesse's post to reflect this.