Order flow data missing for September

[Skifree]

I'm backtesting something that uses order flow data everything works up to Sept 1 but after that, for no silver micro is there any OF data. I've tried all contracts. Why?

[NinjaTrader_Clayton]

Hello Skifree,

Thank you for the message.

Is the Merge Policy of your platform set to "Do not merge"? If yes, this will only download available historical data for the specific contract that was entered.

• To change, go to the Control Center > Tools > Options > Market Data Category > Global merge policy > Switch to either Merge back adjusted or Merge non back adjusted.
• This will make it so it automatically downloads the different contracts and data for each span where they were active.

I just tested and was able to download Historical Tick Data for the SIL back 1 year to September 10th, 2023.

• 

If you try the download settings as shown in my picture and it still isn't downloading more data:

• Go through the "Historical Data" list that is there and remove all of the currently downloaded SIL data.
• Then, use the following steps to Update/Reset the Instruments within your platform and then Restart NinjaTrader:
  • Disconnect from any data providers by navigating to the Control Center > Connections menu, and left-click on “disconnect” that appears to the right when hovering your mouse over any active connection
  • Reset your instruments by navigating to the Control Center > Tools > Database Management
  • In the Database Management window, under the “Update instruments” section, ensure that “General properties”, "Futures expiries" and “Symbol mappings” options are checked and then click Update
  • Restart NinjaTrader
  • Try downloading the Historical Tick Data again.

Please let us know if we may provide any further assistance.

[Skifree]

I'm using MergeBackAdjusted but the problem seems to be less about the data now, and just it doesn't load sometimes - and the only correlation i can find is date range of the backtest. Some ranges work fine and others just stop.

I'm still having some issues here - and it seems to be related to the timeframe of data requested. I notice that backtesting the volumetric data does this for all instruments - this is data under SIL DEC24. I'm paying for the OrderFlow features but they are completely hamstrung if the data doesnt load. As you can see in the screenshots, the data works and then it doesn't, despite all the price and volume data being available.

01-2024 to 09-13 2024, the data shows up fine
08-2023 to 09-13 2024, only the volumetric data is cut off starting Aug 20 21:25
08-2023 to 03-01 2024, only the volumetric data is cut off starting Feb 5 20 20:05 (You can see in the pic, the backbrush orange is where no data is coming in



My debug shows that the volumetric does indeed stop loading

SIL DEC24 Date: 2024-02-05 BarsTypeLength: 18302 CURRENTBAR 18303
SIL DEC24 Date: 2024-02-05 BarsTypeLength: 18302 CURRENTBAR 18304
SIL DEC24 Date: 2024-02-05 BarsTypeLength: 18302 CURRENTBAR 18305
SIL DEC24 Date: 2024-02-05 BarsTypeLength: 18302 CURRENTBAR 18306
SIL DEC24 Date: 2024-02-05 BarsTypeLength: 18302 CURRENTBAR 18307
SIL DEC24 Date: 2024-02-05 BarsTypeLength: 18302 CURRENTBAR 18308
SIL DEC24 Date: 2024-02-05 BarsTypeLength: 18302 CURRENTBAR 18309
SIL DEC24 Date: 2024-02-05 BarsTypeLength: 18302 CURRENTBAR 18310
SIL DEC24 Date: 2024-02-05 BarsTypeLength: 18302 CURRENTBAR 18311
SIL DEC24 Date: 2024-02-05 BarsTypeLength: 18302 CURRENTBAR 18312
SIL DEC24 Date: 2024-02-05 BarsTypeLength: 18302 CURRENTBAR 18313


Code:
BarsType is defined for a protected variable

if(State == State.DataLoaded)
{
// Assign the volumetric bar type here
barsType = BarsArray[2].BarsType as NinjaTrader.NinjaScript.BarsTypes.VolumetricBarsTy pe;

if (barsType == null)
{
Print("Error: BarsArray[2] is not a Volumetric bar type");
return;
}
}

barsType is not null, but it's length has stopped growing??

public OrderFlowPattern OrderFlowPatternCreation()
{



lastFunction = "OrderFlowPatternCreation";
string msg1 = " 0";
bool isMatched = false;
OrderFlowPattern nullPattern = new OrderFlowPattern
{
predictionScore = 0,
OFPrediction = signalReturnActionType.Neutral,
};
if (barsType == null)
{
Print("Error: BarsArray[2] is not a Volumetric bar type");
return nullPattern;
}
// Ensure c is within valid bounds
if (CurrentBars[0] > barsType.Volumes.Length)
{
BackBrush = Brushes.Cyan;
return nullPattern;
}

[Skifree]

I've traced this issue just now to AddVolumetric() which is loading partial data:
Both of these are expected to load the same quantity of bars as primary series 10 min SILDEC24

AddVolumetric(Instrument.FullName,BarsPeriodType.M inute,10,VolumetricDeltaType.BidAsk,1);
AddVolumetric("SIL DEC24",BarsPeriodType.M inute,10,VolumetricDeltaType.BidAsk,1);

SIL DEC24 Date: 2024-03-28 BarsType 20922 CURRENTBARS[0] 25683 AddVolumetric 20922

[Skifree]

extensive debugging points to what feels like a massive rugpull.

After building what feels like a viable strategy based on NT8 Volumetric Data and Order Flow data, when I launch campaigns live the realtime Volumetric data is not up to date with the primary series. It is ALWAYS delayed, by many bars.

Even in just loading of the data, before anything runs:

Index 0 = SIL 10Min
Index 1 = Tick 50
Index 2 = Volumetric 10 Min
barsType is based on Index 2

BEGIN DataLoaded CurrentBars[0] -1 BarsArray[0].Count 8965 barsType length 8957 BarsArray[2].Count 8957

it seems that this issue exists across Rithmic and NT data feeds, and I cant seem to fix this such that the Volumetric/OF data is loading in realtime along side the primary series? I am not using any charts, I am only deploying strategies.

[Skifree]

Progress: For anyone else encountering this issue - it seems to stem from time series bars not aligning with OrderFlow bars, because they are based on Volume and not time, therefor they have no given time alignment. Volumetric[1234] wont necessarily align with CurrentBars[1234], and OrderFlow bars have no time component.

it seems the right way to do this requires some extra pains to ensure your timeseries time matches your volumetric time, and if so then by assoc your OF data will match your timeseries.

@Support = lots of extra coding could be removed if barsType volume also had some time attribute capability inherited by the volumetric bars! barsType.Volumes[index].Time[0] should work!