Help Understanding Bar Offset in Realtime & Historical

[ChainsawDR]

Hi,
As the title indicates, I'm hoping to ask for some guidance when it comes to how Ninjatrader transitions from Historical to Realtime data. Here's the situation I'm struggling with... I set off my strategy which uses 10 sec bars and run it in demo mode (it pulls some historical data then when up to date starts processing realtime data). For each bar I print various datapoints (including CurrentBar) to a file. I then shut off the strategy and immediate run Strategy Analyzer over the same time period, and when I look at the prints for the Strategy Analyzer and Live-Demo, I see different Bar counts for each time. For example:

In my Strategy Analyzer run: Time: 7/25/2022 3:04:30 PM, Ticker: AEMD, State: Historical, boughtToday: True, cutOff: False, regMarketClose: 1.06, CurrentBars[0]: 922, _lastOrderBar: 232
In my Live-Demo run: Time: 7/25/2022 3:04:30 PM, Ticker: AEMD, State: Realtime, boughtToday: True, cutOff: False, regMarketClose: 1.06, CurrentBars[0]: 923, _lastOrderBar: 232

I remember that NinjaTrader_Jim used some offset logic within his BuySellVolumeOneTick example, however despite following the code annotations I'm really struggling to understand why the CurrentBar count is different between Backtests & Live (e.g. for 3:04:30 PM why is one showing 922 and another 923?). Would anyone be able to help me please with a dumbed down explanation of why CurrentBars differs between realtime and historical (or more literally, a strategy that pulls historical data then transitions to realtime, vs just historical backtests). My trades aren't firing in realtime and this CurrentBars discrepancy seems to be the root cause (other datapoints are different but it all seems to stem from the same CurrentBars issue).

Many thanks in advance

ChainsawDR

[NinjaTrader_ChrisL]

Hi, thanks for posting. The calculation in BuySellVolumeOneTick: int indexOffset = BarsArray[1].Count - 1 - CurrentBars[1]; This is used to determine if the script is in real time or historical mode. If the total amount of bars minus the current bar is > 0 then the current bar is less than the total amount of bars so it's still processing historical bars. Try printing the BarsArray[1].Count - 1 and BarsArray[0].Count - 1 values in both a real time chart and the strategy analyzer to ensure you are comparing the exact same data series. The real time chart will have an extra bar since it will have a real time bar that the strategy analyzer will not have.

Kind regards,
-ChrisL

[ChainsawDR]

Thanks Chris. I understand that realtime will have an extra bar because as time progresses its still formulating/building the next bar as time passes (e.g. on 10sec bars, for the first 9secs there will be an extra bar on live until it completes). The piece that I'm struggling with is the time aspect. When the next bar is being built on realtime, it is still building that bar for a set period of time. In the example above, for the time bar 7/25/2022 3:04:30 PM it is bar 922 on Historical and 923 on Realtime. I seem to have a mental block on understanding how the same period of time can have different bars. I understand that depending upon realtime/historical the bar would be at different stages of completeness, but an extra bar for the same time is one that is stumping me. Any thoughts please?

[ChainsawDR]

In case it helps, I went through the prints to determine when the extra bar came in:

REALTIME (which started off historical before transitioning to realtime):
CPE1 Time: 7/25/2022 1:59:20 PM, Ticker: AEMD, State: Historical, boughtToday: True, cutOff: False, regMarketClose: 1.06, CurrentBars[0]: 657, _lastOrderBar: 232
CPE1 Time: 7/25/2022 1:59:40 PM, Ticker: AEMD, State: Historical, boughtToday: True, cutOff: False, regMarketClose: 1.06, CurrentBars[0]: 658, _lastOrderBar: 232
CPE1 Time: 7/25/2022 2:00:50 PM, Ticker: AEMD, State: Realtime, boughtToday: True, cutOff: False, regMarketClose: 1.06, CurrentBars[0]: 659, _lastOrderBar: 232

HISTORICAL (after running realtime, went back and ran strategy analyzer):
CPE1 Time: 7/25/2022 1:59:20 PM, Ticker: AEMD, State: Historical, boughtToday: True, cutOff: False, regMarketClose: 1.06, CurrentBars[0]: 657, _lastOrderBar: 232
CPE1 Time: 7/25/2022 2:00:50 PM, Ticker: AEMD, State: Historical, boughtToday: True, cutOff: False, regMarketClose: 1.06, CurrentBars[0]: 658, _lastOrderBar: 232

You can see from above, in the realtime run at the point of switching over from Historical to Realtime state, it created a historical bar at 1:59:40 PM (which is I think the time of the day when I set off the strategy in realtime) which did not appear in the purely historical backtest. Is this expected (aka at the point of transition from historical to realtime, would it create an extra bar)?

[NinjaTrader_ChrisL]

Hi, thanks for your reply. There is something in your script that I'm missing. In a simple test I ran a strategy with a single line of code:

protected override void OnBarUpdate()
{
    Print(CurrentBar + " " + Time[0]);
}

Here is my result:

https://clauber.tinytake.com/msc/NzE1OTMzMV8xOTkzMjQ0Nw

Could you give me a reduced test script that reproduces and demonstrates the issue?

Kind regards.

[ChainsawDR]

Hi Chris,

The steps to recreate are slightly different. Here's a slightly updated version of your code (more prints):

#region Using declarations
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.ComponentModel.DataAnnotations;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Windows;
using System.Windows.Input;
using System.Windows.Media;
using System.Xml.Serialization;
using NinjaTrader.Cbi;
using NinjaTrader.Gui;
using NinjaTrader.Gui.Chart;
using NinjaTrader.Gui.SuperDom;
using NinjaTrader.Gui.Tools;
using NinjaTrader.Data;
using NinjaTrader.NinjaScript;
using NinjaTrader.Core.FloatingPoint;
using NinjaTrader.NinjaScript.Indicators;
using NinjaTrader.NinjaScript.DrawingTools;
#endregion

//This namespace holds Strategies in this folder and is required. Do not change it.
namespace NinjaTrader.NinjaScript.Strategies
{
public class teststrategy : Strategy
{
protected override void OnStateChange()
{
if (State == State.SetDefaults)
{
Description = @"testing times";
Name = "teststrategy";
Calculate = Calculate.OnEachTick;
EntriesPerDirection = 1;
EntryHandling = EntryHandling.AllEntries;
IsExitOnSessionCloseStrategy = true;
ExitOnSessionCloseSeconds = 30;
IsFillLimitOnTouch = false;
MaximumBarsLookBack = MaximumBarsLookBack.TwoHundredFiftySix;
OrderFillResolution = OrderFillResolution.Standard;
Slippage = 0;
StartBehavior = StartBehavior.WaitUntilFlat;
TimeInForce = TimeInForce.Gtc;
TraceOrders = false;
RealtimeErrorHandling = RealtimeErrorHandling.StopCancelClose;
StopTargetHandling = StopTargetHandling.PerEntryExecution;
BarsRequiredToTrade = 20;
// Disable this property for performance gains in Strategy Analyzer optimizations
// See the Help Guide for additional information
IsInstantiatedOnEachOptimizationIteration = true;
}
else if (State == State.Configure)
{

AddDataSeries(BarsPeriodType.Tick, 1); //added this and commented out the next one to access bidask.
}
}

protected override void OnBarUpdate()
{
//Add your custom strategy logic here.
Print(CurrentBar + " " + Time[0]);
Print(string.Format("Debug Time: {0}, Ticker: {1}, State: {2}, BarsInProgress: {3}, Close: {4}, Open: {5}, CurrentBars[0]: {6}, CurrentBars[1]: {7}, BarsArray[0].Count-1: {8}, BarsArray[1].Count-1: {9}", Time[0], Instrument.FullName, State, BarsInProgress, Close[0], Open[0], CurrentBars[0], CurrentBars[1], BarsArray[0].Count-1, BarsArray[1].Count-1));

}
}
}

If you first run this script for 7/25 to 7/26 (today) using stock VS and US extended hours using the strategy tab of the console (clicking enable etc - which effectively runs in part historical before switching to realtime) and copy the output you will see the following:

4619 7/26/2022 5:47:30 PM
Debug Time: 7/26/2022 5:47:30 PM, Ticker: VS, State: Historical, BarsInProgress: 0, Close: 0.52, Open: 0.52, CurrentBars[0]: 4619, CurrentBars[1]: 50157, BarsArray[0].Count-1: 4620, BarsArray[1].Count-1: 50158
50158 7/26/2022 5:49:13 PM
Debug Time: 7/26/2022 5:49:13 PM, Ticker: VS, State: Historical, BarsInProgress: 1, Close: 0.52, Open: 0.52, CurrentBars[0]: 4619, CurrentBars[1]: 50158, BarsArray[0].Count-1: 4620, BarsArray[1].Count-1: 50158
4620 7/26/2022 5:49:20 PM
Debug Time: 7/26/2022 5:49:20 PM, Ticker: VS, State: Historical, BarsInProgress: 0, Close: 0.52, Open: 0.52, CurrentBars[0]: 4620, CurrentBars[1]: 50158, BarsArray[0].Count-1: 4620, BarsArray[1].Count-1: 50158
Enabling NinjaScript strategy 'teststrategy/270219711' : On starting a real-time strategy - StartBehavior=WaitUntilFlat EntryHandling=All entries EntriesPerDirection=1 StopTargetHandling=Per entry execution ErrorHandling=Stop strategy, cancel orders, close positions ExitOnSessionClose=True / triggering 30 seconds before close SetOrderQuantityBy=Strategy ConnectionLossHandling=Recalculate DisconnectDelaySeconds=10 CancelEntriesOnStrategyDisable=False CancelExitsOnStrategyDisable=False Calculate=On each tick IsUnmanaged=False MaxRestarts=4 in 5 minutes
4621 7/26/2022 5:53:20 PM
Debug Time: 7/26/2022 5:53:20 PM, Ticker: VS, State: Realtime, BarsInProgress: 0, Close: 0.51, Open: 0.51, CurrentBars[0]: 4621, CurrentBars[1]: 50159, BarsArray[0].Count-1: 4621, BarsArray[1].Count-1: 50159
50159 7/26/2022 5:53:16 PM
Debug Time: 7/26/2022 5:53:16 PM, Ticker: VS, State: Realtime, BarsInProgress: 1, Close: 0.51, Open: 0.51, CurrentBars[0]: 4621, CurrentBars[1]: 50159, BarsArray[0].Count-1: 4621, BarsArray[1].Count-1: 50159
4622 7/26/2022 5:53:30 PM
Debug Time: 7/26/2022 5:53:30 PM, Ticker: VS, State: Realtime, BarsInProgress: 0, Close: 0.52, Open: 0.52, CurrentBars[0]: 4622, CurrentBars[1]: 50160, BarsArray[0].Count-1: 4622, BarsArray[1].Count-1: 50160
50160 7/26/2022 5:53:20 PM
Debug Time: 7/26/2022 5:53:20 PM, Ticker: VS, State: Realtime, BarsInProgress: 1, Close: 0.52, Open: 0.52, CurrentBars[0]: 4622, CurrentBars[1]: 50160, BarsArray[0].Count-1: 4622, BarsArray[1].Count-1: 50160

If you then stop the strategy script, and run the strategy using the Strategy Analyzer for the same 7/25 to 7/26 period using same settings, the output will look like this:
4619 7/26/2022 5:47:30 PM
Debug Time: 7/26/2022 5:47:30 PM, Ticker: VS, State: Historical, BarsInProgress: 0, Close: 0.52, Open: 0.52, CurrentBars[0]: 4619, CurrentBars[1]: 50157, BarsArray[0].Count-1: 4621, BarsArray[1].Count-1: 50159
50158 7/26/2022 5:53:16 PM
Debug Time: 7/26/2022 5:53:16 PM, Ticker: VS, State: Historical, BarsInProgress: 1, Close: 0.51, Open: 0.51, CurrentBars[0]: 4619, CurrentBars[1]: 50158, BarsArray[0].Count-1: 4621, BarsArray[1].Count-1: 50159
4620 7/26/2022 5:53:20 PM
Debug Time: 7/26/2022 5:53:20 PM, Ticker: VS, State: Historical, BarsInProgress: 0, Close: 0.51, Open: 0.51, CurrentBars[0]: 4620, CurrentBars[1]: 50158, BarsArray[0].Count-1: 4621, BarsArray[1].Count-1: 50159
50159 7/26/2022 5:53:20 PM
Debug Time: 7/26/2022 5:53:20 PM, Ticker: VS, State: Historical, BarsInProgress: 1, Close: 0.52, Open: 0.52, CurrentBars[0]: 4620, CurrentBars[1]: 50159, BarsArray[0].Count-1: 4621, BarsArray[1].Count-1: 50159
4621 7/26/2022 5:53:30 PM

Please notice that in both reports Bar 4619 is for 7/26/2022 5:47:30 PM, but for the later bars after realtime kicked in the 10sec bars go out of sync. 7/26/2022 5:53:30 PM in the first example is bar 4622, while in the second example its bar 4621. If you look at the 10 sec bars in each of these, the 1st example has a bar for 5:49:20 PM whereas the historical does not. This time is immediately before the switchover from processing historical data to beginning realtime. It seems like the moment you start a strategy and it processes the historical data and is up to date and ready to switch to real-time, that it seemingly creates a new 10sec bar (and increments the bar count by one vs pure historical). From the debugging I think I can likely solve my strategy issue now but still curious if the system is creating an extra bar at the transition from historical to realtime. Thanks again Chris

[NinjaTrader_ChrisL]

Hi, I filtered the bar updates to only the 1 minute bars and it looks fine for me:

https://clauber.tinytake.com/msc/NzE2MzYyM18xOTkzODIyNg