Difference between live and historical data

[cliffhu]

Hi,

My strategy is running on 10 min bars of two instruments. I keep my positions overnight and restart NT before each session. I understand the position I had yesterday was based on live data and restarting NT today gives me the same position using yesterday's historical data. I had no problem in the past.

Recently I replaced the close of 10 min bars by using average of the last three 1 min bars of every 10 min interval. But now I get very often different positions when I restart the strategy. I guess the problem is due to difference between live and historical data of the 1 min bars, especially when there is no transaction in the market during that minute. My data provider is Kinetick.

Any idea how to fix this issue? I feel the historical data is more reliable to use. Is it possible to force the strategy to run on latest historical data? and how much delay comparing to live feed?

Thanks

[NinjaTrader_BrandonH]

Hello cliffhu,

Thanks for your post.

There would not be a way to force the strategy to use only historical data when running the strategy real-time.

When a strategy is enabled, it processes historical data to determine trades that the strategy would have made on the data that is already on the PC/chart and to determine what position the strategy is in. (Strategy positions are separate from actual Account positions.)

If you are wanting to resume a position after a strategy is disabled/enabled, the Immediately Submit Start Behavior could be used. This automatically submits working orders from when the strategy processed historical data, and assumes the strategy position and account position are where you want it when you enable the strategy. This is typically used to have a strategy resume a position after disabling/enabling. If the strategy already had live orders running, the orders will resume with the new enablement of the strategy if they match the historically calculated orders. If the orders calculated from historical data do not match the live working orders, the live working orders will be canceled and replaced by those calculated from historical data.

https://ninjatrader.com/support/helpGuides/nt8/syncing_account_positions.htm

Note that in historical data there is no intrabar granularity available. Only the OHLC information of a bar is known and the script can only process using Calculate.OnBarClose. This means actions cannot happen intra-bar, fills cannot happen intra-bar. All prices and actions come from and occur when the bar closes as this is all the information that is known.

You could try adding intrabar granularity and enabling Tick Replay to help with this.

Tick Replay would be used to ensure that the market data (bid/ask/last) that went into building a bar is loaded in the exact sequence of market data events. This guarantees that your indicators and strategies are historically calculated tick-per-tick exactly as they would have been if the indicator/strategy was running live during a period. However, this does not allow for intra-bar order fills. You would need to add a single-tick data series and submit orders to that single-tick data series for a strategy that uses Tick Replay.

SampleIntrabarBacktest 'Backtesting NinjaScript Strategies with an intrabar granularity' - https://ninjatrader.com/support/helpGuides/nt8/backtesting_ninjascript_strate.htm?zoom_highlights ub=sampleintrabar

TickReplay — https://ninjatrader.com/support/help...ick_replay.htm

Developing for Tick Replay -
https://ninjatrader.com/support/helpGuides/nt8/developing_for__tick_replay.htm

Additional information may be found in this NinjaTrader Forum post —
https://ninjatrader.com/support/forum/forum/ninjatrader-8/strategy-development/100192-comparing-real-time-historical-and-replay-performance?t=102504

To determine exactly how your strategy is behaving, debugging prints should be added to the script that prints out all the values being used for your conditions to place orders.

Below is a link to a forum post that demonstrates how to use prints to understand behavior.
https://ninjatrader.com/support/foru...121#post791121

[cliffhu]

Hi Brandon,

Thanks for your quick reply.

My idea was to replace the close of 10 min bar with last three 1 min bars of that 10 min interval. I added both bar series, with the 10 min as primary. When it close, it takes the average of last three 1 min bars. Do you see what could be the problem in this approach?

I feel the positions based on historical data is what I usually get. My problem is the 1 min live bars doesn't trigger that position, or giving me wrong calculation. So I wouldn't save live data as historical and continue from that position. Best would be to re-calculate as soon as possible using historical data from server.

[NinjaTrader_BrandonH]

Hello cliffhu,

Thanks for your notes.

It sounds like you have changed the logic of your script and are now seeing a difference between how that logic calculates historically compared to real-time.

You would need to debug the script by adding prints to the script that prints out all the information being used to place orders to understand how your conditions are evaluating in each use case to get an idea of what is happening differently.

To understand why the script is behaving as it is, it is necessary to add prints to the script that print all the values used for the logic of the script to understand how the script is evaluating.

Below is a link to a forum post that demonstrates how to use prints to understand behavior.
https://ninjatrader.com/support/foru...121#post791121

Additional information may be found on this forum thread: https://forum.ninjatrader.com/forum/...mance?t=102504

[cliffhu]

Hi Brandon,

The only change I made is like this,

primary data series, period = 10
add secondary data series, period = 1
....
if (BarsInProgress == 1)
return;
....
then replace all Closes[0][0] with SMA(Closes[1], 3)[0]

No change in the logic of my strategy. It is only the difference between live and historical data that made the difference in positions.
Thanks for the links. But they are not what I need.

[NinjaTrader_BrandonH]

Hello cliffhu,

Thanks for your notes.

Adding a secondary series and replacing the primary series Close price with an SMA indicator value being calculated from the Close of the added series would be considered a change in the logic of your script since you are no longer using the same values as before.

Prints would need to be added to the script to understand how the logic is behaving differently in each use case to get an idea of what is happening differently after changing that code.

Prints will appear in a New > NinjaScript Output window.

The forum thread linked on post # 4 demonstrates how to use prints to understand the behavior of a script.

[cliffhu]

Hi,

I took a look at the database on two different computers. The upper one is historical data downloaded from server. This computer never connect during the session. The lower one is from my trading computer which receives live feed and restarts before each session.

Let's take example of 15:30. I understand in the upper one the 10 min bar is not formed until15:36. what do we get from the average of three 1 min bar at this time? For the lower one, does the 10 min bar close at 15:30? what do we get from the average of three 1 min bar at this time?

In addition, the volume at 15:37 and 15:25 are also different.

[NinjaTrader_BrandonH]

Hello cliffhu,

Thanks for your notes.

What version of NinjaTrader are you using? Please provide the entire version number. This can be found under Help -> About (Example: 8.?.?.?)

What instrument are you using to test this?

What is the contract expiry of the instrument you are testing?

Are you downloading Historical Minute data in the Historical Data window?

Who are you connected to for live data? This is displayed in green on the lower-left corner of the Control Center window.

If you download Historical data on both machines, do you see the downloaded Historical data match?

A 10-minute bar at 15:30 would contain data from 15:20 to 15:30.

A 1-Minute bar at 15:30 would contain data from 15:29 to 15:30.

[cliffhu]

Please see my answers below.

Hello cliffhu,

Thanks for your notes.

What version of NinjaTrader are you using? Please provide the entire version number. This can be found under Help -> About (Example: 8.?.?.?)

Ver. 8.0.28.0

What instrument are you using to test this?
SVXY

What is the contract expiry of the instrument you are testing?
no expiry date

Are you downloading Historical Minute data in the Historical Data window?
I downloaded the upper one from a chart

Who are you connected to for live data? This is displayed in green on the lower-left corner of the Control Center window.
Kinetic, as mentioned before

If you download Historical data on both machines, do you see the downloaded Historical data match?
Is this necessary? I will be surprised if they are different in that way.

A 10-minute bar at 15:30 would contain data from 15:20 to 15:30.
Obviously. But not answering my question.
Or maybe not so sure, if no transaction in the market on that exact time. That's was my question.

A 1-Minute bar at 15:30 would contain data from 15:29 to 15:30.

[cliffhu]

I compared these historical data from IB and here is my finding.

The upper one in my previous screenshot is from Kinetick. I assume / hope it is the same as live feed. There are many missing data points due to no transaction in the market. They are not filled with dummy data, neither in database nor at running a strategy. So just doesn't trigger any calculation.

The lower one is from my trading computer. It has two live feeds, IB and Kinetick, with Kinetick as preferred for both live and historical. As Kinetick has so many missing data point, I think NT downloaded automatically from IB when I restart it every time.

The following screenshot is a comparison, upper panel from Kinetic and lower one from IB. SMA using each of them are plotted together on the upper panel. I can see there is huge difference. The blue one using Kinetick is totally useless.

Now my question is whether it is possible for NT to fill those missing data points with dummy values like IB does? Otherwise I believe I have to drop Kinetic. Though IB is said not good in some other aspects, at least it doesn't mess up my calculations. I very sad that my time was wasted on development using Kinetick and loss caused in live trading.

Another question to other users in this forum, does anyone know a good data provider that doesn't make this problem?

[NinjaTrader_BrandonH]

Hello cliffhu,

Thanks for your notes.

Is the SVXY instrument and the SVXY5 instrument the exact same instruments?

Do you have the 'Get data from server' option enabled in the Control Center > Tools > Options > Market Data tab?

If you right-click in the Chart window and select 'Reload All Historical Data', do you see the historical data repopulate on the chart and resolve the behavior you are reporting?

https://ninjatrader.com/support/helpGuides/nt8/reload_historical_data.htm

[cliffhu]

Hi,

Is the SVXY instrument and the SVXY5 instrument the exact same instruments?

SVXY and SVXY5 are exact same instruments. They are just to separate data from different providers.

Do you have the 'Get data from server' option enabled in the Control Center > Tools > Options > Market Data tab?

'Get data from server' option is enabled.

If you right-click in the Chart window and select 'Reload All Historical Data', do you see the historical data repopulate on the chart and resolve the behavior you are reporting?

If connected to Kinetick, there are same holes in historical 1-min bars like real time.

[NinjaTrader_BrandonH]

Hello cliffhu,

Thanks for your notes.

If you are comparing Kinetick historical data to IB historical data then there will likely be differences in the historical data since different data providers are being compared.

You would need to make sure the same historical data was downloaded from the same provider on both machines by using the same connection and then compare the data.

To test this, I suggest disconnecting from your connections on both PCs, navigating to the Tools > Historical Data window, and deleting the Historical Data in this window by right-clicking on the instrument and selecting 'Remove'.

After removing the Historical data, connect to the same connection on both PCs, such as connecting to only the Kinetick connection on both PCs.

Then, open a New > Chart window on both PCs for the same instrument. Next, right-click on the chart window, select 'Reload All Historical Data'.

Finally, compare the data on both PCs to see if the data matches.

[cliffhu]

Hi,

I know data from different providers can be different. I have set Kinetick as preferred for both live and historical, and connect at start, but database has always historical from IB. The positions I have in live will be different when restart as they will be calculated from historical.

Those holes in 1-min bar from Kinetick should be filled with dummy values like IB does. Otherwise they will mess up the OBU calculation and difficult to sync with other series.

[NinjaTrader_BrandonH]

Hello cliffhu,

Thanks for your notes.

You would not have any historical data from IB unless you connected to only IB at startup and then loaded the chart before connecting to Kinetick.

You will need to make sure that you connect to the Kinetick connection first when starting NinjaTrader and then connect to IB so the Kinetick data is downloaded.

Further, Kinetick does not have dummy values added to their data and I am not aware of any data providers that do this. If IB provides dummy values then this would affect how a script processes.

You should make sure that your script works as you want by using the Kinetick data if that is the Live and Historical data provider you want to use.

Otherwise, you would need to use only IB and not use Kinetick.