Historical Bid/Ask Data Series during Backtesting

[kimsteve]

Hi!

I have a strategy that computes a variable using Historical Bid/Ask Data Series:

private double variable;
if (State == State.Configure)
{
AddDataSeries(null, BarsPeriodType.Tick, 1, MarketDataType.Last);
AddDataSeries(null, BarsPeriodType.Tick, 1, MarketDataType.Ask);
AddDataSeries(null, BarsPeriodType.Tick, 1, MarketDataType.Bid);
AddDataSeries(null, BarsPeriodType.Minute, 1);
}

protected override void OnBarUpdate()
{
region Volumetric Bar Updates
if (BarsInProgress == 0)
{
// helper function to plot the variable value on top of the bar
variable = 0;
}
#endregion

region Last Updates
else if (BarsInProgress == 1)
{
// logic to place orders based on variable value
}
#endregion

region Ask Update
else if (BarsInProgress == 2)
{
double price = Close[0];
double volume = Volume[0];
/*
Compute Variable here
*/
}
#endregion

region Bid Update
else if (BarsInProgress == 3)
{
double price = Close[0];
double volume = Volume[0];
/*
Compute Variable here
*/
}
#endregion
}

I am noticing that there is a discrepancy in the variable's value in Strategy Analyzer (Loading historical data) vs Market Replay. The variable is computed for each bar and printed at the top of the bar (the variable is reset at the close of each Volumetric Bar (main series)).

Historical/Strategy Analyzer:

Market Replay:


Please let me know where the discrepancy is happening, and if this is something that can be fixed via code or if this is just a backtesting problem.

[NinjaTrader_Jesse]

Hello kimsteve,

The code for how you calculate the variable is not included so there would be no way to answer the question with the given information. You likely need to use prints to see what values are changing between those two tests and how that affects your calculation to know how to fix it.

[kimsteve]

Hi Jesse,

The variable is computed using the Close[0] and Volume[0] of the Bid and Ask bars:

region Ask Update
else if (BarsInProgress == 2)
{
double price = Close[0];
double volume = Volume[0];
/*
variable = func(price, volume, prevAskPrice, prevAskVolume);
*/
prevAskPrice = price;
prevAskVolume = volume;
}
#endregion

region Bid Update
else if (BarsInProgress == 3)
{
double price = Close[0];
double volume = Volume[0];
/*
variable = func(price, volume, prevBidPrice, prevBidVolume);
*/
prevBidPrice = price;
prevBidVolume = volume;
}
#endregion
}

I cannot specify the formula to compute the variable, but it is a function of the price, volume, and the previous bid/ask price/volume. Please let me know if you need any additional information.

[NinjaTrader_Jesse]

Hello kimsteve,

You will need to use prints in your script if you cannot provide a sample of what you are calculating,

Prints are always a first step at diagnosing any kind of differences you see in testing modes, you would need to print out the values you use in your calculation along with any other variables used to see what is different.

[kimsteve]

Hi Jesse,

I will follow up with you with the logs, although I don't see a need to Print diagnostics in this case, as the variable is simply a function of the historical bid/ask series. If we are seeing discrepancies in the variable values as shown in the pictures I've attached, the only discrepancy that is possible is within the historical bid/ask values and bid/ask series in Market Replay. My question is if differences between the historical bid/ask series and market replay bid/ask series is expected.

Thanks for your quick responses.

[NinjaTrader_Jesse]

Hello kimsteve,

The logs won't be helpful to me without seeing the code you are printing so that would mainly only be helpful to you to determine what part of your calculation is happening differently.

As I don't have a specific example of what you are trying to calculate I am very limited in how I can assist, you will need to explore what you made by using prints to get a more in depth idea of what difference is happening in those two use cases.

You can see differences when testing in different modes for a lot of reasons and in almost all cases you would have to use prints to explore those differences. There is a post in the following link that goes over some common differences that you may see when testing in different modes and how you would approach using prints to debug that situation. You would have to verify that everything was identical between the two tests. If anything happens different between the two tests you should not expect an identical result.

https://ninjatrader.com/support/forum/forum/ninjatrader-8/strategy-development/100192-comparing-real-time-historical-and-replay-performance

[kimsteve]

To reiterate, the variable is simply a function of the Close[0], Volume[0], and previous Close, and previous Volume. For instance, here is a simplified version of the variable:

if (BIP = 2) // BIP for Historical Ask Series
if (Close[0] > prevClose){
variable += prevVolume;
}

prevClose = Close[0];

The only way we will see differences in the variable is if the ask series differs from backtesting data, vs. market replay.

Its quite a simple issue, I hope you can answer whether this is expected or not.

[kimsteve]

If it helps, I can create a new strategy that computes a dummy variable based on ask and bid series, and show you they compute different values for each bar. Would this help in determining what the issue is?

[NinjaTrader_Jesse]

Hello kimsteve,

To reiterate, the variable is simply a function of the Close[0], Volume[0], and previous Close, and previous Volume. For instance, here is a simplified version of the variable:

This does not help to explain why it may be different between tests. You still need to use prints to explore what you made to know what specific difference is happening so you can explore that further.

If it helps, I can create a new strategy that computes a dummy variable based on ask and bid series, and show you they compute different values for each bar. Would this help in determining what the issue is?

You can do that but that would also create double work for you and you would still need to add prints to show what specific difference you are seeing between tests. I can't debug your code for you but I can suggest ways that you can debug the code. Showing the original calculation would be helpful so that I could suggest specific parts of the code to add prints, without that you would just have to use your best judgement to explore what you created by using prints.

Prints are the best option when comparing two different use cases, you would need to output how your logic is working or the values used in the calculation in each use case so you can get a more specific idea of what was different. Once you know that you can adjust the logic as needed to change the test settings so they match identically. As mentioned if you have any differences between the two tests you should not expect an identical result. The tests have to be completely identical and the way to know that is to use prints.

[cforum542]

I have the CBOT_DOM data provided from ninja trader, I like the concept of the plan but I only get the data with a 10 minute delay, also I have other data plans that I would like to access but it wont let me configure my data settings. I am running ninja trader on my macbook through windows parallels and have had no problems yet. But i havent been able to set up my data to be able to trade with. Any help would help. (I have a picture of the data setup bar in the bottom)

Thank You,
Andrea ****son.

[kimsteve]

Hi Jesse,

After printing to the logs, this is what I found:

2023-09-19 06:35:10.848 Last Price: 15356.75 Volume: 1
2023-09-19 06:35:10.872 Last Price: 15357 Volume: 1
2023-09-19 06:35:10.868 Bid Price: 15356.5 Bid Volume: 4
2023-09-19 06:35:10.872 Bid Price: 15356.75 Bid Volume: 1
2023-09-19 06:35:10.872 Last Price: 15357 Volume: 1
2023-09-19 06:35:10.872 Bid Price: 15356.5 Bid Volume: 4
2023-09-19 06:35:10.872 Bid Price: 15356.75 Bid Volume: 1
2023-09-19 06:35:10.872 Bid Price: 15356.75 Bid Volume: 2
2023-09-19 06:35:10.872 Bid Price: 15356.75 Bid Volume: 3
2023-09-19 06:35:10.872 Last Price: 15357 Volume: 1
2023-09-19 06:35:10.872 Bid Price: 15356.75 Bid Volume: 2
2023-09-19 06:35:10.876 Bid Price: 15356.75 Bid Volume: 1
2023-09-19 06:35:10.876 Bid Price: 15356.75 Bid Volume: 2
2023-09-19 06:35:10.876 Bid Price: 15356.75 Bid Volume: 1
2023-09-19 06:35:10.876 Last Price: 15356.75 Volume: 1
2023-09-19 06:35:10.928 Last Price: 15357.25 Volume: 1
2023-09-19 06:35:10.928 Last Price: 15357.25 Volume: 1
2023-09-19 06:35:10.892 Bid Price: 15356.75 Bid Volume: 2
2023-09-19 06:35:10.928 Bid Price: 15356.75 Bid Volume: 3
2023-09-19 06:35:10.928 Bid Price: 15356.75 Bid Volume: 4
2023-09-19 06:35:10.928 Bid Price: 15356.75 Bid Volume: 3
2023-09-19 06:35:10.928 Last Price: 15357.25 Volume: 1
2023-09-19 06:35:10.940 Bid Price: 15356.75 Bid Volume: 2
2023-09-19 06:35:10.964 Bid Price: 15356.75 Bid Volume: 1
2023-09-19 06:35:10.968 Bid Price: 15356.5 Bid Volume: 5
2023-09-19 06:35:10.968 Bid Price: 15356.5 Bid Volume: 4
2023-09-19 06:35:10.964 Last Price: 15356.75 Volume: 1
2023-09-19 06:35:11.004 Bid Price: 15356.75 Bid Volume: 1
2023-09-19 06:35:11.020 Bid Price: 15356.75 Bid Volume: 2
2023-09-19 06:35:11.020 Bid Price: 15356.75 Bid Volume: 3
2023-09-19 06:35:11.020 Bid Price: 15356.75 Bid Volume: 4
2023-09-19 06:35:11.024 Bid Price: 15356.75 Bid Volume: 3
2023-09-19 06:35:11.024 Bid Price: 15356.75 Bid Volume: 2
2023-09-19 06:35:11.024 Bid Price: 15356.75 Bid Volume: 1
2023-09-19 06:35:11.024 Bid Price: 15356.5 Bid Volume: 6
2023-09-19 06:35:11.024 Bid Price: 15356.5 Bid Volume: 5
2023-09-19 06:35:11.024 Bid Price: 15356.5 Bid Volume: 4
2023-09-19 06:35:11.020 Last Price: 15357 Volume: 1
2023-09-19 06:35:11.024 Bid Price: 15356.5 Bid Volume: 3
2023-09-19 06:35:11.024 Bid Price: 15356.75 Bid Volume: 1
2023-09-19 06:35:11.024 Bid Price: 15356.75 Bid Volume: 2

The above are print statements from the following:

region Last Update
if (BarsInProgress == 1)
{
double price = Close[0];
double volume = Volume[0];
string timestamp = Time[0].ToString("yyyy-MM-dd HH:mm:ss.fff");
Print(timestamp + " Last Price: " + price + " Volume: " + volume);
}
region Ask Update
else if (BarsInProgress == 2)
{
double price = Close[0];
double volume = Volume[0];
string timestamp = Time[0].ToString("yyyy-MM-dd HH:mm:ss.fff");
Print(timestamp + " Bid Price: " + price + " Bid Volume: " + volume);
}
#endregion

region Bid Update
else if (BarsInProgress == 3)
{
double price = Close[0];
double volume = Volume[0];
string timestamp = Time[0].ToString("yyyy-MM-dd HH:mm:ss.fff");
Print(timestamp + " Ask Price: " + price + " Ask Volume: " + volume);
}
#endregion
}

From this I see that the ticks are not provided in the order of the timestamps for Market Replay, while backtesting sorts the events by timestamp. What can I do to ensure that the Last, Bid, and Ask price series are provided in the same order as MarketReplay in backtesting? And how come it is the case that time stamps are not chronological?

[NinjaTrader_Jesse]

Hello kimsteve,

In historical data the timestamp of the data is already known ahead of time so they are ordered based on that. In realtime there is no specific order in which ticks can come in so there would not be a way to force them to be ordered in a certain way. Backtesting uses historical data and playback uses realtime data so those two modes will inherently have differences. You could potentially use the historical playback instead of using playback data so that you can play through the historical data you have. You would otherwise need to compare tests where the data being used is the same type of data.

The post that I had previously linked to goes over some of the common differences that you would see when comparing a historical test vs a realtime test which includes a note about not comparing using different types of data because that will cause differences. To get an identical result you would need to compare either two backtests or two realtime tests so that the data is identical.

[kimsteve]

Hi Jesse,

This would mean that backtesting is completely obsolete for testing series that use Last, Bid, and Ask, since backtesting using these series would always result in different results than real time data, unless there is an option to run backtesting with the same "real time" data that is used for Market Replay. Is this correct?

[NinjaTrader_Jesse]

Hello kimsteve,

That would not make backtesting obsolete you are just using two different types of data. Backtesting uses historical data and playback uses recorded realtime data, you cannot directly compare the two data sources and how a script will process in those different use cases. You are using two different types of data for a comparison so we should in general expect differences. Backtesting is not a mirror image of what you would see in realtime so we shouldn't expect results from bactests to be exactly identical to a realtime testing mode. You can find some common differences that you would see when testing between historical and realtime here: https://ninjatrader.com/support/help...ightsub=discre

[kimsteve]

Hi Jesse,

To me, backtesting would be useful if it mimics real time data as closely as possible such that I can test how my strategies would have performed if they were ran live the the time period the back test was run. This is not the case, since the backtest engine orders the data by timestamp, which results in a different order than realtime or market replay. Could you help me understand how backtesting could be useful if its data is ordered differently from market replay?