Backtesting Data Anomolies?

[HaveGunsWillTravel]

I am working on a strategy using advance order handling techniques, which I am pretty sure I have correct.
I used your example templates and other notes here as a reference.
I got some very peculiar back-testing anomalies. I saw the following outlier trades:


Here is a zoomed in part:


You can see the back test is showing trades that are WAY off the chart.

As a troubleshooting step, I deleted all my database historical data for NQ06-22 then re-downloaded it.

Now my strategy is completely unprofitable while changing NOTHING but the data download! Like a $15,000 difference.

So I figured I wold turn off 'use local data' in SA properties, but it was NOT checked.

So I checked it, restarted the platform, re-ran the back-test, but still got the later crappy results, which leads me to believe the first anomalies were NOT on local data.

So I scrolled through trade results and the chart, and this time do not see outliers like above (which makes sense I guess, crappy results make more sense for me unfortunately)

But, what happened here? I am unsure how to go forward not knowing what is the actual cause...

Thank you.

[NinjaTrader_Kate]

Hello HaveGunsWillTravel,

Thank you for your post.

This looks like a data issue, either with bar caching or the historical data being corrupted.

When you encounter an issue like this, deleting and re-downloading the data is a good first step. You can also try clearing the cache.

To delete the cache, follow the steps below.

• Shut down NinjaTrader.
• Open the Documents > NinjaTrader 8 > db folder.
• Delete the sub-folder named 'cache'.
• Restart NinjaTrader and test.

Please let us know if we may be of further assistance.

[HaveGunsWillTravel]

Hello HaveGunsWillTravel,
This looks like a data issue, either with bar caching or the historical data being corrupted.
When you encounter an issue like this, deleting and re-downloading the data is a good first step. You can also try clearing the cache.
To delete the cache, follow the steps below:

I deleted the cache as you listed (after making a backup), restarted computer, then restarted NT8.
Upon restarting NT8...it immediately started running a back test unprompted. The results I saw were different from anything ever seen before, not sure where they came from.

I re-ran the back test same parameters as before in previous post (use local data checked and ALSO not checked)
Results were identical to the 'not great' results I got in the later case.
So it appears delete cache has not changed anything from my own step of re-downloading data (aside from the arrant unknown start-up back test).
Bad results are consistent across local data and not checked local data.

The unprompted back test NT8 ran on start-up raises more questions since it was favorable, I have no idea where it got the inputs from.

This concerns me still, since deleting cache does not seem to change much.
I can assume my results now are accurate? I only assume that because they are poor (statistically probable) and match the results after my own data re-download and the delete cache.

The question remains: What was happening before? (Corrupt Data? Then when and how?)
Why the strange off chart entries and closes? (Does not appear to be cache related)
How do I know I am not in that situation while iterating? (No obvious precursor)

This can be problematic for obvious reasons with back testing.

Thank you for your continued investigation and help.

[HaveGunsWillTravel]

And here is another anomaly I do not understand during back testing the same....

While performing a Walk-Forward optimization...my output shows this for an extremely long time:


My walk-forward back test range is 1-1-2020 to 12-31-2021


WHY is it showing toggling between 1-12-2020 and 4-20-2022!? 2022 is not even in the range?

What is the reason for this?

[NinjaTrader_Kate]

Hello HaveGunsWillTravel,

Thank you for your replies.

As far as the data issues go, most likely it's a bad download, could be a little blip in the connection. If you see that happen again, delete the data and redownload it.

Regarding the prints - I'd like to test that on my end, but it appears to be the date and time the script is being run rather than the time of the currently processing bar.

Where are you printing those lines from and what's the code you're using in the print?

I note you have an added data series in the script - what is the added series?

Thanks in advance; I look forward to assisting you further.

[HaveGunsWillTravel]

Ok, on the data. It appears to work now. Strange.

I don't have any Prints that would print what you see in the window there. After about 1-2 hours that went away then I started seeing the normal orders and the Prints I do have cycling through.

I do have an added series of a 1-tick default symbol I use for executing the orders on for higher granularity. I do not have prints for date or time anywhere. Only for order executions and orders.


I put this in there, for later. It is the first I have used it, but I don't think it would do that would it?

[B]else if (State == State.Realtime)[/B][INDENT][B]{
// one time only, as we transition from historical
// convert any old historical order object references
// to the new live order submitted to the real-time account
if (entryOrder != null)[/B][/INDENT][INDENT=2][B]entryOrder = GetRealtimeOrder(entryOrder);[/B][/INDENT][INDENT][B]if (stopOrder != null)[/B][/INDENT][INDENT=2][B]stopOrder = GetRealtimeOrder(stopOrder);[/B][/INDENT][INDENT][B]if (targetOrder != null)[/B][/INDENT][INDENT=2][B]targetOrder = GetRealtimeOrder(targetOrder);[/B][/INDENT][INDENT][B]}[/B][/INDENT]

Maybe this is an issue? I only say that because the windows had bars in progress stuff going on. Maybe this it the reason?
[0] is 1-minute and [1] is 1-tick series for orders.

[B]if (CurrentBars[0] < 60 || CurrentBars[1] < 3000)[/B][INDENT][B]return;[/B][/INDENT]

[NinjaTrader_Kate]

Hello HaveGunsWillTravel,

Thank you for your reply.

No, none of those would print that - however, are you using CancelAllOrders() anywhere in your script? Do you have any addons that might try to cancel all orders after a specific time?

Thanks in advance; I look forward to assisting you further.

[HaveGunsWillTravel]

Hello HaveGunsWillTravel,

Thank you for your reply.

No, none of those would print that - however, are you using CancelAllOrders() anywhere in your script? Do you have any addons that might try to cancel all orders after a specific time?

Thanks in advance; I look forward to assisting you further.

Negative, no cancle all.....

here is the bulk of the advanced handling part...I redacted only logical parts i want to keep private, if you want to inspect other parts let me know:

sorry about the indentation, tried to get it like it appears in the strategy, took a bit.

protected override void OnOrderUpdate(Order order, double limitPrice, double stopPrice, int quantity, int filled, double averageFillPrice, OrderState orderState, DateTime time, ErrorCode error, string nativeError)
{[INDENT]// Handle entry orders here. The entryOrder object allows us to identify that the order that is calling the OnOrderUpdate() method is the entry order.
// Assign entryOrder in OnOrderUpdate() to ensure the assignment occurs when expected.
// This is more reliable than assigning Order objects in OnBarUpdate, as the assignment is not guaranteed to be complete if it is referenced immediately after submitting

if (order.Name == "L1")
{[/INDENT][INDENT=2]entryOrder = order;

// Reset the entryOrder object to null if order was cancelled without any fill
if (order.OrderState == OrderState.Cancelled && order.Filled == 0)
{[/INDENT][INDENT=3]entryOrder = null;
sumFilled = 0;[/INDENT][INDENT=2]}[/INDENT][INDENT]}[/INDENT][INDENT]if (order.Name == "S1")
{[/INDENT][INDENT=2]entryOrder = order;

// Reset the entryOrder object to null if order was cancelled without any fill
if (order.OrderState == OrderState.Cancelled && order.Filled == 0)
{[/INDENT][INDENT=3]entryOrder = null;
sumFilled = 0;[/INDENT][INDENT=2]}[/INDENT][INDENT]}[/INDENT]
 
 }


protected override void OnExecutionUpdate(Execution execution, string executionId, double price, int quantity, MarketPosition marketPosition, string orderId, DateTime time)
{[INDENT]/* We advise monitoring OnExecution to trigger submission of stop/target orders instead of OnOrderUpdate() since OnExecution() is called after OnOrderUpdate()
which ensures your strategy has received the execution which is used for internal signal tracking. */
if (entryOrder != null && entryOrder == execution.Order)
{[/INDENT][INDENT=2]if (execution.Order.OrderState == OrderState.Filled || execution.Order.OrderState == OrderState.PartFilled || (execution.Order.OrderState == OrderState.Cancelled && execution.Order.Filled > 0))
{[/INDENT][INDENT=3]// We sum the quantities of each execution making up the entry order
sumFilled += execution.Quantity;
Print("SUM Filled " + sumFilled +" " + "Execution Name: " + execution.Name + " Order ID " + orderId);
//Determine if Execution is Long or Short[/INDENT][INDENT=2]if (execution.Name == "L1")[/INDENT][INDENT=3]{[/INDENT][INDENT=4]Print(execution.Order.ToString());
Print("___");
// Submit exit orders for partial fills Long Side, rounding order price to instr. tick size, based on XXXXXXXXXX etc...[/INDENT][INDENT=3]if (execution.Order.OrderState == OrderState.PartFilled)
{[/INDENT][INDENT=4]Print("Long Stop Partial");
stopOrder = ExitLongStopMarket(1, true, execution.Order.Filled, Instrument.MasterInstrument.RoundToTickSize(execution.Order.AverageFillPrice - (XXXXXXXXXX)), "InitStop", "L1");
Print("Long Target Partial");
targetOrder = ExitLongLimit(1, true, execution.Order.Filled, Instrument.MasterInstrument.RoundToTickSize(execution.Order.AverageFillPrice + (XXXXXXXXXX)), "L1Target", "L1");
Print("___");[/INDENT][INDENT=3]}[/INDENT][INDENT=3]// Update our exit order quantities once orderstate turns to filled and we have seen execution quantities match order quantities
else if (execution.Order.OrderState == OrderState.Filled && sumFilled == execution.Order.Filled)[/INDENT][INDENT=4]{[/INDENT][INDENT=5]// Stop-Loss order for OrderState.Filled Long Side, rounding order price to instr. tick size, based on XXXXXXXXXX etc...
Print("Long Stop");
stopOrder = ExitLongStopMarket(1, true, execution.Order.Filled, Instrument.MasterInstrument.RoundToTickSize(execution.Order.AverageFillPrice - (XXXXXXXXXX)), "InitStop", "L1");
Print("Long Target");
targetOrder = ExitLongLimit(1, true, execution.Order.Filled, Instrument.MasterInstrument.RoundToTickSize(execution.Order.AverageFillPrice + (XXXXXXXXXX)), "L1Target", "L1");
Print("___");[/INDENT][INDENT=4]}[/INDENT][INDENT=3]}[/INDENT][INDENT=2]else if (execution.Name == "S1")[/INDENT][INDENT=3]{[/INDENT][INDENT=4]Print(execution.Order.ToString());
Print("___");[/INDENT][INDENT=4]// Submit exit orders for partial fills Short Side, rounding order price to instr. tick size, based on ATR tgt factor etc...
if (execution.Order.OrderState == OrderState.PartFilled)[/INDENT][INDENT=5]{[/INDENT][INDENT=6]Print("short Stop Partial");
stopOrder = ExitShortStopMarket(1, true, execution.Order.Filled, Instrument.MasterInstrument.RoundToTickSize(execution.Order.AverageFillPrice + (XXXXXXXXXX)), "InitStop", "S1");
Print("Short Target Partial");
targetOrder = ExitShortLimit(1, true, execution.Order.Filled, Instrument.MasterInstrument.RoundToTickSize(execution.Order.AverageFillPrice - (XXXXXXXXXX)), "S1Target", "S1");
Print("___");[/INDENT][INDENT=5]}[/INDENT][INDENT=4]// Update our exit order quantities once orderstate turns to filled and we have seen execution quantities match order quantities
else if (execution.Order.OrderState == OrderState.Filled && sumFilled == execution.Order.Filled)[/INDENT][INDENT=5]{[/INDENT][INDENT=6]// Stop-Loss order for OrderState.Filled Short Side, rounding order price to instr. tick size, based on XXXXXXXXXX etc...
Print("Short Stop");
stopOrder = ExitShortStopMarket(1, true, execution.Order.Filled, Instrument.MasterInstrument.RoundToTickSize(execution.Order.AverageFillPrice + (XXXXXXXXXX)), "InitStop", "S1");
Print("Short Target");
targetOrder = ExitShortLimit(1, true, execution.Order.Filled, Instrument.MasterInstrument.RoundToTickSize(execution.Order.AverageFillPrice - (XXXXXXXXXX)), "S1Target", "S1");
Print("___");[/INDENT][INDENT=5]}[/INDENT][INDENT=3]}[/INDENT][INDENT=2]// Resets the entryOrder object and the sumFilled counter to null / 0 after the order has been filled
if (execution.Order.OrderState != OrderState.PartFilled && sumFilled == execution.Order.Filled)
{[/INDENT][INDENT=3]entryOrder = null;
sumFilled = 0;[/INDENT][INDENT=2]}[/INDENT][INDENT]}[/INDENT]
 
 }

[NinjaTrader_Kate]

Hello HaveGunsWillTravel,

Thank you for your reply.

I'm not seeing anything within that code that would trigger the prints you were getting. I'd keep an eye out and if it happens again, note any indicators, strategies or add-ons that may be running at the time. You can also remove them one by one to see when the prints cease to narrow down what may have been the cause.

Please let us know if we may be of further assistance to you.