Is the order engine broken on historical orders? Orders stuck in working state.

[koganam]

Well, for what it is worth, given my recent travails with NT support, and much to my chagrin, here I am reluctantly again bringing up an issue, but only because I am not satisfied with my code-around for the issue.

Here goes.

The issues:

1. Historical entry orders can get stuck in 'Working' state.
2. Such stuck orders ignore attempts to cancel them.

How to replicate:

• I have attached a Strategy file to demonstrate the issue.
• The file still has extensive print statements that I used to track order flow while trying to isolate the problem.
• The file also has copious comments, detailing my observations, and suggesting some tests (by uncommenting statements), to demonstrate that while I can code around the issue, the viability of the workaround is questionable.

Here is how to use that file to demonstrate the issue. We are demonstrating specifically that after a non-specific number of "entries and exit" on the same historical bar, some historical entry orders are stuck in 'Working' state, and CAN completely stop execution of any more orders if all entries are to be made from checking that entry orders are first null before entry.

Make targets and stops close enough that multiple historical bars will have entry and exit on the same bar. This seems to be a big part of the issue. When most trades exit on bars after entry, the problem does not arise quite as often.

• I used a daily timeframe on SPY. (I have also seen it on various futures and other stocks)
• Target ticks = 150.
• Stop ticks = 150.

The strategy makes no statement about profitability: it is written as simply as possible to demonstrate a specific issue that has nothing to do with the performance of the strategy.

I have attached some annotated diagrams from my installation, to illustrate a bit more clearly.






There is also a picture showing my NinjaTrader properties. This is from a brand new fresh installation on a physical computer (not virtual machine): there is no database needing to reset, but I will if that is what you direct.

[bltdavid]

Merry Christmas, Koganam!

Is this behavior limited just to the strategy analyzer?
I mean, you don't talk very specifically about the conditions for replication.
Ever happen in Sim101 with real-time data?
Besides SPY, what other instruments?
Besdies Daily, any other bar series?

I thought I'd help out, but all I have is futures data.

-=o=-

I see in your code that you don't assign orders in OnOrderUpdate.
The missing code in OnOrderUpdate would look like this,

if (order.Name == "Long" && orderState != OrderState.Filled)
    _oLongEntry = order;

I think this is the 'recommended way' in NT8.
Any reason why you forego doing it that way?

Just curious.

[koganam]

Merry Christmas, Koganam!

Is this behavior limited just to the strategy analyzer? - No. I first saw on a chart that sometimes trades just quit early on the chart. I started investigating, and it turns out that using the backtest is the most convenient and quickest way to track down strategy processing issues.
I mean, you don't talk very specifically about the conditions for replication. - I thought that that went without saying, as I had provided pictures, but it is the same whether done directly on the chart, with its "open dialog, disable strategy, change parameters, enable strategy, click OK button", or simply "change parameters and hit 'Run'" in the StrategyAanalyzer. Which would you rather do, if you had to interactively create statements to chart process flow? :-)
Ever happen in Sim101 with real-time data? - I am not sure that I understand the question, simple as it seems. Once entries are gated, the entries stop. It does not matter when. It is just that it was easier to track using the backtest mechanism, but yes, once it happens, real time orders cease too, pretty much as I expected.
Besides SPY, what other instruments? - Too many others. /MES, /ES, /RTY, /NQ, /YM, JPM, IBM, (I forget the others on which I tried).
Besdies Daily, any other bar series? - I tested on daily timeframe only, because the problem showed up on a complex swing trading strategy that I was using. I wrote this file as the simplest possible analog, for use with a single entry, trying to demonstrate the problem, albeit we do not know when it will happen. One just needs a long enough chart span to ensure that there are enough trades for it to happen.

I thought I'd help out, but all I have is futures data.

-=o=-

I see in your code that you don't assign orders in OnOrderUpdate.
The missing code in OnOrderUpdate would look like this,

if (order.Name == "Long" && orderState != OrderState.Filled)
_oLongEntry = order;

I think this is the 'recommended way' in NT8.
Any reason why you forego doing it that way? - It made no difference; it is convoluted. I simply assign the order when it is made. I have seen the documentation: I simply chose to do things more directly, especially after I had seen what I thought were glitches in how OnOrderUpdate was called. (The glitch that I though I saw was not there, it was a PBKAC ). An overload of the statement to place an order returns an 'Order' as its output; I simply assigned the order.

Just curious.

Merry Christmas to your too David. You were one of the folks that I was hoping might care to take a look

My responses, in blue, directly in your text that I quoted.

[bltdavid]

Hmm, the attached strategy in the original posting ... where did it go?

[EDIT: Never mind, I see it now]

[koganam]

Hmm, the attached strategy in the original posting ... where did it go?

[EDIT: Never mind, I see it now]

Yes, it got lost. I could not see it either when I was responding to your post. I had to remove it, and then attach a new copy.

As far as the OnOrderUpdate order assignment, now it is confusing, because in this post Ninjatrader_ChelseaB says specifically that it is a bad idea to assign orders there!?

ref: https://ninjatrader.com/support/foru...973#post828973

[bltdavid]

As far as the OnOrderUpdate order assignment, now it is confusing, because in this post Ninjatrader_ChelseaB says specifically that it is a bad idea to assign orders there!?

ref: https://ninjatrader.com/support/foru...973#post828973

Yes, but looking at his NT8 sample code, you can see that is exactly what his OnOrderUpdate
code does.

I agree his comments are a bit confusing, esp his last sentence, which strikes me as a complete
contradiction to the OnBarUpdate code in his NT8 sample.

Hopefully, we'll get his clarification soon.

-=0=-

Btw, these comments in his code (which also appear in code samples in the NT8 help guide),

// Assign Order variable from OnOrderUpdate() to ensure the assignment occurs when expected.
// This is more reliable than assigning the return Order object from the submission method as the
// assignment is not guaranteed to be complete if it is referenced immediately after submitting

are entirely unclear to me. The rationale about "the assignment is not guaranteed to be completed",
I'm a professional software engineer with over 30 years of experience, and I don't know what the
heck that means. Assignments are always 'completed', it's a simple ILASM command to 'assign'
a return reference to a variable.

If they mean the reference is returned before some members of the object are fully assigned, well,
why don't they frigging say that?

If they mean the reference to the object returned is not guaranteed to be the 'permanent' object,
that a newer Order object could be allocated and supersede the returned object, and that because
OnOrderUpdate is always guaranteed to have the correct 'permanent' object (no matter which one
it is, or when it was allocated), and that therefore they want you to (re)assign the Order reference
to your own variable from OnOrderUpdate -- well, if that's what is meant, it's such a god awful and
incomplete explanation that it makes an expert understanding impossible.

We should not have to speculate what is meant by NT8 documentation, but in this case it's rather
difficult to fully comprehend unless you 're-interpret' and 'expand' their comment -- because
'incomplete assignment' is too generic and uselessly begs more questions to my engineering
mind than it answers.

Sorry for the rant.
Just my 2˘.

[koganam]

...

Btw, these comments in his code (which also appear in code samples in the NT8 help guide),

// Assign Order variable from OnOrderUpdate() to ensure the assignment occurs when expected.
// This is more reliable than assigning the return Order object from the submission method as the
// assignment is not guaranteed to be complete if it is referenced immediately after submitting

are entirely unclear to me. The rationale about "the assignment is not guaranteed to be completed",
I'm a professional software engineer with over 30 years of experience, and I don't know what the
heck that means. Assignments are always 'completed', it's a simple ILASM command to 'assign'
a return reference to an object.

If they mean the reference is returned before some members of the object are fully assigned, well,
why don't they frigging say that?

...

We should not have to speculate what is meant by NT8 documentation, but in this case it's rather
difficult to fully comprehend unless you 're-interpret' and 'expand' their comment -- because
'incomplete assignment' is too generic and uselessly begs more questions to my engineering
mind than it answers.

Sorry for the rant.
Just my 2˘.

Once again, you say more clearly what I meant to say. I am not sorry about your rant. LOL.

And again, it looks like the cs file has disappeared from the original post. I guess I may have to redo the entire thing?

[NinjaTrader_ChelseaB]

Yes, but looking at his NT8 sample code, you can see that is exactly what his OnOrderUpdate
code does.

I agree his comments are a bit confusing, esp his last sentence, which strikes me as a complete
contradiction to the OnBarUpdate code in his NT8 sample.

Hopefully, we'll get his clarification soon.

-=0=-

Btw, these comments in his code (which also appear in code samples in the NT8 help guide),

// Assign Order variable from OnOrderUpdate() to ensure the assignment occurs when expected.
// This is more reliable than assigning the return Order object from the submission method as the
// assignment is not guaranteed to be complete if it is referenced immediately after submitting

are entirely unclear to me. The rationale about "the assignment is not guaranteed to be completed",
I'm a professional software engineer with over 30 years of experience, and I don't know what the
heck that means. Assignments are always 'completed', it's a simple ILASM command to 'assign'
a return reference to a variable.

If they mean the reference is returned before some members of the object are fully assigned, well,
why don't they frigging say that?

If they mean the reference to the object returned is not guaranteed to be the 'permanent' object,
that a newer Order object could be allocated and supersede the returned object, and that because
OnOrderUpdate is always guaranteed to have the correct 'permanent' object (no matter which one
it is, or when it was allocated), and that therefore they want you to (re)assign the Order reference
to your own variable from OnOrderUpdate -- well, if that's what is meant, it's such a god awful and
incomplete explanation that it makes an expert understanding impossible.

We should not have to speculate what is meant by NT8 documentation, but in this case it's rather
difficult to fully comprehend unless you 're-interpret' and 'expand' their comment -- because
'incomplete assignment' is too generic and uselessly begs more questions to my engineering
mind than it answers.

Sorry for the rant.
Just my 2˘.

My post was a complete typo. I had meant to say OnBarUpdate(). Corrected in the post.

The assignment might not be done because things are happening in different unsynchronized threads. The Instrument thread is running is running the script and when the order method is called, that happens on the order update thread and takes time to assemble and submit. OnBarUpdate() will have move on to processing code below and if that order is used, it's not the final updated order object that is actually active and updating in OnOrderUpdate(). So yes, different objects. Order vs IOrder object.
See the AssignOrderToVariable() method in my example. This is called from OnOrderUpdate() as that is method that will be updating with the final order object.


koganam,

Stuck historical orders typically end up being historical overfills in the managed approach.

Is this a managed or unmanaged strategy?

[koganam]

My post was a complete typo. I had meant to say OnBarUpdate(). Corrected in the post.

The assignment might not be done because things are happening in different unsynchronized threads. The Instrument thread is running is running the script and when the order method is called, that happens on the order update thread and takes time to assemble and submit. OnBarUpdate() will have move on to processing code below and if that order is used, it's not the final updated order object that is actually active and updating in OnOrderUpdate(). So yes, different objects. Order vs IOrder object.
See the AssignOrderToVariable() method in my example. This is called from OnOrderUpdate() as that is method that will be updating with the final order object.


koganam,

Stuck historical orders typically end up being historical overfills in the managed approach.

Is this a managed or unmanaged strategy?

Managed. But the orders that get stuck reach working state. Would a putative overfill not cause some order to be cancelled, or show some other indication of a process error?

And why can the stuck order not be cancelled, albeit it can be nullified?

[NinjaTrader_ChelseaB]

Hello koganam,

The symptom is the order specifically gets stuck in a Working state.

The typically cause is: 'A historical overfill whereby 3 orders (entry/target/stop) filled on the same bar'.

Do you have 3 orders that may have filled on that bar?

[koganam]

Hello koganam,

The symptom is the order specifically gets stuck in a Working state.

The typically cause is: 'A historical overfill whereby 3 orders (entry/target/stop) filled on the same bar'.

Do you have 3 orders that may have filled on that bar?

Your explanation, if that is what happened would make sense, but that is not what happened. For ease of reference, let us look at the events before and at the last trade that got suck.

On 8/19/2015, we see what looks like correct processing to me. There was an entry and exit on the bar, and we can clearly see from the Orders log that:

1. Entry filled
2. Stop cancelled
3. Target filled

Not all 3 filled.

Marked with the yellow rectangle in the picture below.



My detailed flow log shows those orders as they proceed.



The entry order gets stuck in working state.

[NinjaTrader_ChelseaB]

Hello koganam,

While I cannot see the time stamps, it appears there is a market order, limit order, and market order which would total 3 orders.

From the output it looks like all 3 were submitted on the 4:00 PM bar.

Are you seeing the 2 market orders were sent on different bars and would not have both filled with the limit order?

If these are sent on the same bar, does sending them on different bars still reproduce the behavior?

Or with Order Fill Resolution set to High does the behavior persist?

[koganam]

Hello koganam,

While I cannot see the time stamps, it appears there is a market order, limit order, and market order which would total 3 orders.

From the output it looks like all 3 were submitted on the 4:00 PM bar.

Are you seeing the 2 market orders were sent on different bars and would not have both filled with the limit order?

If these are sent on the same bar, does sending them on different bars still reproduce the behavior?

Or with Order Fill Resolution set to High does the behavior persist?

Only 3 orders were processed on 8/19/2015: An entry(Long) was filled , a Target (TgtLong) was filled and a Stop (StopLong) was cancelled.

The stuck order is dated 8/20/2015, the next day.

The order flow clearly shows the details of the sequence as the orders were placed and updated in their lifespan. I am not sure if we are not talking at cross-purposes.

Order Fill Resolution set to High and "Exit on session close" to false seems to take care of it. Only problem is that orders are being placed at midnight instead of on market open or close. I may have to make a custom session template. Just the limitations of backtesting, I guess.

I do find it a bit strange still: I am sure that I tested with high Order Fill Resolution, and found no difference. I must have been mistaken.

[NinjaTrader_ChelseaB]

Hello koganam,

If High Order Fill Resolution corrects the behavior, this likely tells us the issue is a historical overfill, as both the symptom and resolution match the behavior of a historical overfill.

1 tick intra-bar granularity and TickReplay with Calculate set to OnPriceChange or OnEachTick can be used instead of High Order Fill Resolution. Use FirstTickOfBar to trigger things after a bar has closed.

https://ninjatrader.com/support/forum/forum/ninjatrader-8/strategy-development/94098-isfirsttickofbar-vs-onbarclose-for-backtest-live?p=773377#post773377

I will test the script on my end. Please export and repost.

To export a NinjaTrader 8 NinjaScript so this can be shared and imported by the recipient do the following:

1. Click Tools -> Export -> NinjaScript...
2. Click the 'add' link -> check the box(es) for the script(s) and reference(s) you want to include
3. Click the 'Export' button
4. Enter a unique name for the file in the value for 'File name:'
5. Choose a save location -> click Save
6. Click OK to clear the export location message

By default your exported file will be in the following location:

• (My) Documents/NinjaTrader 8/bin/Custom/ExportNinjaScript/<export_file_name.zip>

Below is a link to the help guide on Exporting NinjaScripts.
http://ninjatrader.com/support/helpG...-us/export.htm

[koganam]

Hello koganam,

If High Order Fill Resolution corrects the behavior, this likely tells us the issue is a historical overfill, as both the symptom and resolution match the behavior of a historical overfill.

1 tick intra-bar granularity and TickReplay with Calculate set to OnPriceChange or OnEachTick can be used instead of High Order Fill Resolution. Use FirstTickOfBar to trigger things after a bar has closed.
https://ninjatrader.com/support/foru...377#post773377

I will test the script on my end. Please export and repost.

To export a NinjaTrader 8 NinjaScript so this can be shared and imported by the recipient do the following:

1. Click Tools -> Export -> NinjaScript...
2. Click the 'add' link -> check the box(es) for the script(s) and reference(s) you want to include
3. Click the 'Export' button
4. Enter a unique name for the file in the value for 'File name:'
5. Choose a save location -> click Save
6. Click OK to clear the export location message

By default your exported file will be in the following location:

• (My) Documents/NinjaTrader 8/bin/Custom/ExportNinjaScript/<export_file_name.zip>

Below is a link to the help guide on Exporting NinjaScripts.
http://ninjatrader.com/support/helpG...-us/export.htm

Thanks for your directions Chelsea, but I attached the .CS file to the first post when I made it. If you really prefer me to export a zip, of course, I shall, but is the CS file really insufficient.