NinjaTrader + Rithmic = Not updating positions

[NinjaTrader_ChelseaB]

Hello ntcoder256,

As you are willing to test things for me, I have additional questions that may be sensitive.

Would you be willing to have a private conversation with me through email?

If so, please send an email to scriptingsupport [at] ninjatrader.com.
In the email please write Attn: Chelsea #06020609

[ntcoder256]

Hello ntcoder256,

May I confirm you are using NinjaTrader 8.1.4.2?

Correct I am currently on 8.1.4.2

[ntcoder256]

Hello ntcoder256,

As you are willing to test things for me, I have additional questions that may be sensitive.

Would you be willing to have a private conversation with me through email?

If so, please send an email to scriptingsupport [at] ninjatrader.com.
In the email please write Attn: Chelsea #06020609

absolutely, will do. thanks

[Rudmax]

Hello ntcoder256,

Our developers are looking into the issue and the case is still open with them.

As there are 3 parties involved, it can be challenging to precisely identify the specific cause.

I have my own suspicions that the issue comes from addon code. I've requested that our business development reach out to Apex to assist me in investigating but unfortunately I have not received any information back.

See post # 78 where I've discussed my suspicions.

The gist is that the Account thread is the thread in NinjaTrader that creates and sends orders, and processes order, execution, and instrument updates. If the account thread is overloaded, this causes missed updates and missed order submissions because it can no longer actively listen for updates until the processor is freed up.

This is similar to an issue calling a thread sleep on an instrument thread which causes the entire platform to stop receiving data. After the thread is freed that data does not get processed because the updates were missed.

Another suspicion I have is the addon may be exceeding the order action rate for the connection type / brokerage.


An analogy would be trying to move a swimming pool amount of water through a straw. The straw representing a single logical processor that is processing updates, creating orders, and transmitting these, the water representing the amount of instructions including logic in the account event handlers as well as the background order processing and brokerage communication work. Trying to move 100,000 liters of water through a straw in less than 1 millisecond will create so much pressure the straw fails (the straw would burst, but in the analogy the processor just locks up because the L2 cache has no more free slots).
However, if you were to pace the transmission and send 1 liter of water through the straw every 5 seconds, the straw would have a much better chance of keeping up. The goal would be to come up with a procedure that works with different qualities of straw strength and different amounts of water that works well in most situations.


If the issue is with the addon, it might not be possible for our development team correct on our end.
However, without any feedback I don't have a way to confirm these suspicions.

I've created a test script which duplicates orders to multiple accounts, which I've provided to our business development team, which is what I have requested feedback on.

In my personal testing, I've stress tested the test script extremely hard with no issues on a single Rithmic simulation account and with multiple NinjaTrader demo accounts, however I do not have access to a large number of Rithmic accounts to test this myself.

I'll happy share this here if you would like to do some testing on my behalf. The insight gained could give myself and our developers more information to tackle the issue.
PaceTradeCopyAddonExample_NT8.zip


I would like this tested on as many accounts as possible. My hope is for someone with 100+ accounts could test to see if there are limits to the number of accounts and limits to the pace of the submissions.

Requested Testing:

• Add several (as many as possible) live demo (or live funded) accounts to the recipientAccounts collection on lines 127 to 131.
  • Test if there is a specific amount of accounts that can introduce issues
    • Test 1 additional account, 2 , 3 , 5 , 7, 10, 20, 50, and 100 additional accounts if possible
    • Note how long it takes for the last account to fill an order and how far from the fill price of the first account this is
• Test different delay milliseconds between an order submission
  • The timerInterval integer on line 112 holds the number of milliseconds for the timer Elapsed event
    • Test adjusting this to 1ms, 2ms, 3ms, 4ms, 5ms, 6ms, 7ms, 10ms, 15ms, 20ms, 50ms.
• Test multiple instances of the window being open and if they how they conflict
  • Test multiple instances on different accounts but the same instrument
  • Test multiple instances on the same account but different instrument
  • Test multiple instances on the same account and same instrument.
    • It would be expected the position is doubled, but we specifically want to know if the updates are being received correctly

That said, even if I don't get the feedback I need, our development is still actively investigating.

Hello Clelsea,

I would like to respectfully bring to your attention that the script in question appears to be ineffective in addressing the underlying issue. The problem does not lie in the sending of orders to the broker (Apex Rithmic). Rather, it lies in the fact that the exit orders, which have already been sent and are working in the broker, are later filled in several accounts at the same time, and the information about the filled order is received by NinjaTrader, Execution and Position information are received by the NinjaTrader. The problem occurs at this moment, regardless of how the script submits the orders to the broker. We are reporting that the position update callback from one or more accounts is either not being received by the NinjaTrader or the NinjaTrader is failing to update this information.

If you'd like to understand what's happening and identify where the problem lies, it would be helpful to contact APEX and Rithmic. You could try simulating several accounts being updated through exit orders at the same time. This is something that doesn't occur with other brokers, such as Tradovate.

This issue is approaching its one-year mark, and there appears to be no resolution in sight. As Add-on developers, we do not have direct control over the connection to the broker or the return of position updates to NinjaTrader. Therefore, any attempt to resolve this through NinjaScript may not be effective and could potentially cause further confusion among the community and our customers who are affected by this issue, potentially leading to account, financial, and time losses.

To illustrate this further, we would like to share another example of the issue that was reported to us this week, one of hundreds already reported. This example demonstrates that the issue is not being generated by any third-party add-on, but rather by the broker's failure to update to NinjaTrader or by NinjaTrade itself being unable to update the position of several accounts at the same time (Account, Order and Execution numbers were hidden).

The customer had his account APEX-000000-00 position with 5 contracts:

2025-04-08 11:44:06:528|1|64|Instrument='CL MAY25' Account='APEX-000000-00' Average price=58.980000000000004 Quantity=5 Market position=Short Operation=Operation_Add

Then the Target 1 was filled:

2025-04-08 11:44:09:158|1|32|Order='00000000001/APEX-000000-00' Name='Target1' New state='Filled' Instrument='CL MAY25' Action='Buy to cover' Limit price=58.93 Stop price=0 Quantity=5 Type='Limit' Time in force=GTC Oco='aaaaaaaaaaaaaaaaaaaaaaa' Filled=5 Fill price=58.93 Error='No error' Native error=''
2025-04-08 11:44:09:186|1|8|Execution='111111|1111111|0000000 0001' Instrument='CL MAY25' Account='APEX-000000-00' Exchange=Nymex (New York Mercantile Exchange) Price=58.93 Quantity=5 Market position=Long Operation=Operation_Add Order='00000000001' Time='04/08/2025 11:44:08'

Closing the position at the broker's side, but the NinjaTrader didn't received any position update from the broker or did not update the position for the account. Implying that there was an open position, so the customer clicked to the Close button to close the position that was outdated on the NinjaTrader.

2025-04-08 11:44:39:850|1|32|Order='00000000002/APEX-000000-00' Name='Close' New state='Filled' Instrument='CL MAY25' Action='Buy to cover' Limit price=0 Stop price=0 Quantity=5 Type='Market' Time in force=DAY Oco='' Filled=5 Fill price=58.97 Error='No error' Native error=''
2025-04-08 11:44:39:850|1|8|Execution='111111|1111111|000 00000002' Instrument='CL MAY25' Account='APEX-000000-00' Exchange=Nymex (New York Mercantile Exchange) Price=58.97 Quantity=5 Market position=Long Operation=Operation_Add Order='00000000002' Time='04/08/2025 11:44:39'

At this same moment, the position was updated to Flat (30 seconds after) then a new long position was opened:

2025-04-08 11:44:39:832|1|64|Instrument='CL MAY25' Account='APEX-000000-00' Average price=58.980000000000004 Quantity=5 Market position=Flat Operation=Remove
2025-04-08 11:44:39:836|1|64|Instrument='CL MAY25' Account='APEX-000000-00' Average price=58.97 Quantity=5 Market position=Long Operation=Operation_Add

This shows that the position update information was not received or was not processed by NinjaTrader and was only received after the user sent a Close order because he understood that there was an improperly opened position, so the position was updated late and a new one was opened.

This situation serves as a classic example of how, once again, we have shown the issue so that you can take action, and, once again, the feedback we have received has been unproductive and without any action on your part to resolve the problem. You had committed to reviewing Rithmic's logs, and we have not yet received any feedback. This suggests that perhaps the resolution of this situation is not a top priority. I would appreciate it if you could provide evidence that this is not the case.

[NinjaTrader_ChelseaB]

Hello Rudmax,

Yes, I understand the specific issue.

My suspicions are about why the update is not being processed if is actually being transmitted by Rithmic.

The thread processing the messages might be overworked, which might explain why some users have commented trade copiers from other venders are not experiencing this issue on Rithmic.

Unfortunately, I do not have multiple Rithmic accounts to test. And the goal is to collect further information by doing tests and reviewing the output.

More information helps to resolve issues.

[Rudmax]

Hello Chelsea,

Let's come back to that point and I'll make my comments shortly afterwards

However, I wanted to investigate further and reached out to the Rithmic support team to investigate any odd behavior on their end. They've reported back no issue was identified, but they are adding further RAPI logging which may help the investigation moving forward.

Have all the logs we've sent been investigated in Rithmic's logs to find out why the position update information isn't being processed by Ninjatrader? Whether this data has actually been passed on or not? If I'm not mistaken, this information was sent by you on 26 July 2024.

This is essential to make sure that the data is arriving in NinjaTrader, otherwise we'll be posting a bunch of assumptions here.

Now to the comments on your last post.

My suspicions are about why the update is not being processed if is actually being transmitted by Rithmic.

Perhaps it would be beneficial to address this uncertainty by consulting Rithmic's RAPI records for confirmation as you yourself have said in a past post. Or should we continue to discuss this matter until 2030?

The thread processing the messages might be overworked, which might explain why some users have commented trade copiers from other venders are not experiencing this issue on Rithmic.

So can we say that NinjaTrader's threads are being cooked when there are several Rithmic APEX accounts having their exit orders filled by the market being unable to process all the accounts' position updates?

If NinjaTrader receives information that the position has been updated, why isn't this logged in the logs and traces files? My understanding is that this should be done in a separate thread so as not to affect the processing of the trade, I believe this is how it is already done by NinjaTrader, but after all, there is no information in the logs about the position update at times when filling exit orders from multiple APEX Rithmic accounts. I would be very interested to know whether you believe that a third-party add-on has an influence on this by making NinjaTrader unable to write to the logs.

I say this because every third-party add-on runs in processes separate from NinjaTrader and should not at any time influence the main program, separate processes work with exclusive threads unrelated to each other. If this is not true, which I don't believe, then we have a serious architectural problem in NinjaTrader.


Let's look at the example of this copier which, because it is an add-on, unlike an indicator, runs in separate processes.

With regard to other copiers, we know that they are reporting the same problem, also people who don't even use copiers are also experiencing the same issue.

Examples:

APEX Trader Copier: https://forum.ninjatrader.com/forum/...ed-apex-copier
No Copier: https://forum.ninjatrader.com/forum/...e-not-reliable
No Copier: https://forum.ninjatrader.com/forum/...n-all-accounts
No Copier: https://forum.ninjatrader.com/forum/...position-error

To prove that your statement is incorrect, these are just a few examples. Also when using Tradovate accounts, this issue is not there, people with 40+ Tradovate accounts has never been face this, so no thread overcooking at all by using 3rd party copiers add-on.

Unfortunately, I do not have multiple Rithmic accounts to test. And the goal is to collect further information by doing tests and reviewing the output.

Can't a company with a market value of $1.5 billion get Rithmic APEX accounts to run tests on its own product to identify a flaw? So will this be overlooked so that customers continue to have their accounts destroyed and their time wasted?

We are here to pass on the contact details of Darrell, CEO of APEX, privately, so that NinjaTrader can request the accounts, I'm sure he will be happy to assist you with 20 accounts for testing. But you have to want it.

I believe that the time has come to escalate this issue to higher levels, with all due respect to you and all the support you have provided, but the time has come to have this resolved, the community is waiting for a firm and quick solution from those involved. Maybe NinjaTrader_Erick (Erick P.) can help us.

[NinjaTrader_ChelseaB]

Hello Rudmax,

Thank you for your feedback.

I will pass this along to the development team.