NinjaTrader + Rithmic = Not updating positions

[NinjaTrader_ChelseaB]

Hello Rudmax,

Thank you for your feedback.

I will pass this along to the development team.

[Rudmax]

Hello Chelsea,

Let's come back to that point and I'll make my comments shortly afterwards

However, I wanted to investigate further and reached out to the Rithmic support team to investigate any odd behavior on their end. They've reported back no issue was identified, but they are adding further RAPI logging which may help the investigation moving forward.

Have all the logs we've sent been investigated in Rithmic's logs to find out why the position update information isn't being processed by Ninjatrader? Whether this data has actually been passed on or not? If I'm not mistaken, this information was sent by you on 26 July 2024.

This is essential to make sure that the data is arriving in NinjaTrader, otherwise we'll be posting a bunch of assumptions here.

Now to the comments on your last post.

My suspicions are about why the update is not being processed if is actually being transmitted by Rithmic.

Perhaps it would be beneficial to address this uncertainty by consulting Rithmic's RAPI records for confirmation as you yourself have said in a past post. Or should we continue to discuss this matter until 2030?

The thread processing the messages might be overworked, which might explain why some users have commented trade copiers from other venders are not experiencing this issue on Rithmic.

So can we say that NinjaTrader's threads are being cooked when there are several Rithmic APEX accounts having their exit orders filled by the market being unable to process all the accounts' position updates?

If NinjaTrader receives information that the position has been updated, why isn't this logged in the logs and traces files? My understanding is that this should be done in a separate thread so as not to affect the processing of the trade, I believe this is how it is already done by NinjaTrader, but after all, there is no information in the logs about the position update at times when filling exit orders from multiple APEX Rithmic accounts. I would be very interested to know whether you believe that a third-party add-on has an influence on this by making NinjaTrader unable to write to the logs.

I say this because every third-party add-on runs in processes separate from NinjaTrader and should not at any time influence the main program, separate processes work with exclusive threads unrelated to each other. If this is not true, which I don't believe, then we have a serious architectural problem in NinjaTrader.


Let's look at the example of this copier which, because it is an add-on, unlike an indicator, runs in separate processes.

With regard to other copiers, we know that they are reporting the same problem, also people who don't even use copiers are also experiencing the same issue.

Examples:

APEX Trader Copier: https://forum.ninjatrader.com/forum/...ed-apex-copier
No Copier: https://forum.ninjatrader.com/forum/...e-not-reliable
No Copier: https://forum.ninjatrader.com/forum/...n-all-accounts
No Copier: https://forum.ninjatrader.com/forum/...position-error

To prove that your statement is incorrect, these are just a few examples. Also when using Tradovate accounts, this issue is not there, people with 40+ Tradovate accounts has never been face this, so no thread overcooking at all by using 3rd party copiers add-on.

Unfortunately, I do not have multiple Rithmic accounts to test. And the goal is to collect further information by doing tests and reviewing the output.

Can't a company with a market value of $1.5 billion get Rithmic APEX accounts to run tests on its own product to identify a flaw? So will this be overlooked so that customers continue to have their accounts destroyed and their time wasted?

We are here to pass on the contact details of Darrell, CEO of APEX, privately, so that NinjaTrader can request the accounts, I'm sure he will be happy to assist you with 20 accounts for testing. But you have to want it.

I believe that the time has come to escalate this issue to higher levels, with all due respect to you and all the support you have provided, but the time has come to have this resolved, the community is waiting for a firm and quick solution from those involved. Maybe NinjaTrader_Erick (Erick P.) can help us.

[NinjaTrader_ChelseaB]

Hello Rudmax,

Yes, I understand the specific issue.

My suspicions are about why the update is not being processed if is actually being transmitted by Rithmic.

The thread processing the messages might be overworked, which might explain why some users have commented trade copiers from other venders are not experiencing this issue on Rithmic.

Unfortunately, I do not have multiple Rithmic accounts to test. And the goal is to collect further information by doing tests and reviewing the output.

More information helps to resolve issues.

[Rudmax]

Hello ntcoder256,

Our developers are looking into the issue and the case is still open with them.

As there are 3 parties involved, it can be challenging to precisely identify the specific cause.

I have my own suspicions that the issue comes from addon code. I've requested that our business development reach out to Apex to assist me in investigating but unfortunately I have not received any information back.

See post # 78 where I've discussed my suspicions.

The gist is that the Account thread is the thread in NinjaTrader that creates and sends orders, and processes order, execution, and instrument updates. If the account thread is overloaded, this causes missed updates and missed order submissions because it can no longer actively listen for updates until the processor is freed up.

This is similar to an issue calling a thread sleep on an instrument thread which causes the entire platform to stop receiving data. After the thread is freed that data does not get processed because the updates were missed.

Another suspicion I have is the addon may be exceeding the order action rate for the connection type / brokerage.


An analogy would be trying to move a swimming pool amount of water through a straw. The straw representing a single logical processor that is processing updates, creating orders, and transmitting these, the water representing the amount of instructions including logic in the account event handlers as well as the background order processing and brokerage communication work. Trying to move 100,000 liters of water through a straw in less than 1 millisecond will create so much pressure the straw fails (the straw would burst, but in the analogy the processor just locks up because the L2 cache has no more free slots).
However, if you were to pace the transmission and send 1 liter of water through the straw every 5 seconds, the straw would have a much better chance of keeping up. The goal would be to come up with a procedure that works with different qualities of straw strength and different amounts of water that works well in most situations.


If the issue is with the addon, it might not be possible for our development team correct on our end.
However, without any feedback I don't have a way to confirm these suspicions.

I've created a test script which duplicates orders to multiple accounts, which I've provided to our business development team, which is what I have requested feedback on.

In my personal testing, I've stress tested the test script extremely hard with no issues on a single Rithmic simulation account and with multiple NinjaTrader demo accounts, however I do not have access to a large number of Rithmic accounts to test this myself.

I'll happy share this here if you would like to do some testing on my behalf. The insight gained could give myself and our developers more information to tackle the issue.
PaceTradeCopyAddonExample_NT8.zip


I would like this tested on as many accounts as possible. My hope is for someone with 100+ accounts could test to see if there are limits to the number of accounts and limits to the pace of the submissions.

Requested Testing:

• Add several (as many as possible) live demo (or live funded) accounts to the recipientAccounts collection on lines 127 to 131.
  • Test if there is a specific amount of accounts that can introduce issues
    • Test 1 additional account, 2 , 3 , 5 , 7, 10, 20, 50, and 100 additional accounts if possible
    • Note how long it takes for the last account to fill an order and how far from the fill price of the first account this is
• Test different delay milliseconds between an order submission
  • The timerInterval integer on line 112 holds the number of milliseconds for the timer Elapsed event
    • Test adjusting this to 1ms, 2ms, 3ms, 4ms, 5ms, 6ms, 7ms, 10ms, 15ms, 20ms, 50ms.
• Test multiple instances of the window being open and if they how they conflict
  • Test multiple instances on different accounts but the same instrument
  • Test multiple instances on the same account but different instrument
  • Test multiple instances on the same account and same instrument.
    • It would be expected the position is doubled, but we specifically want to know if the updates are being received correctly

That said, even if I don't get the feedback I need, our development is still actively investigating.

Hello Clelsea,

I would like to respectfully bring to your attention that the script in question appears to be ineffective in addressing the underlying issue. The problem does not lie in the sending of orders to the broker (Apex Rithmic). Rather, it lies in the fact that the exit orders, which have already been sent and are working in the broker, are later filled in several accounts at the same time, and the information about the filled order is received by NinjaTrader, Execution and Position information are received by the NinjaTrader. The problem occurs at this moment, regardless of how the script submits the orders to the broker. We are reporting that the position update callback from one or more accounts is either not being received by the NinjaTrader or the NinjaTrader is failing to update this information.

If you'd like to understand what's happening and identify where the problem lies, it would be helpful to contact APEX and Rithmic. You could try simulating several accounts being updated through exit orders at the same time. This is something that doesn't occur with other brokers, such as Tradovate.

This issue is approaching its one-year mark, and there appears to be no resolution in sight. As Add-on developers, we do not have direct control over the connection to the broker or the return of position updates to NinjaTrader. Therefore, any attempt to resolve this through NinjaScript may not be effective and could potentially cause further confusion among the community and our customers who are affected by this issue, potentially leading to account, financial, and time losses.

To illustrate this further, we would like to share another example of the issue that was reported to us this week, one of hundreds already reported. This example demonstrates that the issue is not being generated by any third-party add-on, but rather by the broker's failure to update to NinjaTrader or by NinjaTrade itself being unable to update the position of several accounts at the same time (Account, Order and Execution numbers were hidden).

The customer had his account APEX-000000-00 position with 5 contracts:

2025-04-08 11:44:06:528|1|64|Instrument='CL MAY25' Account='APEX-000000-00' Average price=58.980000000000004 Quantity=5 Market position=Short Operation=Operation_Add

Then the Target 1 was filled:

2025-04-08 11:44:09:158|1|32|Order='00000000001/APEX-000000-00' Name='Target1' New state='Filled' Instrument='CL MAY25' Action='Buy to cover' Limit price=58.93 Stop price=0 Quantity=5 Type='Limit' Time in force=GTC Oco='aaaaaaaaaaaaaaaaaaaaaaa' Filled=5 Fill price=58.93 Error='No error' Native error=''
2025-04-08 11:44:09:186|1|8|Execution='111111|1111111|0000000 0001' Instrument='CL MAY25' Account='APEX-000000-00' Exchange=Nymex (New York Mercantile Exchange) Price=58.93 Quantity=5 Market position=Long Operation=Operation_Add Order='00000000001' Time='04/08/2025 11:44:08'

Closing the position at the broker's side, but the NinjaTrader didn't received any position update from the broker or did not update the position for the account. Implying that there was an open position, so the customer clicked to the Close button to close the position that was outdated on the NinjaTrader.

2025-04-08 11:44:39:850|1|32|Order='00000000002/APEX-000000-00' Name='Close' New state='Filled' Instrument='CL MAY25' Action='Buy to cover' Limit price=0 Stop price=0 Quantity=5 Type='Market' Time in force=DAY Oco='' Filled=5 Fill price=58.97 Error='No error' Native error=''
2025-04-08 11:44:39:850|1|8|Execution='111111|1111111|000 00000002' Instrument='CL MAY25' Account='APEX-000000-00' Exchange=Nymex (New York Mercantile Exchange) Price=58.97 Quantity=5 Market position=Long Operation=Operation_Add Order='00000000002' Time='04/08/2025 11:44:39'

At this same moment, the position was updated to Flat (30 seconds after) then a new long position was opened:

2025-04-08 11:44:39:832|1|64|Instrument='CL MAY25' Account='APEX-000000-00' Average price=58.980000000000004 Quantity=5 Market position=Flat Operation=Remove
2025-04-08 11:44:39:836|1|64|Instrument='CL MAY25' Account='APEX-000000-00' Average price=58.97 Quantity=5 Market position=Long Operation=Operation_Add

This shows that the position update information was not received or was not processed by NinjaTrader and was only received after the user sent a Close order because he understood that there was an improperly opened position, so the position was updated late and a new one was opened.

This situation serves as a classic example of how, once again, we have shown the issue so that you can take action, and, once again, the feedback we have received has been unproductive and without any action on your part to resolve the problem. You had committed to reviewing Rithmic's logs, and we have not yet received any feedback. This suggests that perhaps the resolution of this situation is not a top priority. I would appreciate it if you could provide evidence that this is not the case.

[ntcoder256]

Hello ntcoder256,

As you are willing to test things for me, I have additional questions that may be sensitive.

Would you be willing to have a private conversation with me through email?

If so, please send an email to scriptingsupport [at] ninjatrader.com.
In the email please write Attn: Chelsea #06020609

absolutely, will do. thanks

[ntcoder256]

Hello ntcoder256,

May I confirm you are using NinjaTrader 8.1.4.2?

Correct I am currently on 8.1.4.2

[NinjaTrader_ChelseaB]

Hello ntcoder256,

As you are willing to test things for me, I have additional questions that may be sensitive.

Would you be willing to have a private conversation with me through email?

If so, please send an email to scriptingsupport [at] ninjatrader.com.
In the email please write Attn: Chelsea #06020609

[NinjaTrader_ChelseaB]

Hello ntcoder256,

May I confirm you are using NinjaTrader 8.1.4.2?

[NinjaTrader_ChelseaB]

Hello ntcoder256,

"It may be a little hard to test some of the live cases because this does not copy ATMs"

The script was written to duplicate ALL order submissions, including those from ATMs. It does not replicate order modifications or cancellations because it's only meant to be a concept test and not full trade copier addon for use, but stop and target order submissions from ATMs should be duplicated.

"and frankly if this test code is more reliable to get me out of positions than the commercial ones I'm using I'll just switch to static profit targets."

If the test succeeds then we know a little more about the issue and any one of us (the platform, connection technology, or brokerage) may be able to make improvements with the knowledge gained so the entire industry benefits because the issue is mitigated for all of our mutual users.

"One last note you mentioned you hadn't heard from Apex, if you haven't already you might reach out to FlowBots who makes Replikanto"

I will relay that to our business development team who is the bureaucratic interface between my team and the wider ecosystem team of vendors.

[ntcoder256]

Thank you SO much for sharing this and working on this. I will absolutely go through your test cases as best I can and provide feedback, I am testing it already now.

It may be a little hard to test some of the live cases because this does not copy ATMs (I'm not expecting it to I get you're just trouble shooting and this is minimal test code) and I rely on a runner and trailing stop every trade. However I will still try some live testing with static profit targets, and frankly if this test code is more reliable to get me out of positions than the commercial ones I'm using I'll just switch to static profit targets.

One last note you mentioned you hadn't heard from Apex, if you haven't already you might reach out to FlowBots who makes Replikanto

https://flowbots.ninja/contact

They're actually the ones who referred me to this thread, after I contacted them regarding this issue. I've found their support very fast and responsive and they really seem to care about their products. Also I'm not sure if it's true but I've heard the Apex copier is made by FlowBots.

Best regards

[NinjaTrader_ChelseaB]

Hello ntcoder256,

Our developers are looking into the issue and the case is still open with them.

As there are 3 parties involved, it can be challenging to precisely identify the specific cause.

I have my own suspicions that the issue comes from addon code. I've requested that our business development reach out to Apex to assist me in investigating but unfortunately I have not received any information back.

See post # 78 where I've discussed my suspicions.

The gist is that the Account thread is the thread in NinjaTrader that creates and sends orders, and processes order, execution, and instrument updates. If the account thread is overloaded, this causes missed updates and missed order submissions because it can no longer actively listen for updates until the processor is freed up.

This is similar to an issue calling a thread sleep on an instrument thread which causes the entire platform to stop receiving data. After the thread is freed that data does not get processed because the updates were missed.

Another suspicion I have is the addon may be exceeding the order action rate for the connection type / brokerage.


An analogy would be trying to move a swimming pool amount of water through a straw. The straw representing a single logical processor that is processing updates, creating orders, and transmitting these, the water representing the amount of instructions including logic in the account event handlers as well as the background order processing and brokerage communication work. Trying to move 100,000 liters of water through a straw in less than 1 millisecond will create so much pressure the straw fails (the straw would burst, but in the analogy the processor just locks up because the L2 cache has no more free slots).
However, if you were to pace the transmission and send 1 liter of water through the straw every 5 seconds, the straw would have a much better chance of keeping up. The goal would be to come up with a procedure that works with different qualities of straw strength and different amounts of water that works well in most situations.


If the issue is with the addon, it might not be possible for our development team correct on our end.
However, without any feedback I don't have a way to confirm these suspicions.

I've created a test script which duplicates orders to multiple accounts, which I've provided to our business development team, which is what I have requested feedback on.

In my personal testing, I've stress tested the test script extremely hard with no issues on a single Rithmic simulation account and with multiple NinjaTrader demo accounts, however I do not have access to a large number of Rithmic accounts to test this myself.

I'll happy share this here if you would like to do some testing on my behalf. The insight gained could give myself and our developers more information to tackle the issue.
PaceTradeCopyAddonExample_NT8.zip


I would like this tested on as many accounts as possible. My hope is for someone with 100+ accounts could test to see if there are limits to the number of accounts and limits to the pace of the submissions.

Requested Testing:

• Add several (as many as possible) live demo (or live funded) accounts to the recipientAccounts collection on lines 127 to 131.
  • Test if there is a specific amount of accounts that can introduce issues
    • Test 1 additional account, 2 , 3 , 5 , 7, 10, 20, 50, and 100 additional accounts if possible
    • Note how long it takes for the last account to fill an order and how far from the fill price of the first account this is
• Test different delay milliseconds between an order submission
  • The timerInterval integer on line 112 holds the number of milliseconds for the timer Elapsed event
    • Test adjusting this to 1ms, 2ms, 3ms, 4ms, 5ms, 6ms, 7ms, 10ms, 15ms, 20ms, 50ms.
• Test multiple instances of the window being open and if they how they conflict
  • Test multiple instances on different accounts but the same instrument
  • Test multiple instances on the same account but different instrument
  • Test multiple instances on the same account and same instrument.
    • It would be expected the position is doubled, but we specifically want to know if the updates are being received correctly

That said, even if I don't get the feedback I need, our development is still actively investigating.

[ntcoder256]

Hello Rudmax,

Our developers are looking into the issue.

Unfortunately, I will not be able to say if a correction will be ready by the next release.

Once the release is out, we'll check the release notes, and if listed we'll have users test for the behavior.

Hi, was this ever addressed in any updates? I'm am on the latest version and still see it. I'm using Apex, with both Replikanto and Apex trade copiers. Happens very consistently, I've lost several accounts from this.

It seems like there's a bit of a circular firing squad situation going on with the customer stuck in the middle.

I don't like Rithmic, I sometimes have problems closing positions left open even via RTrader and there's no resolution for that either.

On the other hand Ninja Trader is a great company, I would hope maybe Ninja Trader could be the bigger party and "fix" it even if it's really something Rithmic should do...because there's no outreach to them and I think they could care less.

please consider it.

thanks

[dragutin]

I do not recommend every one to trade with trade copiers.
i had this week same problems like in this thread explained from many users. Ninja trader is not programmed for multiple accounts execution ect …..
Same with rithmics i was up 500 doller went short was not able to close position with Replicanto than i opend rithmic pro and closed the position and i was down 200 total Although market never went against me horrible.

[defa0009]

still waiting for this issue to be fixed by the NinjaTrader.

I don't see that issue in NinjaTrader when using the Lumens Trade Copier for NinjaTrader.

[Rudmax]

I don't think it's an APEX Rithmic issue because some traders in our group use Sierra Charts or Quantower and don't have problems with positions still showing as open.

Another clue is that I have three different copy trader copiers—Apex, Replikanto, and ADTS Smart Trade Copier—and the same issue appears regardless of the copier. So, it would seem that all roads point back to a Rithmic-Ninjatrader interface issue.

Thanks for your feedback!

Yes, we understand it's a problem between NinjaTrader and Rithmic-type accounts, but not necessarily a problem with Rithmic but with NinjaTrader itself when it comes to updating the positions of several accounts at the same time.

Unfortunately, we are still waiting for this issue to be fixed by the NinjaTrader.