Latest Version Compile Takes too long

[jeronymite]

One contributing factor I discovered in previous versions back as far as 8.0.27.0 was that when editing/compiling in the NinjaScript Editor (not using VS), if one has multiple Editor windows open, the duration of the compile seems to take proportionally longer as the number of editor windows open. By closing editor windows, I was able to significantly reduce compile times.

Having said that, it may have no relevance to what others are observing in 8.1.2.0. I have not tested compile times much on 8.1.2.0, although for a single editor window it does seem to take just a bit longer than expected, although not unreasonably long.

It's worth noting too that it seems NinjaTrader has gone further than anticipated in 8.1.2.0 and upgraded the embedded Actipro Syntax Editor! It's now at V23.1.0.0 which is the current major version. Well done, NinjaTrader! ... ... Keep on going though: there's still Infragistics, SharpDX, etc to work on now.

Thanks.

[QuantKey_Bruce]

I think it's more like an O(n^2) on the length of the codebase. I don't know the details yet but just anecdotally, it wasn't very slow with a default install but copy in a bunch of code from 8.1.1.7 and it takes way longer (not just longer than without the code - longer than with the code on 8.1.1.7 by a factor of maybe 10x or more for a large codebase).

[kevinenergy]

Are you able to consistently reproduce the steps leading to the compiler loading seemingly indefinitely? If so, what steps are you taking to trigger this longer load time? I would like to test and see if I am able to get the same reproducible steps on my end.

Hey Emily,

The behaviour I saw was when copying over a large number of files at once. You can replicate this by doing the following:

- Open Ninjascript editor.
- Right click on Indicator Folder
- Choose "new indicator", and Hit "Generate"
- Repeat this until you've got lots of indicators (in my case it was 900 or so individual code files but try it with 50-100 and see how it goes).

When you've got all these indicators save them and then go into Documents\NinjaTrader 8\bin\Custom\Indicators and move all the new indicators out of there and into a folder somewhere else.

Then copy them all back into Documents\NinjaTrader 8\bin\Custom\Indicators together. This will simulate what I did.

The Ninjatscript editor has a FileSystemWatcher which is constantly scanning for changes to its folder. When it picks up changes such as files being copied over it triggers a build.

When multiple files are copied in together it seems to get backed-up.

As I mentioned, I don't see any performance degradation for my regular builds, only when copying over a large number of files at once.

[NinjaTrader_Emily]

Hello kevinenergy,

Thank you for your reply.

Manually copying over scripts in this way is not the documented and supported way to move scripts in/out of the NinjaScript Editor. You would need to either use the Export/Import functions from the Control Center > Tools menu or utilize the Backup/Restore function to backup NinjaScript files and then later restore them.

• https://ninjatrader.com/support/help...nt8/export.htm
• https://ninjatrader.com/support/help...nt8/import.htm
• https://ninjatrader.com/support/help...up_archive.htm
• https://ninjatrader.com/support/help...up_archive.htm

As for deleting/removing NinjaScript files from an installation, you should either utilize the Remove NinjaScript Assembly function from the Tools menu or remove files in the NinjaScript Editor from the right-click menu. Manually moving files in a Windows File Explorer could have unexpected results, especially moving a large number of files at the same time while NinjaTrader is still open. These other functions are built into the platform and if you were experiencing this behavior from one of those methods we would be able to look into it further. Otherwise, it is unsupported and not documented in the help guide so you should only do so at your own risk.

Thank you for your time and patience.

[QuantKey_Bruce]

NinjaTrader_Emily Clearly what is happening here seems to be that NinjaTrader is queueing up a request to recompile every time the FileSystemWatcher detects a change. What should happen, and this seems pretty clear, is a flag should be set "needs recompile" such that as soon as the compiler is done compiling, if the flag is set, it clears the flag, then starts a recompile. What is happening instead it seems is that if you do something like save a file change 100 times it attempts to recompile 100 times sequentially which is a poor solution, and not the good implementation we had in 8.1.1.7. Why was this change made? This seems like it must be unintentional, and deflecting that it's "unsupported" to copy in files is missing the fact that development from Visual Studio is the way much work gets done, and all it takes is to change a bunch of files in Visual Studio then click the little "save" button on the toolbar and if you changed ten files, NT will now apparently try to compile ten times, which is clearly, indefensibly wrong. And, to top it all off, it didn't do this in 8.1.1.7 soi it's clearly not necessary to do it this way. I'm grateful for all the improvements in 8.1.2.0 - but when something goes the wrong direction amidst other changes that are clearly beneficial it stands out only the more starkly like this one which seems like it was probably an accident and just needs to be cleaned up.

[ntbone]

I have tested out 8.1.2 and have observed the same behaviors. If you edit only one file, it compiles fine, but the more files you edit, the longer it takes, drastically.

Reproduction Steps:

1. Open the NinjaTrader workspace in Visual Studio 2022
2. Open 10 or 20 files. Add a simple white space. Do not save any file until all files have been modified.
3. Save all

If you watch the bin\Custom folder and look at NinjaTrader.Custom.dll you can see the time stamps update as it builds it over and over and over again.

The hypothesis that was proposed regarding the file watcher and NinjaTrader rebuilding seems spot on.

Regardless of the supported method for adding/removing files, editing files as above should be a supported scenario. Development in Visual Studio is important for both debugging purposes and provides far more features then the editor provided with NinjaTrader.

Further

• The only time NinjaTrader automatically rebuilds is when the NinjaTrader editor is open. At this point any changes made to any existing file, any file deletion, or file addition to the bin folder results in NinjaTrader automatically rebuilding.
• If you use the editor to edit files, it will only rebuild if you manually rebuild. Specifically, if you modify the files from within their own editor, and save it will not rebuild. You have to manually initiate compiling.

Based on these two points, the file watching ability is specifically present for editing from external applications using either Visual Studio, or your favorite text editor. In versions prior to 8.1.2 NinjaTrader was smarter in how it handled file watching and the upgrade has clearly introduced a performance bug that results in it building over and over again for each detected file change.

A workaround is to

1. Close the editor until you are ready to build.
2. Save all changes, copy all files, delete all files as necessary
3. When ready to build, open the editor, manually initiate a build. Close the editor when complete.

This is a very cumbersome workaround. If you get compiling errors, you will need to fix them all, without saving any of them, then go back to step 1 through 3 above.

At this point it would be better if NinjaTrader disabled the file watching of the folder allowing us to just manually build using the editor, as if we were using the editor to build manually.

The ideal fix would restore behavior of NinjaTrader prior to 8.1.2. If its already in the process of building, it should never queue up more then one additional build when other file changes are detected.

[jeronymite]

I agree with the excellent analysis of ntbone and QuantKey_Bruce above. Based on my own experience with extended compile times proportional to number of NinjaScript Editor windows, I would venture to say a similar behaviour is occurring there too, and that should be part of the investigation/rectification.

Thanks.

[ntbone]

Based on my own experience with extended compile times proportional to number of NinjaScript Editor windows, I would venture to say a similar behaviour is occurring there too

Thanks.

Are you saying that if you have more then one editor window open, it also extends the compile time?

[jeronymite]

That was certainly the behaviour in prior versions, and as far as I can tell, it seems to be the same in 8.1.2.0. The difference between one editor window (not just tab) open and 20 open is significant. A test of this by repeatedly compiling and timing produced this [# windows | 5x test results (secs) | average]:

• 20 | 45, 16, 25, 22, 23 | 26
• 10 | 14, 17, 15, 18, 18 | 16
• 2 | 13, 15, 13, 15, 14 | 14

Clearly, there is some correlation with number of editor windows. The very first test produced 45 secs, noticeably more than any other, which may indicate a first compile after platform startup overhead?

Anyway, it does show the issue is also seen with increasing number of editor windows. For reference, it's compiling 41 source files with a total line count of 135K.

Thanks.

[NinjaTrader_Emily]

I have tested out 8.1.2 and have observed the same behaviors. If you edit only one file, it compiles fine, but the more files you edit, the longer it takes, drastically.

Reproduction Steps:

1. Open the NinjaTrader workspace in Visual Studio 2022
2. Open 10 or 20 files. Add a simple white space. Do not save any file until all files have been modified.
3. Save all

If you watch the bin\Custom folder and look at NinjaTrader.Custom.dll you can see the time stamps update as it builds it over and over and over again.

The hypothesis that was proposed regarding the file watcher and NinjaTrader rebuilding seems spot on.

Regardless of the supported method for adding/removing files, editing files as above should be a supported scenario. Development in Visual Studio is important for both debugging purposes and provides far more features then the editor provided with NinjaTrader.

Further

• The only time NinjaTrader automatically rebuilds is when the NinjaTrader editor is open. At this point any changes made to any existing file, any file deletion, or file addition to the bin folder results in NinjaTrader automatically rebuilding.
• If you use the editor to edit files, it will only rebuild if you manually rebuild. Specifically, if you modify the files from within their own editor, and save it will not rebuild. You have to manually initiate compiling.

Based on these two points, the file watching ability is specifically present for editing from external applications using either Visual Studio, or your favorite text editor. In versions prior to 8.1.2 NinjaTrader was smarter in how it handled file watching and the upgrade has clearly introduced a performance bug that results in it building over and over again for each detected file change.

A workaround is to

1. Close the editor until you are ready to build.
2. Save all changes, copy all files, delete all files as necessary
3. When ready to build, open the editor, manually initiate a build. Close the editor when complete.

This is a very cumbersome workaround. If you get compiling errors, you will need to fix them all, without saving any of them, then go back to step 1 through 3 above.

At this point it would be better if NinjaTrader disabled the file watching of the folder allowing us to just manually build using the editor, as if we were using the editor to build manually.

The ideal fix would restore behavior of NinjaTrader prior to 8.1.2. If its already in the process of building, it should never queue up more then one additional build when other file changes are detected.

Hello ntbone,

Thank you for your response.

I was unable to reproduce this with the steps provided. As shown in the video below, I edited 20+ files and then saved them all. I had the NinjaScript Editor window opened and the compile was successful within seconds:

https://www.screencast.com/t/lMfW5CKbY

Although we have been unable to reproduce this behavior so far, we do understand concerns with performance-related items and this has been reported to the development team. The issue number is NT-10898; please refer to this issue number when contacting support regarding this item. It is expected that the Roslyn compilations (the latest to version of NinjaTrader 8.1.2.0 did change to the Roslyn compiler) will take longer than the previous compiler though we have not been able to see it take several minutes in our testing. Please be sure that you do not have debug mode enabled (right-click the NinjaScript Editor and be sure there is no checkmark next to Debug Mode) when trying to test this item. In general, there have been scenarios where having a lot of assemblies results in slower compile times. Some scripts take longer to compile than others and the amount of scripts also likely has an impact on compile times. Visual Studio extensions, if present, could also be potentially related to this behavior.

Since the support team does not have the same custom files that you may have in your installation, testing in a fresh installation of NinjaTrader as well as a fresh installation of Visual Studio 2022 without any extensions would be ideal. If you are able to reproduce with a single, simple test script that is mostly empty, that would be helpful and informative. Otherwise, I will gladly update this thread if I receive any meaningful updates regarding issue NT-10898 to share.

Thank you for your time and patience.

[NinjaTrader_Emily]

That was certainly the behaviour in prior versions, and as far as I can tell, it seems to be the same in 8.1.2.0. The difference between one editor window (not just tab) open and 20 open is significant. A test of this by repeatedly compiling and timing produced this [# windows | 5x test results (secs) | average]:

• 20 | 45, 16, 25, 22, 23 | 26
• 10 | 14, 17, 15, 18, 18 | 16
• 2 | 13, 15, 13, 15, 14 | 14

Clearly, there is some correlation with number of editor windows. The very first test produced 45 secs, noticeably more than any other, which may indicate a first compile after platform startup overhead?

Anyway, it does show the issue is also seen with increasing number of editor windows. For reference, it's compiling 41 source files with a total line count of 135K.

Thanks.

Hello jeronymite,

Thank you for your note.

I was unable to reproduce this behavior. I opened 15 NinjaScript Editor windows, made a change to an indicator, and compiled. I then closed 14 of the windows, changed the file again, and compiled. Both compiles took the same amount of time; there was no noticeable or proportionate difference in the length of time to compile. Do you also have Visual Studio open while performing this test or is this strictly related to having NinjaScript Editor windows open and editing/compiling in the NinjaScript Editor? Please provide the exact steps you are following to produce these results so I may perform additional tests.

I look forward to your reply.

[jeronymite]

NinjaTrader_Emily Thanks for testing. The procedure is simple: Open 20 separate NinjaScript Editor windows with one tab per window, each containing a different file. Ensure a total file count of at least 40 files to compile, with a total line count of at least 135,000 lines. Compile. Make no changes at any time.Compile.Compile.Compile.Compile. ... Close 10 windows. Repeat. Close 8 windows. Repeat.

Visual Studio is not involved in any way.

Thanks.

[NinjaTrader_Emily]

NinjaTrader_Emily Thanks for testing. The procedure is simple: Open 20 separate NinjaScript Editor windows with one tab per window, each containing a different file. Ensure a total file count of at least 40 files to compile, with a total line count of at least 135,000 lines. Compile. Make no changes at any time.Compile.Compile.Compile.Compile. ... Close 10 windows. Repeat. Close 8 windows. Repeat.

Visual Studio is not involved in any way.

Thanks.

I was unable to reproduce with the steps you provided. I opened over 20 NinjaScript Editor windows, each with a different indicator or strategy opened in it. I compiled with all of the windows open, made no changes and closed all but one window, then compiled again. The time it took to compile was approximately the same with no noticeable difference. Here is a video of the test:

https://www.screencast.com/t/jxnblTuyC

You could try a fresh installation of NinjaTrader and introduce more scripts (both source code and compiled assemblies) and test the length to compile as more scripts are added, or test with different scripts being open in the editor windows, to see at what point you start to notice the difference in the length of time to compile. It could also be a question of PC resources that are available, though the steps you provided to reproduce don't seem to work consistently across the board for anyone using NinjaTrader and are specific to something in your environment causing the slower compile time.

Thank you for your patience.

[jeronymite]

NinjaTrader_Emily How many files were compiled? How many lines of code in total?

Having been a professional Software Engineer for more than four decades, I'm aware of system performance matters. There are none.

Thanks.

[Conceptzx]

I'm experiencing freezing of charts while compiling the indicator I'm working on when it's loaded. It's been compiling for about 8 minutes now, charts are frozen but the bid/ask and DOM are working fine.



Edited, 50 minutes and it still hasn't compiled. Just pretends it's doing something.