Slow startup

[lorenzosu]

Describe the bug
When starting Muse 4.1 on Linux it takes very long to actually start the programme which 'hangs' at the splach screen. On my relatively 'modern' machine it taskes from around 71 to 80 seconds (as reported by the Total start-up time message when loading from the terminal). I tested this about 5-6 times.

To Reproduce
Steps to reproduce the behavior:

1. Start Muse
2. The splash-screen reports creating cache and initialising plugins
3. Something to note is that because the muse window is not initialised, if I focus on some other window the splash-screen 'disappears' (I am on KDE), so I might think that something has gone wrong with the loading

Expected behavior

• Faster loading / cache management
• Better UI feedback to the user that the programme is actually working but something (loading plugins) is happening (e.g. not use the slpashscreen for that)

Desktop (please complete the following information):

• OS: Manjaro
• Desktop: KDE 5.27.7
• QT: 5.15.10

Additional context

• Muse 4.2.0 AppImage version download from releases

Running from the command line there seems to be quite a bottleneck when this appears:
Initializing Native VST support. Using VESTIGE compatibility implementation.

I do indeed have quite a few (a lot of) plugins (especially LV2) but as a benchmark these are loading times / plug-in cache creation for other daws on the same systems (so same plug-ins installed):

• Qtractor: load time is almost instant as it doesn't perform plugin scanning on start-up. At first scan or an on-demand scan (which can be triggered) it takes between 21 and 23 seconds
• Ardour: load time after creating a new session (needed on start-up) and clicking open: circa 10 seconds. A discover updated (with the 'Autoskip unresponsive plugins' option enabled) which essentially does a scan took about 65 seconds (I did not test the full rescan as I don't want to reset the cache and data)
• Carla plugin 'refresh' (i.e. scan) for LADSPA, DSSI, LV2, VST2 and VST3 circa 32 seconds. It took circa 43 seconds when enabling the 'processing' checkbox which should do some processing to see if the plug-in is prone to crash
• Reaper (unregistered version). Circe 2 seconds to start-up to buy me screen. For full re-scan Reaper separates VSTs and LV2/CLAP. VST rescan took circa 8 seconds, LV2 and CLAP plugins felt almost instant (maybe 2-3 seconds each)

[terminator356]

Hello. Try isolating which plugin support is slowing it down.
Start muse with -N (Don't load LinuxVST plugins). Test loading time.
Then start muse with -2 (Don't load LV2 plugins). Test loading time.

Also, try adding -D (debug) to the commands to get more info about where it might be slowing.
It should tell you about all the plugins that it is scanning or loading.
See if you can pinpoint where it is slowing down.

Another thing that can be tried is this option, to see if the cache is constantly being re-created each time:
-C (Do not re-create plugin cache. Avoids repeated re-creations in some circumstances. Use with care.)

For LADSPA and LinuxVST, muse creates a cache the very first time which can be slow.
But after that, the loading should be fast since it refers to the cache.
The purpose of this scanning and cache is to ensure that plugins do not crash simply by loading them,
and to extract information from them which requires actually loading them to find out.
The cache will be re-scanned whenever changes are detected in the plugin paths given in Global Settings,
such as new plugins installed or plugins removed etc.
Try to ensure that the given plugin paths do not point to folders that are constantly changing,
which might re-trigger a cache scan and slow things down each time muse is started.

For LV2, no cache is used at all. An LV2 cache is not necessary since all the plugin info is stored in TTL files.
So if you have a lot of LV2 plugins, it might be slow every time starting.
Kybos (muse dev) reported very slow loading times due to his LV2. You might have the same issue.
I could not reproduce that here.
There wasn't much that I could see to do about that, since we need to read every LV2 plugin's information.

[lorenzosu]

Hi Tim,
Thanks for your feedback.

Some quick tests I did (same set-up no changes in plugins).

-N takes load time to around 40 seconds

-2 load time is around 53 seconds

Using both of those together (unsurprisingly) takes about 4 seconds. But then, of course, only DSSI plugins are available.

-C load time is around 28 seconds

In this last case I still see lots of lilv_world_add_plugin() messages when running on the terminal

Something else I noticed running in the terminal is that muse was complaining about duplicate plugins found in both /usr/lib64/lv2/ and /usr/lib/lv2/. Indeed here on Manjaro the former is just a symlink to the latter. I tried removing one entry in the LV2 search path but start time doesn't change
The same goes for VST paths. Although it seems removing one of the 'duplicate' /usr/lib entries for VST brouth load time down about 15 seconds.
This pattern of duplicate plugins being ignored (but, I assume possibly being scanned), seems similar for all plugin types.

So I tried removing all paths which essentially drive to 'duplicate' plugins. On here that meant removing all of the /usr/local/lib ones and leaving just the /usr/lib64 ones.

I essentially brought plugin paths to the following (taken from the cfg file):

    <pluginLadspaPathList>/home/lo/ladspa:/home/lo/.ladspa:/usr/lib64/ladspa</pluginLadspaPathList>
    <pluginDssiPathList>/home/lo/dssi:/home/lo/.dssi:/usr/lib64/dssi</pluginDssiPathList>
    <pluginVstsPathList>/home/lo/vst win 32bit:/home/lo/.vst win 32bit</pluginVstsPathList>
    <pluginLinuxVstsPathList>/home/lo/vst:/home/lo/lxvst:/home/lo/.vst:/home/lo/.lxvst:/usr/lib64/vst:/usr/lib64/lxvst</pluginLinuxVstsPathList>
    <pluginLv2PathList>/home/lo/lv2:/home/lo/.lv2:/usr/lib64/lv2</pluginLv2PathList>

Load time is still around 41 seconds. An improvement compared to the initial ~ 63 but still quite a bit IMHO.

Something I noticed is that for LADSPA and DSSI muse still complains about duplicates found in /usr/lib/ladspa/ and '/usr/lib/dssi' respecrively even if these are removed from the settings. Might these paths be 'hard coded' somewhere or picked up somewhere else?

Overall, I'm wondering if there's a more efficient way to do plug-in scanning, well knowing that plug-in situation is always a big mess depending on user set-up, distribution, etc. etc.

Some UX suggestions would be:

• First start the application (i.e. the main window) then trigger whatever plug-in scanning is needed and minimize the splash-screen, because if the plugin caching process takes long and the user de-focuses the splash is lost and the application seems 'ghosted'
• Try do do plug-in 'diffs' based maybe just on filenames / time-stamps / check-sums (i.e. detect just added / changed plugin) and only scan those
• Offer to do full plugin re-scan within the application on demand (i.e. not on startup)
• Offer to skip plugins which take a while to scan / detect / load and possibly 'black-list' them
• Offer to set-up plug-in directories at first start and maybe decrease the number of default directories?

These are just suggestions from a user perspective ;-)

Hope this helps.

[terminator356]

Hi sorry for the delay. Thanks for the tests.

Some quick tests I did (same set-up no changes in plugins).

-N takes load time to around 40 seconds

-2 load time is around 53 seconds

Using both of those together (unsurprisingly) takes about 4 seconds. But then, of course, only DSSI plugins are available.

-C load time is around 28 seconds

In this last case I still see lots of lilv_world_add_plugin() messages when running on the terminal

That is because we don't cache LV2 scans. They are done always each time.
So with -C (Do not re-create plugin cache), that 28 seconds is mostly LV2 scanning each time,
plus reading the existing cache.

The -C test seems to suggest that (non-LV2) plugin scanning is taking place each time muse is started.
Something seems to be triggering a scan each time.
Because if -C make it faster, that seems to be what is happening.

If you run with just -D (debug) alone, you might see a line somewhere near the top which says
"Re-scanning and creating plugin cache files..."
If you see this line each time, then something is not right and scanning is taking place each time.

In addition to -V and -2, there is one more plugin disabling option I forgot to mention, that you could test:
-p (Don't load LADSPA plugins)

I would like to have an idea of just how many plugins we are dealing with here.
If you have some time, could you open each of the scanner cache files in your home user cache folder:
~/.cache/MusE/MusE/scanner
and tell me how many lines are in each file.
Or, simply, tell me how large each of these files is in bytes.
In other words, are we dealing with tens, hundreds, or thousands of plugins?

What kind of hardware is involved here?
Desktop PC? Embedded? Mobile?

Something I noticed is that for LADSPA and DSSI muse still complains about duplicates found in /usr/lib/ladspa/ and '/usr/lib/dssi' respecrively even if these are removed from the settings. Might these paths be 'hard coded' somewhere or picked up somewhere else?

If I recall, that might be because some DSSI plugins install themselves and/or links in both places.
I will have to check.

Thanks.

[lorenzosu]

If you run with just -D (debug) alone, you might see a line somewhere near the top which says
"Re-scanning and creating plugin cache files..."
If you see this line each time, then something is not right and scanning is taking place each time.

Indeed I see this:

Setting cache to dirty due to NEW plugins:
New plugin /usr/lib64/ladspa/ladspa_dsp.so:

Not sure why it is considered changed as I'm not aware of any explicit changes

If you have some time, could you open each of the scanner cache files in your home user cache folder:
~/.cache/MusE/MusE/scanner
and tell me how many lines are in each file.

This is the output of du -ah * in that dir:

4.3K    dssi_plugins.scan
229K    ladspa_plugins.scan
159K    linux_vst_plugins.scan
2.0K    mess_plugins.scan
3.4K    unknown_plugins.scan
53      vst_plugins.scan

However it seems those are XML files? So I guess size doesn't give an idea of the number of plugins?
If it helps this is the output of a quick python script parsing the xml in the files and counting the <plugin> nodes:

mess_plugins.scan               5
ladspa_plugins.scan        865
vst_plugins.scan                   0
linux_vst_plugins.scan    252
unknown_plugins.scan      19
dssi_plugins.scan                16

If I recall, that might be because some DSSI plugins install themselves and/or links in both places.
I will have to check.

What I was meaning to say is that I think some search paths might be hardcoded? As a test for this I removed all search directories for LADSPA and DSSI in the GUI and I still see messages about LADSPA and DSSI plugins being imported.

[terminator356]

Fixing empty ladspa_dsp.so support...
Please wait...

[terminator356]

Congratulations! You revealed at least two bugs, and the second you didn't even know about.

So... About ladspa_dsp.so, which is part of the DSP project at https://github.com/bmc0/dsp
The file does not register any plugins until the user creates one or more config files!
See the README documentation at the link.

With MusE, it was not detecting that the file initially has no plugins, and did not throw it into
the 'unknown.scan' cache file. Thus it kept on scanning each time, looking for something.
This has been fixed. If the file has no plugins, it is thrown into the unknown.scan cache file.

Although I have not tried it yet, it should work once you create config files for it.
Note that you may have to manually tell MusE to re-scan either with -R, or via the Global Settings.
This is because I do not think MusE can detect when such config files have been added or changed.
Because there would be no change in the plugin file itself.

I must say, this 'user-configuration' scheme seems dangerous to me.
It seems to go against accepted plugin rules that state a plugin should never change
its number of inputs or outputs or other stuff, because that might break existing
DAW song files that were created with, and expecting, a certain number of ports.
By allowing the user to change these configuration files, it seems a bad idea.
It's unclear how MusE would respond to such configuration changes.
It remains to be tested...

Meanwhile, our MESS synth plugin 'simpledrums' was missing from installation
due to a recent mistake by me. It should be working now.

Try it, lemme know if any trouble.
Then, we can try to reduce the LV2 loading time and look at the LADSPA/DSSI issue you mentioned.

Thanks.

[lorenzosu]

Hi thanks for the details and updated release.

I downloaded the new AppImage and deleted all config and cache files just to make sure I was testing 'fresh'.

• First start was about 78 seconds - of course all plugin caching was re-done
• Second start without changes was about 26 to 28 seconds (tired a few times)

After removing irrelevent / duplicate (e.g. sumlinked) directories from all of the paths (e.g. /usr/local locations), I first got a start time of about 40 seconds, then after that about 15 seconds.

I still do think in any case it would be great to provide more effective feedback to the users, e.g. a progrebar of plugins being scanned etc, and not just in the splash screen, just my two cents.

I've diggged a bit the dsbug (-D) output and noticed some stuff, which maybe is well known and totally irrelevant, but still sharing it.

There's a bunch of ladspa plugins which are detected here:

/tmp/.mount_MusE-mgb8RYI/usr/lib/muse-4.2/plugins/

DSSI plugins seem to be scanned before LADSPA ones (at least looking at the debug log in linear way), while the debug message seems to say they are being scanned before LV2

I rechcked the (estimated) number of plugins, for everything besides LV2 by parsing the cache files:
mess_plugins.scan 5
ladspa_plugins.scan 434
vst_plugins.scan 0
linux_vst_plugins.scan 251
unknown_plugins.scan 21
dssi_plugins.scan 8

For LV2 I simply counted the initLV2 lines in the debug outpu and they are 597, so quite a few.

Still from the debug log ladspa, dssi, vst all take about 1 second to 'scan' (on successive scans).
Just the LV2 ones take circa 5/6 seconds

So it seems the bottleneck might be LV2 here? :-(

[terminator356]

Hi. Just pushed to master:

• LV2: Streamline discovery. Eliminate duplicate load_world and load_all_plugins.
  This might make only slightly faster startup time. But code is more proper now.

Try it, see if it improves a bit.

So... I have reviewed the code and I thought about what the other developer said at the time
about LV2 slow loading at startup.
Currently, at startup we load all LV2 plugins into some lists so that they are all 'ready to go'
when you open the plugin or synth dialogs, or load a song.
He suggested we should not even bother loading anything at all until someone
opens the plugin or synth dialogs, or loads a song.
Thus someone working on a purely midi song would have no use for plugins or synths,
and so these lists of plugins and synths could remain empty for the whole session!
As you suggested, 'on-demand'.
Thinking about this, it makes sense and I think I can see a code path forward here.
With some clever treatment of song loading to make it fast even though it is on-demand.
Whereas the plugin and synth dialogs can afford to be a bit slow [ Edit: upon first opening]
since they show all plugins.

Can't guarantee when, but I would like to try.
Hmm.....

[lorenzosu]

Had the time now to test this.

A 'fresh' run gave me now 38 seconds (re-scanning all plugins)

After that first run (no command line flags) it took about 14 / 15 seconds so there's an improvement.

Regarding how to cache plugins maybe a middle ground would be to do (propose) that within the software when e.g. a user tried to add a synth track or a plugin slot.

So something like "A plug-in scan is needed. Proceed?"

That IMHO would cover users who only use MIDI (they probably won't add plugins) and ones who want to use plugins.

In this way you don't have the 'delay' at startup?