WIP: downloading lua scripts without git installed

[ChristianBirzer]

This is more or less a proof of concept. The idea was (as described in darktable-org/lua-scripts#391) that on end user systems git is usually not installed and therefore they are unable to install the lua script extensions.

Currently it's only implemented for Windows (PowerShell needed, i.e. Win 10) but on Linux the tools (wget, unzip/untar) should be available out of the box on most systems I think...

Tested on a complete fresh Win10 sandbox and seems to work fine.

If git is installed, it will be used (so no change is expected on systems where git is available)

Of course, there is a lot more to do:

• Allow updates from the script manager (old directories must be deleted before copying over the new stuff - but only those that belong to that "category"..)
• Allow downloading new repos from the script manager
• Port it to linux
• Port it to MacOs (I don't have a mac to I don't know the commands there... help needed!)
• Maybe invent something to check if there is an update for a repo available...?
• Maybe use one script instead of multiple commands to reduce the number of opening ugly "DOS-Boxes"

Feedback wanted: Do you think it is worth continuing the work on that? 🤔

[TurboGit]

I'll let @wpferguson comment on the final design, but to me I think this should not be restricted to Windows (here you're using powershell). I think having the option to download via wget directly in Lua without having to install Git would be nice on all platforms.

Now the question should this be the solution or an option and get the Git support?

[ptilopteri]

imnsho, it would be better to only have a non-git download process

[wpferguson]

@TurboGit thanks for pinging me. I meant to respond to this when it was an issue, but I've been crazy busy (have to move me and the family to a new house that's currently under renovation in 10 days, while at work I'm doing a major systems upgrade for one client, keeping the rest of the clients happy and bringing on board a new client).

TL;DR the repository is versioned and git is used to install the version matching the lua API version of the darktable executable.

Let's go back 6 years (actually 6 years and 15 days) to when I submitted my first lua PR.

• The API was stagnant (it had just changed for the first time in 3 years).
• scripts_installer didn't exist
• script_manager didn't exist
• libraries didn't exist
• Scripts got installed however the user wanted
• The only issues were bugs. Either the bug got fixed or the user was told to update their scripts.
• Scripts were standalone and could be installed individually

6 months later....

• common code from the scripts are collected into libraries
• some scripts are still standalone, but some require the libraries to run
• script's are recommended to be installed as a complete tree instead of individual scripts
• there is no recommended or preferred way to install the scripts
• most problems with the scripts are solved with "update your scripts"

2 years after that...

• created script_manager in order to make the scripts easier to use
• and solve the "update the scripts" problem with a simple button and git

6 months after that...

• started working with the API and adding new functionality
• There have been 10 API version changes in the last 3 years and 2 months
• Revised API checks in the scripts to so that they change only when they require a capability added by a new API version

14 months after that...

• added scripts_installer to make it eas(y|ier) to install the lua scripts.
• found out the hard way that a mistake in this script will prevent darktable from starting.
• found out the hard way that no matter how much you test, you will not think of something
• hard way = delayed release/emergency patch

9 months ago...

• added API version awareness so that the correct scripts version is loaded for the darktable lua API.

Reasons we use git

• the repository uses it
• cross platform (linux, macos, win 7,8,8.1,10,11, etc)
• simple commands to install or update
• easy to call from a script
• less chance of stepping on a windows/microsoft "land mine" - meaning we can't test a windows based solution thoroughly enough to make sure it's going to "just work"
• less code to maintain

Alternatives

@darix came up with an idea to package the lua scripts so that they could be installed by the system (https://discuss.pixls.us/t/lua-scripts-in-a-system-wide-path-so-packagers-could-ship-lua-scripts/26133/3 and darktable-org/lua-scripts#352). If we do this, then this would solve the versioning issue (i.e. each darktable release would have a corresponding scripts release) and the ease of use (just use the system package installer).

Add an option to the installer to install git if it's not present, then script the install?

Danger

If we decide to change data/luarc I recommend that we do it early in the 4.2 release cycle so that it can get as much testing as possible. We've already learned the hard way about users doing something we didn't expect or plan for and having to scramble to fix it. With the weekly windows builds on pixls and the nightly builds hopefully we can drive out any bugs before release.

Sorry for the novel

[wpferguson]

Now that I've had a good night's sleep, one last thought...

When we had a low rate of API change most of the problems we had were the users needing to update their scripts. The last 3 years we've had a lot of API changes, but we haven't had many issues with scripts needing updated which in turn has lowered the maintenance burden. Almost all of the issues now are scripts that have a bug or feature requests.

[ChristianBirzer]

@wpferguson thank you very much for this detailed insights.
As you might have already seen from my previous "contributions", I'm very new to darktable at all. Both as a user and as a developer. This might be the reason why I see some things a bit different (or let's say from an outside view). Even if the use of git as a 'package manager' is absolutely clever, I have never seen any other application that does this. For me personally, this is absolutely no problem. git is installed on my machine and I can download the stuff from my own (or anyone else's) repo and updates works with a simple pull. That's really a smart solution.
But since I try to convince some of my friends from darktable, I don't know how to tell them to install a developer tool to get the script extensions (that are a must-have in my eyes).

My approach to use the zip/tar download from github could solve both issues (of course, if it works what isn't sure so far).

• Currently, the API version dependence is solved by branches. The script manager pulls the right branch. This can also be done with the zip download. If no branch for the current API version exists, the download should fail and can fall back to the main/master branch.
• The scripts can still be maintained in the git repo. Nothing changes there. And the git-based installation still works.
• The platform independence can be done by calling different external commands (powershell for windows, bash/tar etc. for linux, something else for MacOs).

I was a bit surprised that the feedback was to use a non-git download as first choice. I would prefer (at least in the first step) to use it as a fallback. If git is not installed on the system, we could try to download the scripts via zip. In the worst case (except for implementation errors of course) this won't work too and we still could tell the user to install git.
I would also fall back to the git installation when calling the download scripts fails (e.g. old Windows with no powershell etc.).
On systems where git is installed, nothing should change since the update/download is still done via git.

Of course, there are still a couple of things to solve:

• How to handle updates? My approach would be to simply delete the "old" directories and unpack the newly downloaded archive. Since different repos can be installed inside the lua tree, only delete the directories not containing any other repo. An alternative would be to simply unpack the new archive over the old (less code, less risk to delete the wrong thing but outdated scripts would not be deleted if they are no longer in the repo).
• How to know if there is something new in the repo? Maybe this is not necessary because if I hit update now, I get the latest stuff. Unless I missed something, there is no way to get a "has something changed and what if yes" now either. So simply downloading and installing the latest stuff might be good enough.
• I have no idea how to make a switch from a "zip" installed lua extenstion to a "git" installed one. Since the zip does not contain a .git directory (it is not a repo, it's just a copy of the files) you cannot use git on that directory afterwards. Simplest solution: Delete the old directory and re-install using git.

A specific question: Does the lua directory contain any data the is changed during runtime? What, if I delete the directory completely? Do I loose anything except the scripts that I can easily download again? (ok, I can patch my scripts locally but that's not the common end user use case)

A lot of error checking must be implemented to get the status of download success etc of course (I think about the API version branches, the choice between master/main branch etc.).
What I don't like at all about this solution (but the git way is not much better): A lot of command line windows are opening during the download/installation. If anyone has an idea how to get this a bit more beautiful, please let me know.

Other alternatives, like a completely standalone installer for scripts (each script repo) like it is done in other applications would also be possible. But this would be a lot more work and would break the benefits of simply maintaining the scripts in the repo and have them updateable. (ok, maybe some git action could pack the installer... I don't know...)

So what is the preferred direction to take? Give it a try or cancel that idea an let users install git to use the scripts?
I guess I could do most of the work but not all by myself (no MacOS).

[parafin]

What about linking to libgit instead of requiring user to install git? I haven’t looked at what’s possible with libgit compared to git tool or how hard is it to use it though.

I agree that maintaining 2 ways to download scripts isn’t desired, we should choose one.

[TurboGit]

What about linking to libgit instead of requiring user to install git? I haven’t looked at what’s possible with libgit compared to git tool or how hard is it to use it though.

I didn't thought about this, really a good idea worth investigating.

[ChristianBirzer]

What about linking to libgit instead of requiring user to install git? I haven’t looked at what’s possible with libgit compared to git tool or how hard is it to use it though.

At first sight it seems that libgit supports everything you need when working with git repos. At least way more than we need to download the scripts.
I think this is a very good idea and worth trying it. It could also eliminate the command line windows that are opening because everything is done internally (some file copy stuff can also be handled without calling external commands).

Only drawback I see: libgit would have to be integrated into dt core application and interface to lua scripts must be written.
I have no idea how to add new commands/libraries to the lua interface but would love to learn it 😁

[wpferguson]

What I don't like at all about this solution (but the git way is not much better): A lot of command line windows are opening during the download/installation. If anyone has an idea how to get this a bit more beautiful, please let me know.

There are multiple threads all over the internet about the command windows opening when running a command from an application. Unfortunately it seems to be a built in "feature" of windows with no work around.

[ChristianBirzer]

Hi, I was doing some tests with libgit, as suggested by @parafin to download the scripts.
It seems to be really easy to add a some lua commands to handle all the stuff that is needed for the script manager.
So far I changed only the luarc for the initial download but that already works.

Only thing I was not able to achieve at all: Integrating libgit as a submodule (or something else) in a way so it gets built from the build process and darktable is linked against that lib. I managed to get the library built and I managed the link process. But not both with one cmake script. I don't know enough about cmake to know how it has to be integrated but I'm sure it is possible.
I helped myself by building libgit and then copying the dll to the directory where darktable.exe is and let the build process for darktable at least search for the include and .a file.
I hope, there is someone who can show me the way how it's done correctly 😊

Since my commit breaks the CI builds, I won't PR it here.
My changes are here

Maybe someone can help me with the build integration. If that succeeds, I'd continue to modify the script manager to use the library instead of the installed git.exe.

Of course, a bit of error handling has to be done too.

Thanks in advance for any help 😁

[darix]

do not do intree libraries ever. That is a last resort and should not be the default.

[ChristianBirzer]

do not do intree libraries ever. That is a last resort and should not be the default.

Ah, thanks a lot for that tip. Well, I have no idea how to do it right in this case.
Shall I use the prebuilt binary?
Can you give me an example that I can copy where it is done right?

[darix]

just find the system libgit2 and link via https://cmake.org/cmake/help/latest/module/FindPkgConfig.html

[ChristianBirzer]

I just wanted you to know about the (non-) progress of this work.

In theory, libgit2 is just perfect for doing what we want. I managed to add the prebuilt lib to dt and also added some lua commands to do the clone etc.
BUT: With the current libgit2 release I cannot clone any repository. I only get a certificate error. According to the open issues this should be solved with a newer version of libssl - which I already have. The same error occurs with the original code examples. At least on my Win10/MinGw/MSys environment.

The current head of the lib, built from scratch, behaves better (or at least different...🙄) but then it would have to be added as submodule...

I already opened an issue at libgit2 and tried to contact the devs on their slack channel but got no answer so far 😥

As soon as I find a solution for this problem, I'll continue the work as I'm still convinced that loading/updating the scripts without having git installed is a good thing...

[github-actions]

This pull request did not get any activity in the past 60 days and will be closed in 365 days if no update occurs. Please verify it has no conflicts with the master branch and rebase if needed. Mention it now if you need help or give permission to other people to finish your work.

[github-actions]

This pull reguest was closed because it has been inactive for 300 days since being marked as stale.