anki: upgrade to latest release

[NilsIrl]

https://github.com/ankitects/anki/releases/tag/2.1.19

The structure of the project has changed in 2.1.17.

@oxij @the-kenny @Profpatsch @Enzime

[NilsIrl]

I've tried to do it but I was unable to do it correctly. Here is my attempt: master...NilsIrl:anki_2_1_19

All the changes made on this branch are in the right direction (from what I can tell) so it might be a nice starting point.

[Profpatsch]

Maybe we could start by adding a few tests first.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/how-to-override-python-package-anki/8213/3

[bhipple]

A month ago I also gave it a stab and also was unsuccessful, but at least merged a couple PRs cleaning up cruft in the anki expression and getting us a little closer.

[Atemu]

I've been making progress writing a derivation for their new Rust stuff yesterday but am now stuck with a rogue build script trying to access things outside the sandbox.

Highly WIP progress so far: https://github.com/Atemu/nixpkgs/tree/anki-update

[Atemu]

Speaking of which, can I somehow get Rust to tell me which path it's getting "Permission denied" on? By default it only prints a cryptic error message that is incredibly unhelpful.

[Profpatsch]

Speaking of which, can I somehow get Rust to tell me which path it's

getting "Permission denied" on? By default it only prints a cryptic error message that is incredibly unhelpful. Unfortunately not by default, the `std::io::Error` cannot allocate anything on the heap, so it is not allowed to allocate a `PathBuf`. If users want to have better error messages, they have to wrap std::io::Error for these cases.

…

On Tue, Jul 21, 2020 at 9:08 PM Atemu ***@***.***> wrote: Speaking of which, can I somehow get Rust to tell me which path it's getting "Permission denied" on? By default it only prints a cryptic error message that is incredibly unhelpful. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#78449 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAYB5ZVOAIVV4DFN45Y2FC3R4XRRLANCNFSM4KLNTASA> .

[Atemu]

I eventually found the culprits using lots of straceing and the rust parts build sucessfully now. I haven't been able to build the whole program to test their correctness yet however.

How would I go about wrapping it in case I encounter something like this in the future? Can I even touch the part which prints the file errors?

[Profpatsch]

How would I go about wrapping it in case I encounter something like this

in the future? Can I even touch the part which prints the file errors? No, this is the `Debug` or `Display` instance of `std::io::ErrorKind`. There has to be other rust users that have solved this problem to some satisfaction. My best guess is wrapping `std::io::Error` into a struct/enum that is constructed using the path when applicable. It’s a rather invasive change that has to be threaded through the application code however.

…

On Wed, Jul 29, 2020 at 5:32 PM Atemu ***@***.***> wrote: I eventually found the culprits using lots of straceing and the rust parts build sucessfully now. I haven't been able to build the whole program to test their correctness yet however. How would I go about wrapping it in case I encounter something like this in the future? Can I even touch the part which prints the file errors? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#78449 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAYB5ZVHDQLMIAIES76KYALR6A6JVANCNFSM4KLNTASA> .

[martinetd]

I've been annoyed by that as well - have you made any progress @Atemu?
I'd be willing to spend some time kludging this around a bit more, but if you've already done stuff like building the rust part I'd rather not spend hours doing what you've already done; so even if you're stuck posting an incomplete commit or something would be great! :)
EDIT: I'm blind, just seen you posted a link to your anki-update branch, I'll try to pick it up a bit from there.

(Cc @cideM)

[Atemu]

Days my friend, days.

Pushed some additional stuff I had laying around. Take the commit message to heart though.

Also https://github.com/Atemu/anki/tree/wip-hacks-to-get-nix-build-working

[martinetd]

Cheers, thanks; already started hacking so not sure I'll use it but I wholeheartedly agree with the comment anyway... Well, I kind of knew what I was getting into as I first built it outside of nix and gasped at all the things it started to pull!
I mean what kind of project would not want to be part python, part rust and part nodejs?! Sounds sane to me! :P

I'll post something again if I give up in a couple of days, right now the sunk cost fallacy wants me to keep trying a bit longer :D

[martinetd]

Well, we're a couple of days later so sunk cost fallacy lost a bit of its appeal :P

I actually did make some progress:

• src is prepared once and shared for intermediate packages, I think it's better than what was done reprepaing the sources everytime.
• ankirspy properly builds within nix; https://github.com/martinetd/nixpkgs/blob/anki/pkgs/games/anki/rspy/default.nix

The last(?) hurdle I'm facing now is orjson, a dep of the pylib, that isn't packaged and pretty similar to rspy. I had (incorrectly) assumed it would be as easy as doing something similar for it but it requires rust nightly very strongly and I'm not too sure what to do with that -- I mean, as a separate repo it might be ok to fetch nixpkgs-mozilla and use rust nightly (it's just compiling a .so and throw rust away with the water so it wouldn't leave marks), but for upstream nixpkgs I don't think that'll fly.

(didn't look at the nodejs part but it can't be worse than rspy... hopefully, and @Atemu's repo has made a shot at it)

If we solve that all that's left would be to put the subpackages as input and build the qt program which probably will also have its share of troubles /sigh

Anyway, I probably have better things to do so I'll just leave this at it... If orjson magically gets in nixpkgs somehow or stops requiring nightly there'll still be time to reconsider :/

Good luck to whoever comes next :D

[cideM]

I haven't made any progress on this one so far sorry

[Atemu]

I think orsjson was the point I gave up at too. Simply pulling it from the overlay probably isn't too bad though. Not good enough for Nixpkgs of course but compared to all the other hacks wee need to at least get something working, it's pretty tame.

The JS parts I just node2nix'd and that seemed to build fine but no idea if it actually works.

[Atemu]

#106638

[timokau]

We might be able to get around the orjson issue by replacing it. See this discussion.

[schmittlauch]

If anyone wants to try again: While no release has been made since, orjson has been dropped as a requirement in ankitects/anki@f9e939a.

I guess those with the sunk cost might be quicker to resume the packaging attempts than me having to dig through it first ;)

[stale]

I marked this as stale due to inactivity. → More info

[MikiVanousek]

For some reason, anki-bin is broken for me, so I use the flatpak version. It's interesting to hear that is not the case for everyone - this is the first time I encountered 'works fine on my machine' type of problem on NixOS.

[Atemu]

Definitely open a bug report on that.

[martinetd]

FWIW this recent activity had me take another look (I was afterall stuck on orjson which is now even packaged in nixpkgs... so should have been able to move forward), but too many basic blocs changed (rspy is gone, maturin is no longer a thing, there's a new sass css thing, the way translations files are managed changed.. or rather the whole build system changed completely from a """simple""" makefile into a bazel project) -- so it'll probably be faster to start over from scratch at this point, and even if we had gotten it working back then it'd be a huge timesink to keep up to date everytime they decide a makeover would be a good idea :|

OTOH we do have a buildBazelPackage so that might be worth just giving a try if someone wants to, but I'm not going to try in a hurry.

[senorsmile]

For some reason, anki-bin is broken for me, so I use the flatpak version. It's interesting to hear that is not the case for everyone - this is the first time I encountered 'works fine on my machine' type of problem on NixOS.

This could be due to the version and whether or not you're using xorg vs wayland. See my comment regarding issues with anki-bin: ankitects/anki#1767 (comment)

In short, anki-bin works fine from the 21.11 channel, but doesn't from unstable when using sway.

[fstamour]

If I understand correctly, this package (anki) is built from source, and the package anki-bin downloads pre-built binaries and patch them?

[martinetd]

If I understand correctly, this package (anki) is built from source, and the package anki-bin downloads pre-built binaries and patch them?

yes -- with the source package being very hard to upgrade.

[mainrs]

Unfortunately, Anki 2.1.15 doesn't have the option to change the timezone handling, so sync is now broken for me. I guess I could delete and recreate my AnkiWeb account, but since the old scheduler is legacy it's likely a new account would use the v2 scheduler.

Pretty sure the scheduler is tight to your app version you use and not the account.

[nessdoor]

OTOH we do have a buildBazelPackage so that might be worth just giving a try if someone wants to, but I'm not going to try in a hurry.

Last time I tried, that route didn't quite work, unfortunately. I cannot say whether the situation has changed.
I remember trying with recursive Nix, too, but to no avail. Then again, recursive Nix is an experimental feature, so the outcome was expected. Again, I don't know if any advances have happened in the meantime. But for sure, I can remember someone from the lead contributors saying that one of the principal use-cases for such functionality was, indeed, the ability to support complex Bazel builds.

I still cannot understand why the ankitects chose to rebase their work on a Bazel system. Everything is so much more complicated...

[senorsmile]

BTW, bazel is no longer used as of 2.1.55: https://github.com/ankitects/anki/releases/tag/2.1.55.

I'm not familiar with "ninja", so cannot comment on whether or not this puts it in a better position now for nixpkgs.

[nessdoor]

Oh god I can't believe they changed again. Time to delete all my local branches, I guess.

Well, Ninja should be more manageable. Right now I don't have time to put behind such a large project (I don't think anything will go particularly smoothly), but I might come back to it in 3/4 months.

[martinetd]

With a twist: that ninja doesn't look like anything like the ninja build system? ... https://github.com/ankitects/anki/blob/main/ninja (and no build.ninja in sight)
Either way we still have to build a bunch of rust crates and then some python, but it looks much simpler than before -- maturin is also gone and I remember that to be a huge headache. (EDIT: ah, already said that this summer :D)

This might actually have become quite packageable, but I also don't have time in the near future, I'm (un)happily running the binaries to have a working anki server compatible with the android client for now...

[nessdoor]

Wait wait, they actually invoke the Ninja we all know, that file is... something.

Anyway, I just noticed we missed out on ankitects/anki#1378 and the intense discussions held therein, or at least I did. Happy to see that we were not the only ones struggling with this.

I am confident that Anki will be properly packaged by us very soon.

[nessdoor]

I take back what I said. There is no build.ninja in sight because, well, it gets generated by the Rust packages...

Why is this piece of software so complicated? Why are they using Rust as a scripting language to generate build-time files?

Again, I think it's doable, but we need to compile a lot of crates, and some of them pull modified dependencies from forks that the Ankitects maintain. I can't understand whether it is extreme dedication to build the most optimized and flexible program on Earth, or something else.

[Atemu]

I'd wait a few releases. Perhaps the next build system they come up with is makefiles invoking nix to generate rust code that builds a bazel declaration which invokes meson and ninja inside. Oh and maturin, can't forget that.

[nessdoor]

So, I thought that generating the build.ninja and using that as the basis for the Nix port would have been a good approach, but it turns out that the generated file is 1MB of unreadable auto-generated build rules. This is not a problem by itself, but it includes a lot of rules that try to download dependencies at build time.

I think that the best approach would be to patch the runner. It seems to be well-structured, so we should be successful. Maybe it is sufficient to disable those download steps and provide our own binaries in the build environment.

[fstamour]

For reference: ankitects/anki#2202

[Atemu]

ankitects/anki#2345 introduces an option for packagers to declare their own binaries for protoc. I imagine similar steps can be made for other things Anki would like to download during the build.

[nessdoor]

Last time I gave a thought to this whole thing (3 weeks ago, maybe?), I kinda came to the conclusion that we're better off reproducing the build steps with Nix directly. It should be doable, but the result might be unstable due to the unavailability of Nix tooling for pinning package dependencies for some languages (i.e. Python and Node). Such dependencies may be too many to keep track of all of them by hand, but I never got past the derivation for the Python env (didn't have enough time), so I cannot give a sound opinion on this.

It's just that there are too many things to patch in that meta-builder, it would all be so brittle...

[dotlambda]

unavailability of Nix tooling for pinning package dependencies for some languages (i.e. Python and Node)

For Python packageOverrides usually works pretty well. For Node buildNpmPackage would work well if upstream had a package-lock.json, so that might be something we could ask upstream to add.

[euank]

I gave this a go, and the result is not super pretty, but it at least builds and produces a working anki binary on my machine. My WIP attempt is over here: #221229
I still intend to do a little more polish, but I think that it's not too far off.

I did intentionally avoid the whole pyoxidizer thing since NixOS doesn't really need any help bundling a python interpreter with a package correctly and reproducibly, but that does mean we're using a slightly different build process from than official upstream releases.

Upstream is using pyoxidizer to effectively embed a python interpreter and their code in a thick binary, and shipping that.
We're building a python wheel, and then using nixpkgs buildPythonApplication to run that wheel.
I don't think that should matter, but I may also have to do a little more digging there to confirm it.

Anyway, I just wanted to leave a note that I had some progress here, and I'm optimistic we might get an updated anki yet 😁

[nessdoor]

Nicely done, @euank, I also thought about building all the Rust packages with Nix beforehand, but didn't come to the conclusion that maybe stubbing the hell out of all those auxiliary programs could have been a viable solution. It makes sense, as it decouples us from that hideous Rust meta-builder.

Maybe you should mention this issue and the previous draft PR in your contribution, so that we know what we can close if your PR gets the green light.

[euank]

If anyone would like to verify the above package (#221229) works for your use of anki, it'd be appreciated!

I think the package is in good shape now, but I've only used it with my decks, which only exercise audio and some html rendering, but not videos or images or whatever other features.

If you want to test it, I think the following should run it (though it's possible it'll have to build or download a buncha stuff, qt is a quite large dependency):

# flakes / modern nix
nix run 'github:NixOS/nixpkgs/refs/pull/221229/head#anki'

# non-flakes
nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/refs/pull/221229/head.tar.gz -p "(import <nixpkgs> {}).anki" --command anki

[senorsmile]

I tried it out (using flakes) on my main deck:

• syncing works
• complex html/styles work
• images work
• sound works

I no longer use any add-ons so I have none to test. So far lgtm.

[sergv]

Also tried it on my NixOs, didn't test much but it starts and existing cards can be browsed and edited. Overall seems to be working.

[nessdoor]

Startup fails with error:

Run with DISABLE_QT5_COMPAT=1 to confirm compatibility with Qt6.
Preparing to run...
Qt warning: Could not find the Qt platform plugin "wayland" in "" 
Qt fatal: This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

Available platform plugins are: vkkhrdisplay, minimalegl, xcb, vnc, minimal, linuxfb, offscreen, eglfs.
 
Aborted (core dumped)

I'm running a NixOS 22.11 machine with Sway (Wayland). Could be a misconfiguration on my part. Will investigate further as soon as I have time.

[luisnquin]

It works for me! @euank nice work

[siraben]

Anki 24.04 has been released.