Freshen To-Do list // Why has Freshen been deprecated?

[Nuc1eoN]

This is supposed to be a meta issue, tracking changes that need to occur (if any) in order to bring Freshen back to live as a minimal viable solution to update the system.

I am not fully in the picture as to why Freshen has been "deprecated". To my knowledge, Freshen being stuck on python2 was the major reason why it was left completely unused recently. However this has now been resolved thanks to @sage-etcher.

But I do know there have been some general usability issues with Freshen in the past?? What were those issues specifically? @mwh maybe you could shed some light? What have been the major pain points with Freshen? Are there any important standing issues that need to be adressed in order to officially bring it back (e.g. in a future GoboLinux release)?

@sage-etcher Have you tried actually using Freshen thus far to update your system? Does it work without nuking; is there anything evidently broken?

[mwh]

I think that the mass-upgrade mode is an attractive nuisance as the recipe set is not sufficiently robust to make that safe to do. The part that lists available updates should be fine, but running dozens of unattended Compile jobs in a row can and has led to broken systems, sometimes severely, and more often simply fails.

Being prepared to fix the breakages that occur has always been part of running a GoboLinux system, but this invites more complex problems for less-prepared users. Even the list of updates probably encourages upgrades that weren't advisable on balance, but that seems more acceptable.

It's aping apt or emerge by having this feature, but the packaging engineering to support that isn't there and I don't realistically think that is going to change in the near term.

[Nuc1eoN]

Thanks @mwh for your valuable input, as always.

Whether or not Freshen will ever be "ready" for general use, I think it is still useful to keep this information available - esp. for new users and potential contributors.

What I am gathering from your assessment is that Freshen requires some predispositions in place, before it can work effectively.

To me, those distill down to:

1. Solid, reliable recipe set
2. Robust Compile
3. Refined packaging infrastructure
4. Freshen with proper error handling/checks (state backups?).

I agree that without those in place, Freshen could cause more trouble than it solves.

Albeit that shouldn't discourage anybody to work on Freshen, it is a good thing to be aware of :)

[mwh]

Yes, essentially. When things were compiled one user intention at a time, you could see the problem happen and address it, and know what you were doing at the time to know what to look for. When something went wrong, you'd fix it and try again. With dozens of updates, ordered by dependencies that weren't quite complete, with build instructions that worked on one machine once, things compound and it would sometimes do serious damage, but also often just get blocked by a failing build it insisted on putting early in the list.

Freshen -U is an invitation to make your system unbootable, but it's also a promise of something it can't actually guarantee to provide. It's not inherently unresolvable, but probably the "better Compile", QA on recipes, and perhaps package build infrastructure precedes it. There are also system-level questions, like whether UseFlags and Aliens are worthwhile, that might need resolving before the limitations of Freshen in those areas are resolved.

It's not clear that there would be much inheritance from this implementation in a future version in that future environment, where everything's been rewritten in Lua (or Teal!) and there's lots of fresh engineering effort going into the system. I'm dubious that that influx will happen, as "the filesystem is the package manager" has more or less been tried out now, and it got increasingly difficult to squeeze new sofware into that box (but we won on merging /usr).

However, at least "here are a list of things you could update if you want to dedicate some time to that" ought to work in Freshen now, and it will give you a list of their declared transitive dependencies too.

[sage-etcher]

@sage-etcher Have you tried actually using Freshen thus far to update your system? Does it work without nuking; is there anything evidently broken?

No I have not. While installing/updating individual programs, I've encountered various issue's while using the -U flag. I figured it unwise to test it on either core packages or the full system. Though, to my memory, the issues weren't fault of Freshen, rather each with the recipes themselves.

[Nuc1eoN]

Great, thanks for sharing.

My derived thoughts on tackling these issues are:

1. Implementing CI/CD for recipes
   Description: It is a reoccurring issue, that recipes will Compile on one system but fail on another. This mostly (or always) stems from a different base system. It is not always easy to determine whether all necessary dependencies have been added to the Dependencies file.
   Proposed Solution:
   Recipe builds should be tested in a dedicated sandbox/jail, that represents a "GoboLinux base system". It could reflect the state of a freshly installed GoboLinux or be a little more slimmed down.
   Implementation:
   I envision two approaches: either an "industry standard" Docker(/Podman) container based implementation or using GoboLinux Runner/Compile --pure. The second solution has the potential to be way simpler and better integrated, but that would depend on the state and stability of Runner/Compile --pure. It would have to be determined what those limitations are and how easy they can be overcome (#46 comes to mind).
   Depending on that I'd choose one of those options, whatever is more easy to implement as of now.
   Further remarks:
   The sandboxed builds could be performed either locally by the user or in "the cloud" (e.g. Github Actions). However an "only in the cloud" solution might be a waste of resources (pipeline minutes), as some programs require significant build time (e.g. linux kernel, Qt, Firefox,...). A hybrid approach would also be possible.
   I think for the start, doing the sandboxed builds locally is fine. It would have to be ensured that a sandboxed Compile has been performed, before merging any new recipes. Existing recipes have to be re-tested (priority on actively used ones).

2. Robust Compile
   Whether the CI/CD implementation is based on Docker images or Compile --pure/Runner, Compile has to be stable and reliable. Otherwise it would just become a nuisance. It has to be determined which bugs have to be addressed or features added to make this work and make it a pleasurable experience.

3. Refined packaging helper tools
   It would definitely make life a lot easier if packaging tools like NewVersion etc. worked more reliably and had a better success rate. Currently it is often hit or miss.
   ContributeRecipe could act as a nice tool that ensures that recipes are properly tested before contribution. In any case fixing the smaller tools like NewVersion should be easier. ContributeRecipe is a mess currently.

4. Freshen
   Once all the above is accomplished I see no issues using Freshen then 😄

[rubyFeedback]

If I recall correctly, Freshen was largely mwh's effort. I think it was originally written in ruby, then switched to python.

It no longer being "en vogue" probably has had a lot to do with time investment, as well as GoboLinux overall "slowing down".

In regards to the use cases, perhaps Freshen could be split up in functionality, e. g. providing a core set functionality, and on top of that functionality people could extend it, including GUIs, web-interfaces and so forth. That may depend more on Compile itself being more useful at batch-compiling something. For instance, I think it would be nice to do:

Compile --kde5
Compile --kde6

but I think GoboLinux currently does not track all these individual applications (or, if it does, the above instruction could probably not compile it). Then Freshen could tap into the above functionality; right now it seems to be as if there is some duplication going on (and, admittedly I understand the problem of implementing something via shell scripts, as opposed to e. g. in lua, ruby or python).

ContributeRecipe could act as a nice tool that ensures that recipes are properly tested before contribution.

It sounds a bit like reproducible recipes to me. Having a stricter specification for recipes would be useful in the long run, as you could enforce that people on different computers, with different software in place, could build / compile software.

Note that better and stronger specifications would be useful no matter which programming language is used. It could also serve as some kind of documentation that could be enforced. And help with other languages implementation.

I keep track of many remote source archives, using .yml files as a basis in a "shorthand notation". This shorthand information is then expanded to the "full dataset" that contains all the information one could ever possibly want to have, for a given source archive. Having that in place I can then, in theory, auto-generate gobolinux recipes, gentoo ebuilds, homebrew packages and so forth. (I write in theory, because in practice each one may have different requirements and oddities, and it is not always easy to know which ones are in place; see also a related project that tries to generate packages for different linux distributions, called fpm: https://github.com/jordansissel/fpm - although my own project does not have that as support yet, I have that on my todo list too, including gobolinux recipes. So anything that improves the specification used by GoboLinux or a Gobolinux-conforming system, would be really great to have. So, TL;DR, I think improving the base here, in regards to recipes + compile, would help with many other "upstream" or "On-top" issues, including freshen's use case of updating a system to the "latest and greatest".)