Add yanking guidance to the readme faq

[LilithHafner]

Continuation of #102636, taken over at Ian Butterworth's request. New PR so that I can accept review suggestions. cc @oxinabox @ericphanson @vchuravy @Seelengrab

[Seelengrab]

Could this sort of too-wide compat be resolved by making a PR to the registry with the correct bounds instead of yanking? After all, the release is (a priori) good, just not for all originally specified versions.

[Seelengrab]

Thanks for taking up the effort!

[ericphanson]

I disagree about the primary purpose of yanking being to make it impossible to install a version. Deleting the version from the registry does that. Yanking has the special property that existing manifests will continue to work. Therefore, I think deleting the version should be used for actively exploited malicious activity.

I think yanking should be used for lesser situations (e.g. in the XZ builds that were yanked, at the time of yanking it was suspected that the backdoor was not present in our builds, so yanking was a less-disruptive first step), or for compat-related issues as detailed here.

BTW: pretty much an aside, but Julia 1.0 does not respect yanks (support was added in v1.1), which is maybe good to know.

[Seelengrab]

Therefore, I think deleting the version should be used for actively exploited malicious activity.

Deleting also has the unintended sideeffect of making the registry (and by extension, registry maintainers/the community) forget that the version was already released previously and subsequently removed. Yes, that information is still technically available in the git history, but that's much less likely to be checked compared to an immediate "hey this was yanked previously".

On the other hand, what exactly is the upside of keeping known-insecure versions installable for existing manifests?

[LilithHafner]

In my very limited experience, adjusting compat bounds on existing versions is pretty hard. But if that can be done easily and reliably then I think it should replace yanking for the second use case. Then Pkg could e.g. print a warning and prompt for confirmation when instantiating a yanked version (IMO there should always be a way to install yanked versions, even if only an UNSAFE_IGNORE_YANKS variable)

[ericphanson]

Therefore, I think deleting the version should be used for actively exploited malicious activity.

Deleting also has the unintended sideeffect of making the registry (and by extension, registry maintainers/the community) forget that the version was already released previously and subsequently removed. Yes, that information is still technically available in the git history, but that's much less likely to be checked compared to an immediate "hey this was yanked previously".

Given that has never actually happened, that doesn't seem that important? And it could be solved many other ways. We could keep a markdown doc in General with a list of such versions if we ever find one.

On the other hand, what exactly is the upside of keeping known-insecure versions installable for existing manifests?

Huh? I am saying to not do that?

---

I think you want to use yanking for something other than what it is meant for. Yanking is a Pkg.jl feature that exists in order to make a version installable-from-a-manifest, but not resolvable. Like that is what is was made for. It has special support in Pkg to make this possible. The thing you want, to make something not installable, is 100% possible without special support in Pkg. Namely, to remove it from the registry. The reason yanking exists is because sometimes we want to make something not resolvable but without breaking reproducibility.

[LilithHafner]

How does this sound? (up to definitions of "very bad" and "very very bad")

Compat bounds were too wide => PR General to retroactively adjust compat bounds
Very bad release, worse than any ordinary bug => Yank
Very very bad release e.g. actively exploited security vulnerability => Delete from registry

[ericphanson]

That sounds reasonable to me. I think yanking is also ok for trying to address compat issues, given the registries compat format is undocumented, we have no tooling setup to adjust bounds that I know of, and we don’t have good CI checks to verify it is correct. IMO if we had some of those things we could prefer adjusting bounds over yanking, but given that we don’t, manually adjusting bounds is error prone and requires careful review, while yanking is straight forward.

[LilithHafner]

Okay. Once #104849 is in better shape we can change the guidance to what I recommended above, and until then also recommend yanking for the too-wide compat bounds case.

[Seelengrab]

Huh? I am saying to not do that?

Both ]add and ]instantiate are an act of installing packages. Instantiating old manifests that have vulnerable versions is IMO just as bad as adding that version to new manifests. I'm proposing we treat these two cases the same, in order to protect the users of the registry from unknowingly installing vulnerable versions, both through instantiation as well as adding. No one is going to actively check their old manifests for security vulnerabilities; thus, this should be something that's handled directly by the registry.

Very bad release, worse than any ordinary bug => Yank

I think this is not precise enough. What exactly is "worse than an ordinary bug"? In what situation would a package developer/registry maintainer choose a yank over a straight-up deletion of a version, if the yank still permits the chance that someone gets a horribly broken version?

Very very bad release e.g. actively exploited security vulnerability => Delete from registry

How are we going to keep record of previously deleted versions, to prevent double registrations? Are we actively going to check the git history of every package?

---

I know that we don't have tooling & documentation around the Registry.toml format and adjusting compat bounds. This is an opportunity to remedy that, since those are quite simply operations that occur during regular maintenance of the registry.

[LilithHafner]

I put my suggestion, with revisions due to @ericphanson's observation that editing compat bounds is harder than yanking, into prose in the latest commit.

[ericphanson]

I think this is not precise enough. What exactly is "worse than an ordinary bug"?

I agree this is imprecise. But I think having something here is better than nothing, so let's merge this and if someone proposes more precise language in another PR we can consider that there.

In what situation would a package developer/registry maintainer choose a yank over a straight-up deletion of a version, if the yank still permits the chance that someone gets a horribly broken version?

From the perspective of the registry, it's basically "fine" to get horribly broken code. That is the ordinary course of business when using code/the registry, and there is an ordinary, regularly-exercised fix: update to a non-broken version. We would never delete a version due to the brokenness of the code. If it is doing the wrong thing, and we delete the bad version, how can you compare the old vs new to understand how much the bug affected your downstream application?

How are we going to keep record of previously deleted versions, to prevent double registrations? Are we actively going to check the git history of every package?

I think we can cross that bridge when we get to it. Hopefully not for a long time.

I know that we don't have tooling & documentation around the Registry.toml format and adjusting compat bounds. This is an opportunity to remedy that, since those are quite simply operations that occur during regular maintenance of the registry.

Is this more of an opportunity than any other time? The problem has been known for years. I'm not going to block merging this helpful advice which is fit for the current status quo in favor of waiting for something that may or may not happen.

[LilithHafner]

@StefanKarpinski, would you like us to reconsider this decision and/or revert this PR?