Repository navigation
Boundaries and rules for "package editing by Trustees" #14
Description
The possibility for Trustees to edit package manifests and publish patched releases is a feature of the current design.
However it's a very sensitive topic, as we're messing with things like build reproducibility, API surface and versioning guarantees.
So if we want to keep this possibility built in, it's quite necessary to clearly specify under which conditions and procedures the Registry maintainers would be allowed to do that.
There's already a section in the README about this, but I'm also opening this issue following the concerns in #4:
I don't think think "without having to ask authors" is the perspective we should take here. As a Haskell package author, it can be very annoying to learn that a Hackage trustee has tweaked version bounds without my knowledge or approval; at the very least I think we should require the trustee to inform the package author that this has happened.
Is the idea that for each package version, there should be an identifiable latest revision of it? It's not entirely clear to me how this will work with the current design.
Generally I don't trust Hackage trustees to understand my code well enough to know when it is safe to weaken a constraint, as breaking changes can be introduced if the APIs of any of the libraries you're using will "bleed through" into your code's public API. For instance, the version of language-javascript we are using in the PureScript compiler is effectively part of the compiler's API, because that determines what we accept in FFI files (aside: this is part of why I don't like it when people package the compiler in places like Nix or Homebrew with different sets of dependencies than those specified by our stack.yaml).
Activity
f-f commentedon Apr 25, 2020
To get back on the concern: the wording of "not having to ask the authors" is actually not nice, yeah. What I meant there is really that the Registry maintainers are able to do things like adjust boundaries if the authors are unresponsive - which is basically the same policy that Hackage Trustees have.
From the README of the hackage-trustees repo:
The guidelines ask to clarify how the package author was contacted, etc. I think here we should have the same obligation to be in touch with the package author first, and have a similar policy document so that it's very clear that:
joneshf commentedon Apr 26, 2020
Can we talk about why editing is necessary? The README explains it's so publishing another version doesn't have to happen. The README doesn't explain why publishing another version is bad.
f-f commentedon Apr 26, 2020
@joneshf there are some cases where authors are burnout, non-responsive, or something happened to them, and you need to adjust version constraints. The only way to do that is to bypass the usual "publishing by authors" and use "publishing by trustees".
Some time ago this exact thing happened to me: the author of a package we included in our build died, and there was no other way to adjust version constraints other than asking the trustees to fix them - issue here/
It was not strictly necessary to do this, but maintaining our fork over
ltsversions was getting very annoying.To sum up the reasons for my position here: it's not like I'm a fan of bypassing package authors for fun, it's just that I'm aware that these problems do happen in practice, and I'd much rather have a mechanism to handle this built into the design since the beginning, rather than bolt one up once we realize we need one.
f-f commentedon Apr 26, 2020
As for "why do we need to keep package constraints updated", there are many reasons, but some of the reasons I care most about are:
baseto keep up with compiler versionsfoo-3.0is compatible withprelude-4.0, and they both end up in a package set, but the bounds forfoo-3point toprelude-3.0. This means that people using a package set will be able to install that, but people using a solver won't. This is of course not ideal.joneshf commentedon Apr 26, 2020
Sorry, I'm not asking why Trustees can edit packages, I'm asking why package editing is necessary.
f-f commentedon Apr 26, 2020
@joneshf I'm not sure what you mean, could you clarify?
If I understand the question correctly, I'll pile up some facts from which we can infer the conclusion of "package editing is necessary":
prelude, or other fundamental libraries)Considering the facts above, are able to find another solution to satisfy all of them that is not "package editing is necessary"?
Publishing another version is very good, but authors might be unavailable to do that
joneshf commentedon Apr 26, 2020
Should I create a new issue? I feel like the conversation is going away from the original concern.
I do think answering the question simplifies the whole situation down to defining when a Trustee can do anything (rather than defining exceptions that a Trustee can exercise). But, I also recognize that there's been a bit of back and forth about something other than what the issue is directly about.
hdgarrood commentedon Apr 26, 2020
I think what @joneshf is getting at is asking why package editing is necessary as opposed to having a trustee publish a new version? I think https://github.com/haskell-infra/hackage-trustees/blob/master/revisions-information.md provides a good answer to at least some of this; and I think the same concerns apply to us here.
f-f commentedon Apr 26, 2020
@hdgarrood the README section I linked in my first post explains why Trustees cannot just edit manifests and instead have to publish a new version (even getting in detail of how to do this while staying into SemVer)
I.e. it's impossible to edit package manifests of already published versions because we're going to hash tarballs - that contain the package manifests themselves - so you cannot edit the manifest otherwise you'd get another tarball for the same package version, but with a different hash, and we definitely don't want that.
hdgarrood commentedon Apr 26, 2020
I mean why change the build metadata (what Hackage calls a “revision”) as opposed to creating a new proper release by bumping the patch component? I don’t see any part of the readme which addresses this question specifically.
f-f commentedon Apr 26, 2020
@hdgarrood we can address this concern more clearly, but the part that talks about that is:
..which in my perspective implied a clear separation between "what a package author can do" (i.e. publish under the first 4 segments of SemVer), and "what a Trustee can do" (i.e. publish only under the last segment, "build metadata").
This is useful because:
v5.1.4,v5.1.4-rc.1are published by authorsv5.1.4+registry1is published by a Trustee, and the difference is clearhdgarrood commentedon Apr 27, 2020
I don't think trustees should be permitted to do anything to a package that package authors aren't permitted to, other than perhaps unpublish them after the initial unpublish window has elapsed, for if they have content which raises legal issues (eg copyright) or is being used for harassment, or something like that. If we are using the semver build metadata exclusively for revisions, then I think we should allow package authors to publish revisions as well as trustees.
hdgarrood commentedon Apr 27, 2020
I think a lot of the contentiousness around this issue in the Haskell case comes from differing ideas about who is responsible for what. From what I've seen, Hackage trustees feel that it is their responsibility to make sure their package builds correctly, but package authors who are annoyed by revisions feel that it should be only theirs, and that if older versions don't build any more, that's fine, because they're unsupported. I'm not sure what the direct implications of this are for us, but I think it's worth being aware of. Perhaps the registry should take less responsibility for making sure things build, even?
f-f commentedon Apr 29, 2020
Good points. The issue with Haskell packages "not building" is that the compatibility with older compilers could just be a matter of bumping some version bounds (i.e. no syntactic changes), and Trustees keep an eye on the build matrix with different compilers, and want to keep that green.
I don't know if we do care about that usecase just yet (I don't), my main concern is "keeping packages installable both with package sets and with a solver", which we encountered in purescript/spago#396. Trustees being able to bump version bounds is the best way we could prevent that from happening IMHO. I would of course put timeouts on this (e.g. "if a package author is unresponsive for two weeks then a trustee is allowed to publish a revision"), as of course I'd like to minimize the amount of resentment between authors and trustees.
joneshf commentedon Apr 29, 2020
In case it's not clear, I'm neither for nor against the ability to edit a published version. I just don't understand why it's necessary.
Pretty much. I'm not asking about Trustees in specific, but about the entire idea.
In nearly every other registry, there's no way to edit a published version of a package and change dependencies. We're stating that we have to be able to edit a published version. There's nothing explaining why it's necessary that we edit published versions of a package instead creating new versions. There's nothing stating what we lose by removing this feature.
Since we seem to be talking past each other, I'm going to move on from this point.
Some package authors don't care if their package's dependencies change or versions become unavailable to them. Some either cannot allow that (for whatever reason), or don't want to (again, for whatever reason). We recognize there's a divide between package authors. Why not make Trustee editing something each author decides per package?
Trustee editing can be opt-in or opt-out by default. Instead of a Trustee being able to make any version changes, they can only do it on packages they're able to. If a Trustee is not allowed to edit a version, and a package has reached the timeout for changes, they remove the offending versions instead of making the change (or however we handle those things).
Since we are implementing this registry as "Just a GitHub repo," we can probably automate it with a combination of the
CODEOWNERSfile and required reviews to decide who can approve a change.hdgarrood commentedon May 8, 2020
This comment is worth reading: https://www.reddit.com/r/haskell/comments/gf7uw8/comment/fpv3dtg?context=1
I think the answer is that if you want solving to work, you need to have some kind of mutation somewhere, otherwise older versions with bad bounds will still break things, even if newer versions have good bounds. Registries which don’t support revisions will instead support “yanking,” which allows a maintainer to tell a solver not to consider a particular version any more when constructing build plans.
joneshf commentedon May 8, 2020
Cool. Can we add that to our documentation? The whole reason I asked about this is because there is no documentation here about why editing is a necessity.
f-f commentedon May 8, 2020
@hdgarrood wow thanks for that link, such a great post!
For context I'll also link to this twitter thread, as I find you drove the conversation nicely there
JordanMartinez commentedon May 9, 2020
I think Alexis' comment here explains the above comment on "yanking" vs "revisions" and pros/cons of each more clearly and succinctly than the original bog post that summarizes the FP Slack haskell channel: https://www.reddit.com/r/haskell/comments/gf7uw8/on_pvp_and_restrictive_bounds/fpv3dtg/
f-f commentedon Oct 11, 2020
@joneshf @hdgarrood @JordanMartinez I tried to address all the concerns in this issue in #76 (now merged).
I'll close this issues now, but I'd love if you could read through the updates and reopen if you notice I missed something there
joneshf commentedon Oct 12, 2020
I'm replying to acknowledge your message. But, I'm bowing out of the discussion.