Package maintainers

We are planning some API operation that would perform sensitive actions directly on this repo, such as:

unpublishing a package
change the repo address

In order to err on the safe side, we could only authorize the @purescript/packaging team to perform such operations.
However, it would be nice if we could allow packages' maintainers to perform such operations too.

The main issue to solve if we are to allow this is: how do we authenticate them?
I.e. how do we know that "Person X" is really a maintainer of package Y?

In order to tie together the entities of "package" and "maintainer" we could store a list of maintainers in the Manifest: since they have to be committed directly to the repo then "having write access" means "I can add myself as a maintainer", and this proves the relation.
However, how do we link "I'm a maintainer" to "I can perform an operation on the registry"?

An option for this would be to store GitHub usernames in the maintainers list: in this way if user X opens an issue to perform some operation then we can just check if they are in the maintainers' list, and safely authenticate them.
Does this tie use too much to GitHub?

A more agnostic option would be to use email addresses instead, but then authenticating them becomes more cumbersome?

I think it's acceptable for the first cut of the registry to require unpublishing / renaming to go through the packaging team -- if we don't come to a solution by the time we're ready to launch.

For some extra reading, here's npm's policy on unpublishing, and I remember that Bower disabled unpublishing for a while due to the security risk of unpublishing and then someone publishing a new package at the same name.

Agreed that we don't need to get to this right away, but the authentication problem still stands, even if we don't automate this: the packaging team will need to verify that someone requesting unpublishing is actually the person they claim to be.

As for the definition of "unpublishing": the way this term is used throghout the spec is about being able to unpublish a single version of a package, and never the package name. Once you take a package name, that's forever. I feel we should probably make this clearer in the spec?

We are planning some API operation that would perform sensitive actions directly on this repo, such as:

unpublishing a package

change the repo address

Regarding this a couple of questions:

Is changing repo address supposed to be per version? Because potentially original sources for different versions may reside in different repos eventually, this esp would be actual when monorepo based development will be used.
Also there other types of operations for example like package deprecation/undepreciation, and also package visibility, ect. I think such things or the possibility of the future introduction of such things should be foreseen in the architecture.

@thomashoneyman

Is changing repo address supposed to be per version? Because potentially original sources for different versions may reside in different repos eventually, this esp would be actual when monorepo based development will be used.

Actually we found a way to do this without an additional command while talking through this with @thomashoneyman. The rough algorithm is: to change the repo address where a package is fetched from the authors could just change it in their package manifest, and we'd update the repo address here in the metadata when updating a package - this will be used to fetch the package sources from the next version on.

I think such things or the possibility of the future introduction of such things should be foreseen in the architecture

Agreed. We already take care of this because the Operation type is extensible with new constructors.

A more agnostic option would be to use email addresses instead, but then authenticating them becomes more cumbersome?

I very much like this +1. Even if using GitHub, people may prefer alternate means of communication. I'm curious why there isn't a separate maintainer list used as keys in packages instead of having a file structure based on GitHub user IDs.

Personally I'd like to see a union something along the lines of this (similar to nixpkgs minus the github ID stuff)

let Key = { Type = { long-key-id : Text, fingerprint : Text } }

let Maintainer =
      { Type =
          { name : Text -- preferred name
          , email : Text -- Optional? Falling back to..?
          , pronouns : List Text
          , keys : List Key.Type
          }
      , default =
        { pronouns = [] : List Text
        , keys = [] : List Key.Type
        }
      }

in    { toastal = Mainainer::{
        , name = "toastal"
        , email = "toastal+purs@posteo.net"
        , pronouns = [ "he/him" ]
        , keys =
          [ { long-key-id = "ed25519/5CCE6F1466D47C9E"
            , fingerprint = "7944 74B7 D236 DAB9 C9EF  E7F9 5CCE 6F14 66D4 7C9E"
            }
          ]
        }
     -- | etc...
      }
    : Type

But I suppose a Map or other structure would make sense.

@toastal

@toastal I'd prefer if we could be entirely GitHub agnostic, but to me "authenticating people" is a more pressing problem than "being agnostic", and piggybacking on GH's authentication allows us to solve this issue with very little effort - so any proposal on this will need to evaluate the tradeoff between "agnosticism" and "effort needed to achieve that".
The bottom line is that we're a team of unpaid volunteers, and any kind of infrastructure maintenance is something that requires comparatively high effort for very little reward (people only notice if things break), and I feel that anything that allows us to lighten this burden is welcome

During yesterday's Registry Weekly we discussed this issue, and in particular how to "authenticate" maintainers to do sensitive operations on their packages.
There was some brainstorming about this, as we don't yet have a solution. From the discussion it emerged that:

we can't really use GitHub usernames because of security and platform-agnosticism concerns
we don't want to use GPG because that's a source of additional friction for library authors, and we'd like to avoid that
we'd like to avoid rolling our own authentication servers

Now, in light of the discussion I have a proposal that fits all these criteria, and it's based on the idea proposed above for changing the URL we store for repos.
The idea is to authenticate maintainers by exploiting information provided through the only channel we can already trust them on: releases. In the case of the proposal above it was about "grabbing the new URL from the new release".
However! If we use this channel for providing a public key, then we can verify their identity on any other channel by asking them to sign things with their private key.
Since we deemed GPG inconvenient, we have to turn to some asymmetric cryptography system that is widespread enough to not be inconvenient for users. It turns out we have one, it's SSH!
The vast majority of users already use SSH to push to their repositories, so the setup cost is zero, and it turns out it's possible to use SSH to sign and verify things - a good walkthrough of how that works can be found in this blog post.

This is how such authentication would work (details for the specific ssh commands are omitted here, refer to the blog post for that):

Maintainers could (or would be required to) add an allowed signers file in the root of their repo, that looks something like this:
```
alice@example.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINq4q0EHXacxMzmcG7TNC1DJpSxpK5dhJA6uAlZ/x8O7
bob@example.net ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBgQuuEvhUXerOTIZ2zoOx60M/HHJ/tcHnD84ZvTiX5b
```
This file contains their public keys, associated to some email, in a format that SSH can use for verification.
Whenever a new release of a package happens, we'd fetch this file and store the contents in the Metadata so they are always ready to use. An alternative to this could be also storing the information (it doesn't need to be exactly in this format, we can generate this file) in the Manifest, and fetching it from the registry-index when needed. The exact details don't matter much right now, the point is that we always know the latest signatures of the maintainers of a package.
Whenever a maintainer would like to perform a sensitive operation, they would (or well, presumably the package manager would do that for them) create a payload with the data necessary for the operation (e.g. "unpublish version X.Y"), sign that with their private key, and attach the signature and the email to this payload.
Then our CI would lookup the right public key by this email, verify the signature, and perform the operation if the signature is valid ✨

I am interested in using SSH keys to verify package ownership. Before going any further, though, I'd like to clarify the question of package ownership.

Package ownership matters for several reasons:

We need to know who is authorized to take specific actions related to a package
We need to know who to contact in the case of a security incident or in the case that a registry trustee needs to adjust a package's metadata

We'd like to accomplish these goals while staying as user-friendly as possible.

Package ownership (wrt authorization) matters when:

A package is pushed to the registry for the first time (the registry then pulls the source, packages it up, stores the result, and adds a new metadata entry / registers the package name) -- corresponding to an Addition operation
A package version is pushed to the registry (the registry then pulls the source, packages it up, and stores the result) -- corresponding to an Update operation
A package version is removed from the registry (the registry drops the version from storage backends and the registry index) -- corresponding to an Unpublish operation

Only a package owner should be able to add a package to the registry, add new versions of the package, or unpublish the package. And package ownership should be reflected across package versions (ie. ownership isn't tied to a specific version, but rather to the package itself).

There are also operations related to ownership that we probably need to support:

We need to know who the owner is when they first upload the package, so that we can authenticate subsequent uploads
We need to let owners add other owners, remove other owners, change their ownership information (for example, change their email), and (potentially) remove themselves as owners.

With all of this in mind, I like the idea of SSH keys paired with email addresses can be used for authentication. We get the public key for authentication and we get the email address for communication.

Maintainers could (or would be required to) add an allowed signers file in the root of their repo

Whenever a new release of a package happens, we'd fetch this file and store the contents in the Metadata so they are always ready to use. An alternative to this could be also storing the information (it doesn't need to be exactly in this format, we can generate this file) in the Manifest, and fetching it from the registry-index when needed.

I'd like to find an alternative to requiring yet another file in the root of someone's repository, as we're already asking them to have a purs.json and most likely spago.dhall file there.

I'm also wary of storing this information in manifest files, as that seems to imply it is per-version. Presumably we'd always ignore old manifests and only look at the latest manifest to know owner information?

To me it feels like the right place for this information to live is in the package metadata, as you already mentioned.

Also, we need to support changes in ownership, as I mentioned before. What if we took a page out of Cargo's book and added a new operation to the API that deals with ownership, like cargo owner? This operation would be specifically for manipulating the Metadata field related to ownership. You could add, change, and remove owners via this operation, but only if you were already an owner.

To summarize, I'm proposing that if we go forward with SSH-based ownership that we then store ownership information in the package metadata and provide an API for changing ownership information, without it having to be tied to publishing a new package version.

I also have a few followup questions:

How do we know that the initial package upload is coming from its rightful owner? We can't just assume that the uploader is the owner, necessarily.
What recovery options does an owner have should they lose access to their email or private key?

I feel that these comments/questions come from thinking our publishing model as "authorized people upload things" - but it's not how it happens: people (anyone, they don't have anything to do with package authors/maintainers!) can ask the Registry "hey can you package up this ref from this whatever repo on the internet?", and the Registry CI will do that.

This has a few implications on the premises of your proposal:

our only trusted channel of communication with package authors/owners/maintainers (all of which I'll call "owners" from now) is a release. I.e. whenever we go to a repo and package a specific ref then we establish a chain of trust with whoever pushed that ref. We can do this because GitHub already established an authentication from the real person to the online place where we fetch stuff from
there is no difference in the amount of trust that this channel has in the case of first-time-uploads vs just releasing new versions. We trust whoever has pushed that ref (and only them) in either case. The only difference is that a first-time-upload also registers a name, but it's not a security/trust concern because the ownership/trust belongs to whoever can push refs, not the person who asked the Registry to register the package.
This is why we can just take and package up all the old packages - we don't need/gain any ownership of these, nor we need to ask permission.

These premises clarified, I'll now address specific details:

Only a package owner should be able to add a package to the registry, add new versions of the package, or unpublish the package

We care about authentication only for unpublishing because that might contain sensitive information that should be taken out of circulation ASAP, and because that can break builds. As noted above we entitle everyone to ask the registry to add new things - and I don't see why we shouldn't - since that doesn't assign them any ownership or authentication. Just like uploading Pursuit docs right now.

We get the public key for authentication and we get the email address for communication

I realize that the fact that my proposal entails email addresses is a bit unfortunate, but that's an artefact of how SSH handles this stuff.
These email addresses are only used to look up the right public key - they don't need to exist at all, and we wouldn't ask owners to make sure that they work, nor we would lean on them to reach out for sensitive matters.
It's great if they also work and if they answer us in case we reach out there, but verifying that they keep working is really not possible unfortunately. I think our main channel to contact them should still be "GitHub issues", as it is today.

I'd like to find an alternative to requiring yet another file in the root of someone's repository, as we're already asking them to have a purs.json and most likely spago.dhall file there

Yeah I share your feeling, which is why I proposed that we include this data in the Manifest instead of requiring a new file. It could look something like:

..
  "owners": [
    { "email": "alice@example.com"
    , "keytype": "ssh-ed25519"
    , "public": "AAAAC3NzaC1lZDI1NTE5AAAAINq4q0EHXacxMzmcG7TNC1DJpSxpK5dhJA6uAlZ/x8O7"
    }
  ]
..

I'm also wary of storing this information in manifest files, as that seems to imply it is per-version. Presumably we'd always ignore old manifests and only look at the latest manifest to know owner information?
To me it feels like the right place for this information to live is in the package metadata, as you already mentioned.

As I already noted on Discord, I don't see why it would imply that it's per-version. Plenty of package ecosystems (JS to name one) store "authors" in a "per-version" package manifest, and I don't think the majority of people would assume that the ownership is limited to that specific version?

The reason why we need to pass this info through a release is because - as noted in premise (1) - that's our only trusted communication channel with package authors.

And yes, we'd look at the latest version for the latest updates in package ownership...

Also, we need to support changes in ownership, as I mentioned before. What if we took a page out of Cargo's book and added a new operation to the API that deals with ownership, like cargo owner? This operation would be specifically for manipulating the Metadata field related to ownership. You could add, change, and remove owners via this operation, but only if you were already an owner.

...which is exactly how we'd enable changes in ownership.

I'll present an example to show how this would work:

Alice is the only owner of package Foo
someone (doesn't need to be her) publishes foo to the Registry, let's say the latest version is 1.2.3
Alice is tired of maintaining the package and finds that the fellow PureScripter Bob is interested in taking over
she then hands over the package to Bob on GitHub, i.e. transferring ownership to him on GitHub
Bob carries on with development and releases version 1.2.4 to the Registry
he then realizes he accidentally committed some sensitive credentials in there, and would like to take this version down!
by reading our docs he learns that he can do such a sensitive operation only by including his public ssh key in the next release. So he adds it to the Manifest, and releases 1.2.5
he then runs spago unpublish 1.2.4. Spago will generate a payload containing an Admin operation (note, this doesn't exist yet, but it would be a new constructor on the Operation) containing:
- the Unpublishing sub-operation
- the content of the above signed with his ssh key
- the email that he registered his public key under
our CI (that is now aware of his public key because it got it in version 1.2.5) can verify this signature and unpublish version 1.2.4

A few notes:

Alice didn't need to get involved at all, nor she needed to change anything in the repository. She just transferred the repo on GitHub
..which means that we don't need at all to deal directly with changes of ownership, we use the underlying git-thingy: as long as someone can push to a repo at a certain URL then we are going to trust the content that they push
..but we are adding a new API operation to deal with sensitive matters in general 🙂

I also have a few followup questions:

How do we know that the initial package upload is coming from its rightful owner? We can't just assume that the uploader is the owner, necessarily.

What recovery options does an owner have should they lose access to their email or private key?

I answered these in longer form above, but to summarize:

we don't need to know/trust who is asking the Registry to publish something
we don't require access to the email (which is only used for looking up the key), and if someone loses access to the key then they can hopefully push an update to the repo in some other way or with some other key. If they can't then they probably lost access to their GitHub account, which is, uh, a bigger problem that we can't really do much about 🙂

I feel that these comments/questions come from thinking our publishing model as "authorized people upload things" - but it's not how it happens: people (anyone, they don't have anything to do with package authors/maintainers!) can ask the Registry "hey can you package up this ref from this whatever repo on the internet?", and the Registry CI will do that.

I didn't know we were doing this, exactly. I'm aware that no one is uploading anything (the Registry fetches and packages the repo), but I wasn't aware that I could, for example, ask the registry to upload any package I want. I guess I always assumed that the package owner is the only person who can choose when they'd like a version of their package to be made available to the registry.

That said, I'm open to doing this. After all, the pull will fail if they haven't properly set up their repository to be added to the registry, which implies that they meant it to be available there. But this should be loud in the spec.

Our only trusted channel of communication with package authors/owners/maintainers (all of which I'll call "owners" from now) is a release. i.e. whenever we go to a repo and package a specific ref then we establish a chain of trust with whoever pushed that ref. We can do this because GitHub already established an authentication from the real person to the online place where we fetch stuff from

The authentication you're referencing here is just that the person who pushed the commit has access to the repository, and therefore we can assume they are in fact an "owner" of the release they've pushed?

This sounds good, but I have some clarifying questions to be sure it will work the way we intend.

It doesn't sound like the chain of trust is only from the registry to the person who pushed the ref; isn't the point of the owners field to allow us to trust more than just this person? Wouldn't we take the contents of the owners field, regardless of who actually pushed the commit?
It's explicitly a design goal to support more than GitHub as a source of packages. Presumably we can also trust providers like Gitlab; should we generally take the stance that if someone has pushed a ref to a repository and then requested that ref be published, then we trust it?

Yeah I share your feeling, which is why I proposed that we include this data in the Manifest instead of requiring a new file.

And yes, we'd look at the latest version for the latest updates in package ownership...

I've spent some more time thinking about this, and I agree that including the data in the manifest is the right way to go. We don't have a reliable way to get their public key otherwise. And we can just take the latest version to be the 'current' ownership information.

Having slept on it, I'm warming to this idea more and more! Despite my questions above, this does feel like the right way to go forward.

Since ownership is only required for unpublishing, should this be an optional field? You can add it when you need it because you’re unpublishing, but not everyone has to go add this information to their package manifest then — a big win for convenience.

purs.json

Sorry to slightly derail, but why not Dhall if Dhall is used in Spago and can be used to guarantee certain types are fulfilled and correct?

+1 to all mentions of supporting alternative Git forges

@toastal

@toastal the initial design of the Registry had everything in Dhall. We found out that was a bad tradeoff, because it made the Registry harder to access for clients written in languages that don't have a Dhall implementation (Spago does not hold a privileged status, it's just a client that has first class support from the team, but we can't/shouldn't/won't design things around it).

Since "typechecking" was the main Dhall feature we were using back then we settled on having the manifests themselves in JSON, and typecheck them against Dhall types with json-to-dhall, which seems to work very well so far

Ah! I'm sorry, I for some reason was thinking we were using the signature from a Git commit to verify ownership, but of course that wouldn't be relevant for something like unpublishing. Upon re-reading I see you said we would get a signature for the payload itself. I'm back on board!

I'll assign myself to this, I'll put together a small PoC of the ssh signing

So, here's a fully-sketched-out flow - you should be able to run all of the following commands locally to verify that this works as it should.

API interface

We'd add an AuthenticatedOperation type and change the Operation type to look something like this:

let Repo = ./Repo.dhall

let AuthenticatedOperation = < Unpublish : { packageName : Text, unpublishVersion : Text, unpublishReason: Text } >

in
  < Addition : { packageName : Text, newPackageLocation : Repo, newRef : Text, addToPackageSet: Bool }
  | Update : { packageName : Text, updateRef : Text }
  | Authenticated : { authenticatedPayload : AuthenticatedOperation, signature : List Text, email : Text }
  >

Flow

Let's imagine a scenario where the owner of the package foo wants to unpublish the version 1.2.3.

They would need to generate a pair of ssh keys to accomplish this - let's do that:

ssh-keygen -t ed25519 -C "owner@foo" -f ~/.ssh/spago

That gives us a private key in the ~/.ssh/spago file:

and a public key in the ~/.ssh/spago.pub file:

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINWzakVhHUDnIRmchzbiQgsxP2a+CsHkIMvPbboLMYOE

Cool. At this point we can add the public key to the owners field of the manifest, something like:

..
  "owners": [
    { "email": "owner@foo"
    , "keytype": "ssh-ed25519"
    , "public": "AAAAC3NzaC1lZDI1NTE5AAAAINWzakVhHUDnIRmchzbiQgsxP2a+CsHkIMvPbboLMYOE"
    }
  ]
..

Then we push a new release (1.2.4 I guess), so that the Registry CI saves this key in the metadata.

Once that goes through we can now run spago unpublish 1.2.3

That generates an AuthenticatedOperation payload (in JSON):

{ "packageName": "foo", "unpublishVersion": "1.2.3", "unpublishReason": "committed bad credentials" }

Then puts this in a file and signs that with the private key (which we assume will be in ~/.ssh/spago, but I guess we could make that configurable):

ssh-keygen -Y sign -f ~/.ssh/spago -n file unpublish.json

Which produces a file containing the following:

-----BEGIN SSH SIGNATURE-----
U1NIU0lHAAAAAQAAADMAAAALc3NoLWVkMjU1MTkAAAAg1bNqRWEdQOchGZyHNuJCCzE/Zr
4KweQgy89tugsxg4QAAAAEZmlsZQAAAAAAAAAGc2hhNTEyAAAAUwAAAAtzc2gtZWQyNTUx
OQAAAECv2gem8zQUFx5lX8KxPht9VobMMv2w9/oMjnxWfkKtVwkkeSZM1isD8qZ233g8OE
a9n6hpd1Jg17MlTNAXSa4O
-----END SSH SIGNATURE-----

The end result of this is that Spago generates a link to open a GH issue (just like spago publish will do), which will contain the following JSON:

{
  "authenticatedPayload":
    { "packageName": "foo", "unpublishVersion": "1.2.3", "unpublishReason": "committed bad credentials" },
  "signature": [
    "U1NIU0lHAAAAAQAAADMAAAALc3NoLWVkMjU1MTkAAAAg1bNqRWEdQOchGZyHNuJCCzE/Zr",
    "4KweQgy89tugsxg4QAAAAEZmlsZQAAAAAAAAAGc2hhNTEyAAAAUwAAAAtzc2gtZWQyNTUx",
    "OQAAAECv2gem8zQUFx5lX8KxPht9VobMMv2w9/oMjnxWfkKtVwkkeSZM1isD8qZ233g8OE",
    "a9n6hpd1Jg17MlTNAXSa4O"
  ]
  "email": "owner@foo"
}

The CI will receive this payload, and need to validate that the signature for the authenticatedPayload has been made with the public key of owner@foo. Let's do that.

We first need to create an allowed signers file with the key(s):

owner@foo ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINWzakVhHUDnIRmchzbiQgsxP2a+CsHkIMvPbboLMYOE

Then we can put this payload in a file (e.g. payload_to_verify), the signature in another one (e.g. payload_signature), and verify the signature with:

ssh-keygen -Y verify -f allowed_signers -I owner@foo -n file -s payload_signature.sig < payload_to_verify

...which will happily return:

Good "file" signature for owner@foo with ED25519 key SHA256:IBScnmaUtbKGyLk8IvjERdJnXYRCSLMWVnj2vt4OnLs

Et voilà, our signature has been authenticated! ✨ 🎉

👏 Looks good! How do you feel about someone implementing this in the API pipeline so that's there and testable? (I can do it, if you're pressed for time.)

@thomashoneyman

@thomashoneyman I'd love that!
My time is extremely scarce at the moment, so I'm trying to allocate it to the most urgent things first - mostly the design discussions, which doesn't leave much time for implementing unfortunately

I fully understand, the design decisions take all the effort 😅

Bumping with a new question: Why Optional (List Owner) over List Owner? What is the difference between None : List Owner and Some ([] : List Owner)?

As far as the schema goes, this is representing that the field itself is optional — it is not necessary to provide an “owners” field in the manifest.

@f-f

@f-f should we close this? we've long supported authenticated operations via SSH signatures (unpublish, transfer) as well as authenticated operations performed by the purescript/packaging team at this point, and spago@next uses the same ssh library to authenticate packages that the registry does so they're in sync.

Yeah I think this is done - though the work in Spago is not through yet

Navigation Menu

Repository navigation

Package maintainers #155

4 remaining items

API interface

Flow

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Package maintainers #155

Description

Activity

thomashoneyman commented on Apr 21, 2021

f-f commented on Apr 21, 2021

wclr commented on May 8, 2021

f-f commented on Aug 18, 2021

toastal commented on Aug 30, 2021

f-f commented on Aug 30, 2021

f-f commented on Jan 28, 2022

thomashoneyman commented on Jan 29, 2022

f-f commented on Jan 29, 2022

thomashoneyman commented on Jan 29, 2022

thomashoneyman commented on Jan 29, 2022

thomashoneyman commented on Jan 29, 2022

toastal commented on Feb 1, 2022

f-f commented on Feb 1, 2022

4 remaining items

thomashoneyman commented on Feb 6, 2022

f-f commented on Feb 8, 2022

f-f commented on Feb 9, 2022

API interface

Flow

thomashoneyman commented on Feb 9, 2022

f-f commented on Feb 9, 2022

thomashoneyman commented on Feb 9, 2022

toastal commented on Jun 19, 2022

thomashoneyman commented on Jun 19, 2022

thomashoneyman commented on Nov 6, 2023

f-f commented on Nov 6, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions