[thomashoneyman]

Closes #233, #220, #218. Supersedes #234.

This PR implements a LegacyImport script instead of a BowerImport script, because we now look at multiple files to try and import a manifest instead of just the Bowerfile. Those files include the LICENSE, package.json, bower.json, and spago.dhall and packages.dhall files.

Implementing the pipeline using multiple files to produce the manifest required shuffling around quite a bit, but the majority of the code moved into the constructManifestFields function which produces the license and dependencies for a package from a combination of the available files.

There are two other notable changes:

1. The withCache function works a little differently. First, it no longer requires EncodeJson and DecodeJson instances, because you can be storing something like a LICENSE file; it now uses String. Second, the withCache function now takes an ImportError -> Boolean function that allows you to cache files even if producing the file throws an exception. The exception is stored instead of the file contents, and when the file is read the exception is thrown. With this in place we can remove all the filterFailedPackages stuff, which is a bit error-prone, and we no longer need to read in the excluded packages.
2. The Error type has been changed around a little to accommodate the new files we're fetching. There's now a ResourceError that represents that files related to the package couldn't be fetched. MissingBowerfile has been replaced since it's not really an error any more, and NoDependencyFiles took its place to represent that both a spago and bower file are missing.

[thomashoneyman]

These are the results, working from a .cache with only releases cached:

Number of successful packages: 1448
Number of failed packages: 375
Number of successful versions: 10234
Number of failed versions: 1281
Failures by error:
  manifestError: 1214 occurrences (314 packages / 1214 versions)
    badLicense: 480 occurrences (76 packages / 480 versions)
    missingLicense: 452 occurrences (141 packages / 452 versions)
    badDependencyVersions: 307 occurrences (114 packages / 307 versions)
    badVersion: 50 occurrences (34 packages / 50 versions)
  noDependencyFiles: 66 occurrences (34 packages / 66 versions)
  resourceError: 32 occurrences (32 packages / 0 versions)
  malformedPackageName: 1 occurrences (1 packages / 1 versions)

It's a significant improvement! Compare that to the current main branch:

Number of successful packages: 1369
Number of failed packages: 474
Number of successful versions: 9565
Number of failed versions: 1694
Failures by error:
  manifestError: 1378 occurrences (371 packages / 1378 versions)
    missingLicense: 928 occurrences (241 packages / 928 versions)
    badDependencyVersions: 283 occurrences (104 packages / 283 versions)
    badLicense: 179 occurrences (45 packages / 179 versions)
    badVersion: 48 occurrences (32 packages / 48 versions)
  nonRegistryDependencies: 246 occurrences (47 packages / 246 versions)
  missingBowerfile: 64 occurrences (34 packages / 64 versions)
  noReleases: 32 occurrences (32 packages / 0 versions)
  malformedBowerJson: 5 occurrences (4 packages / 5 versions)
  malformedPackageName: 1 occurrences (1 packages / 1 versions)

The number of missing licenses has been cut in half, and about 700 more versions are included in the final registry. I would like to keep looking at the exclusions file to see if any packages are suspiciously failing, but on the whole this is a nice improvement over the current stats.

[thomashoneyman]

The tests are failing because I added a Licensee test; we’re going to have to add that to CI as a CLI tool.

[thomashoneyman]

Whew, sorry, had to do a little CI fixup (case sensitivity is a bear!), but this is now actually good to go.

[f-f]

Great work @thomashoneyman! I left a bunch of small comments, and this is almost ready to go

[thomashoneyman]

Some of my responses to your comments are only showing up in the review, and not in this main PR view, so you might have to read them in the review tab. Thanks for your comments! I believe I've addressed everything.

[f-f]

Thanks for addressing the feedback! I left a few minor comments, and I'll approve this since it's basically good to go 🙂

[f-f]

Great! Then since we repeat this pattern many times I was wondering if it'd be worth making a helper that takes, does stuff, and then puts, so that we reduce clutter. No strong opinion on this though, feel free to ignore

[f-f]

Oh, I didn't realize then - thanks for the pointer. I have a slight preference for the lowercase version (I am definitely traumatized by macOS' filesystem shenanigans with that) so I'd appreciate if you can switch to that, but no worries if you wouldn't like to