Installers database?

[jackfirth]

Thus far, I know of four different places to download Racket installers:

• https://download.racket-lang.org/ - The "official" download source
• https://mirror.racket-lang.org/installers/ - Some sort of backup site
• https://pre.racket-lang.org/installers/ - Prerelease installers only
• https://www.cs.utah.edu/plt/snapshots/current/installers/ - Installers for snapshot builds (allows substituting a snapshot ID for "current")

This makes automating installation a little hacky, as automation scripts need to know which site to use for which versions. There's a few other wrinkles:

• Not all installer sites provide installers for the same platforms
• Some installer sites are "archived" and always provide installers for the same version, whereas others are "rolling" and provide installers for the latest release / snapshot / etc.
• The "archived" installer sites put the version number in the path, even though installers already have the version number in the filename
• The installer filename format has not stayed constant over time
• Some installer sites provide "versionless" installers, but not all do

Automation could be simpler if there was some database somewhere that served as a canonical source of truth, mapping Racket versions (including snapshot versions like 6.11.0.2), platforms, initial package configurations, and download formats to full URLs with clients not needing to care about how filenames are constructed and which sites to use.

[samth]

Note that there's also https://plt.eecs.northwestern.edu/snapshots/

Also, mirrors.racket-lang.org is the site that actually hosts the binaries on download.racket-lang.org (we do this because the transfer costs on S3 are substantial otherwise).

[samth]

More broadly, I'm not really sure what you're suggesting. Can you say more about what this database would look like and how it might work?

[jackfirth]

I think I want some HTTP URL where a GET request to that URL yields something like this:

Content-Type: application/vnd.racket.installers+json
[{
  "kind": "release",
  "version": "6.11",
  "format": "script",
  "system-type": {
    "os": "unix",
    "word": 64,
    "link": "static"
  },
  "url": "https://mirror.racket-lang.org/installers/6.11/racket-6.11-x86_64-linux-natipkg.sh",
  "packages": "minimal"
}, {
  "kind": "snapshot",
  "version": "6.11.0.2",
  "format": "source",
  "packages": "main-distribution",
  "url": "https://www.cs.utah.edu/plt/snapshots/current/installers/min-racket-6.11.0.2-src.tgz"
},
...
]

That way I could write scripts and tools that:

• Look for new installers using a single well-known canonical source of truth
• Know what version of Racket each installer provides
• Are completely ignorant of installer filename formatting and treat the download URL as an opaque identifier
• Know what sorts of (system-type) values would be produced by the different installers, enabling platform-specific decisions
• Can scope their operations to only official release installers, snapshot installers, or prerelease installers
• Can know which installers include pre-built main distribution packages and which don't

[samth]

I agree that this would be nice, but right now these services are not coordinated nearly enough to produce that. I could imagine creating a new service which allowed each build to post things to that service, and then the service responded to the queries, but someone would have to build that.

Note also that for most of these, builds go away without that build explicitly being involved, as in the pre-releases.

[jackfirth]

I agree that this would be nice, but right now these services are not coordinated nearly enough to produce that. I could imagine creating a new service which allowed each build to post things to that service, and then the service responded to the queries, but someone would have to build that.

Agreed. Maybe there should be a "wishlist" github label...

Note also that for most of these, builds go away without that build explicitly being involved, as in the pre-releases.

That should be fine for the endpoint I sketched out. Do new snapshots and prerelease builds always have new Racket versions associated with them?

[jackfirth]

Note: most recent nightly (I think) and stable release version info is available at http://download.racket-lang.org/version.txt (found in racket/racket#1500)

[jackfirth]

Another note: this would solve some pain I'm hitting right now in using the docker-compose setup with snapshot builds. For a 64-bit minimal Linux installation the mirror.racket-lang.org/installers/6.11 site uses the name racket-minimal-6.11-x86_64-linux.sh, but the www.cs.utah.edu/plt/snapshots/current/installers site uses the name min-racket-6.11.0.2-x86_64-linux-precise.sh. Sigh.

[jackfirth]

Yet another note: there's also the GitHub Releases source code downloads. But I'm not sure if those are "canonical" and the download.racket-lang.org installers are built from them, or if it goes the other way around, or where the release catalogs for main distribution packages fits into that relationship.

[samth]

I don't think those GitHub releases are intended for use at all, and I didn't even know we were creating them. Maybe it happens automatically with tags? @stamourv would know more.

[stamourv]

We do use tags to mark releases, and not just in the main repo.
As far as I can tell, Github is just creating "releases" based on that. I did not see an option that could turn it on or off.

[jackfirth]

I think it's just the GitHub UI being confusing. If you look at a project with actual releases, the UI is quite a bit different. Maybe the "releases" page shows tags only for repos that don't have any releases?