Apps

MetaCPAN wants to support multiple projects and become a home for perl projects needing hosting.

K8 Infra on Digital Ocean (DO)

Implemented

• Argo CD - Automates application deployment and lifecycle management with GitOps
• Certs manager - To get and manage ssl certs from LetsEncrypt)
• SealedSecrets - So secrets can be encrypted and then put in git)
• PostGres (using DO Postgres)
• Basic PostGres backup solution (saving to B2) using SnapShooter
• NFS mount of DO volume - but this seems to take loads of CPU

Outstanding (or needs working example)

• Centralised Logging (using external data store, maybe B2)
• Monitoring (though we have some basic stuff with DCL)
• RAM disk definition, docs (you specify the medium as Memory)
• Decent PostGres Backup solution (still saving to B2)
• Sending emails from cronjob failures

Eventually would be nice to have:

• Disk options for Containers that only need 1 instance running in the cluster
  • Grep (a low priority service) needs about 40Gig can fetch again from github
  • BackPAN Syncer (a key service) would have about 150G and this must be very robust and not get lost if pod moves (currently a Hetzner volume, so hetzner VM can be recreated from scratch if needed and data not lost - it will be backed up as well)
  • NOTE: we could possibly use rclone mount and then a B2 obj store + local cache as a half way house?

Projects and migration status

• ES moved to cloud
• Postgres in DO available (username passed in as secret to containers)

Sites/services already migrated to DO

• mc:web https://metacpan.org/
• api-v0-shim: https://api-v0-shim.metacpan.org
• Search CPAN Redirect - https://search.cpan.org/ ( sco-redirect )

Sites/services NOT migrated to DO

Requirement	Priority	Status	Notes
Postgres Backup	Medium	TODO	Using SnapShooter for now - and no mission critical data
mc:api - web server	High		Switch to HTTPS for package src, setup ram disk
mc:api - cron	High		About 30 of them ( see mc-ingest project for some of them )
mc:api - worker queues	High		Uses Minion PG - various jobs distributed across servers for speed (e.g. reindex all) + monitor minion queue code puppet
mc:api - watcher	High		Needs access to latest cpan.org changes to create jobs, see below
cpan.mc.org / backpan syncer	High	WIP	Move to B2 + fastly Plan
mc:web cron	Low	TODO	sitemaps - store somewhere
mc explorer	Low		Easy to docker
backup cpan cover	Low	?	Currently 30G on bm-mc-03 /mnt/lv-cpancover-html