diff --git a/dev-handbook/README.md b/dev-handbook/README.md index d12aafb..9dc35cd 100644 --- a/dev-handbook/README.md +++ b/dev-handbook/README.md @@ -40,6 +40,7 @@ * [Adding support for a new distribution](add-new-distro.md) * [Adding support for a new Ruby version](add-new-ruby-version.md) + * [Archiving EOL packages](archiving-eol-packages.md) ## Organizational (for team members) diff --git a/dev-handbook/apt-yum-repo-infra.drawio.svg b/dev-handbook/apt-yum-repo-infra.drawio.svg index f125c00..3f50073 100644 --- a/dev-handbook/apt-yum-repo-infra.drawio.svg +++ b/dev-handbook/apt-yum-repo-infra.drawio.svg @@ -34,7 +34,7 @@ - fullstaq-ruby-server-edition-apt-repo + fsruby-server-edition-apt-repo
@@ -81,7 +81,7 @@
- fullstaq-ruby-server-edition-yum-repo + fsruby-server-edition-yum-repo
diff --git a/dev-handbook/apt-yum-repo.md b/dev-handbook/apt-yum-repo.md index fb265d3..ad25346 100644 --- a/dev-handbook/apt-yum-repo.md +++ b/dev-handbook/apt-yum-repo.md @@ -48,11 +48,15 @@ Our design was made with the following requirements in mind, and addresses the f We self-host our repositories: we store them in Google Cloud Storage buckets. [Google Cloud Storage is strongly consistent.](https://cloud.google.com/storage/docs/consistency) -There are three buckets: +There are five buckets: - - `fullstaq-ruby-server-edition-apt-repo` stores the production APT repository. - - `fullstaq-ruby-server-edition-yum-repo` stores the production YUM repository. - - `fullstaq-ruby-server-edition-ci-artifacts` stores the temporary repositories created during CI runs. + - `fsruby-server-edition-apt-repo` stores the production APT repository. + - `fsruby-server-edition-yum-repo` stores the production YUM repository. + - `fsruby-server-edition-apt-repo-archive` stores packages for EOL distributions (APT). + - `fsruby-server-edition-yum-repo-archive` stores packages for EOL distributions (YUM). + - `fsruby-server-edition-ci-artifacts` stores the temporary repositories created during CI runs. + +The archive buckets are only updated during [EOL migration](archiving-eol-packages.md) — CI never writes to them. Each migration creates a new version that merges newly-archived distros with the existing archive contents. They are served at `apt-archive.fullstaqruby.org` and `yum-archive.fullstaqruby.org`. We don't let users use the production bucket URLs directly. Instead, we let users use `https://apt.fullstaqruby.org` and `https://yum.fullstaqruby.org`. These domains redirect to the appropriate bucket URLs. We do this so that we avoid strongly coupling users with Google Cloud Storage. If in the future we want to move off Google Cloud, we can do so without breaking users' URLs. @@ -144,7 +148,7 @@ A downside of this versioning approach is that each version consumes a lot of sp ### CI bucket -During CI runs, we create temporary repositories in `gs://fullstaq-ruby-server-edition-ci-artifacts/$CI_RUN_NUMBER/{apt,yum}-repo`. These directories look as follows: +During CI runs, we create temporary repositories in `gs://fsruby-server-edition-ci-artifacts/$CI_RUN_NUMBER/{apt,yum}-repo`. These directories look as follows: ~~~ /$CI_RUN_NUMBER/{apt,yum}-repo/ @@ -162,7 +166,7 @@ These directories are very similar to the production buckets' contents. But ther The CI tests run directly against this bucket URL instead of going through `https://{apt,yum}.fullstaqruby.org`. -We create a temporary repository by copying over `gs://fullstaq-ruby-server-edition-{apt,yum}-repo/versions/$LATEST_VERSION`. This way we achieve production data parity during testing. +We create a temporary repository by copying over `gs://fsruby-server-edition-{apt,yum}-repo/versions/$LATEST_VERSION`. This way we achieve production data parity during testing. ## Locking @@ -179,10 +183,10 @@ The lock's critical section is quite large, and covers: The locks are located in the following URLs: - * `gs://fullstaq-ruby-server-edition-apt-repo/locks/apt` - * `gs://fullstaq-ruby-server-edition-yum-repo/locks/yum` + * `gs://fsruby-server-edition-apt-repo/locks/apt` + * `gs://fsruby-server-edition-yum-repo/locks/yum` -When publishing to a testing repository (in `fullstaq-ruby-server-edition-ci-artifacts`) we don't perform any locking, because each CI run is guaranteed to write to its own temporary repository. +When publishing to a testing repository (in `fsruby-server-edition-ci-artifacts`) we don't perform any locking, because each CI run is guaranteed to write to its own temporary repository. ## Backups diff --git a/dev-handbook/archiving-eol-packages.md b/dev-handbook/archiving-eol-packages.md new file mode 100644 index 0000000..619e9ef --- /dev/null +++ b/dev-handbook/archiving-eol-packages.md @@ -0,0 +1,229 @@ +# Archiving EOL packages + +This document describes how to archive packages for end-of-life (EOL) distributions and prune EOL Ruby versions from the repositories. This is a routine maintenance task that frees CI disk space and keeps the repository lean. + +## Background + +The CI publish step downloads the full Aptly state archive (`state.tar.zst`) from Google Cloud Storage on every run. This archive grows with every distribution and Ruby version ever published. When distributions or Ruby versions reach EOL, their packages remain in the state archive indefinitely, consuming disk space on GitHub Actions runners. + +To address this, we maintain **archive repositories** alongside the main repositories: + +| Repository | Bucket | Domain | Purpose | +|------------|--------|--------|---------| +| APT (main) | `fsruby-server-edition-apt-repo` | `apt.fullstaqruby.org` | Current, supported packages | +| APT (archive) | `fsruby-server-edition-apt-repo-archive` | `apt-archive.fullstaqruby.org` | Frozen packages for EOL distributions | +| YUM (main) | `fsruby-server-edition-yum-repo` | `yum.fullstaqruby.org` | Current, supported packages | +| YUM (archive) | `fsruby-server-edition-yum-repo-archive` | `yum-archive.fullstaqruby.org` | Frozen packages for EOL distributions | + +Archive repositories are static — CI never writes to them. They use the same versioned bucket structure as the main repos. Each migration creates a new version that merges newly-archived distros with the existing archive contents, so the archive grows incrementally over time. + +This pattern follows the precedent set by [PostgreSQL](https://apt-archive.postgresql.org/) (`apt-archive.postgresql.org`) and [HashiCorp](https://www.hashicorp.com/en/blog/announcing-the-linux-package-archive-site) (`archive.releases.hashicorp.com`). + +## Two types of cleanup + +There are two independent axes of cleanup, each with its own script: + +### 1. Distro archival — moving entire EOL distribution repos + +When a Linux distribution reaches EOL, we stop building packages for it and move its existing packages to the archive. Users on EOL distributions can still install packages by pointing at the archive repo. + +**Scripts:** + * `internal-scripts/ci-cd/archive/migrate-apt-to-archive.rb` + * `internal-scripts/ci-cd/archive/migrate-yum-to-archive.rb` + +### 2. Package pruning — removing EOL Ruby version packages + +When a Ruby version reaches EOL, we stop building it (by removing it from `config.yml`), but its packages persist inside every distro's repository. Pruning removes these stale packages from the still-supported distro repos to reduce state size. + +**Scripts:** + * `internal-scripts/ci-cd/archive/prune-apt-packages.rb` + * `internal-scripts/ci-cd/archive/prune-yum-packages.rb` + +## Removing an EOL distribution + +### Step 1: Remove from the build system + + 1. Edit `config.yml` and remove the distribution from the `distributions` list (or add it to an exclusion). + 2. Delete the `environments//` directory. + 3. Regenerate CI/CD workflows: + + ~~~bash + ./internal-scripts/generate-ci-cd-yaml.rb + ~~~ + + 4. Commit and merge these changes. + +### Step 2: Migrate packages to the archive + +**Prerequisites:** + * `gcloud` CLI authenticated with write access to the GCS buckets + * `az` CLI authenticated with access to the `fsruby2infraowners` Key Vault (for the GPG signing key) + * `aptly`, `zstd`, and `gpg` installed locally + * Docker running (for `createrepo_c` in YUM migration) + +**Dry run first** to verify which distros will be archived: + +~~~bash +PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-apt-repo \ +ARCHIVE_REPO_BUCKET_NAME=fsruby-server-edition-apt-repo-archive \ +./internal-scripts/ci-cd/archive/migrate-apt-to-archive.rb --dry-run +~~~ + +The script auto-detects EOL distros by comparing `aptly repo list` output against the DEB distributions defined in `config.yml`. You can also specify distros explicitly (DEB-only — RPM distros belong to the YUM script): + +~~~bash +./internal-scripts/ci-cd/archive/migrate-apt-to-archive.rb --dry-run --distros debian-10,ubuntu-20.04 +~~~ + +**Execute the migration** (removes `--dry-run`): + +~~~bash +PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-apt-repo \ +ARCHIVE_REPO_BUCKET_NAME=fsruby-server-edition-apt-repo-archive \ +./internal-scripts/ci-cd/archive/migrate-apt-to-archive.rb +~~~ + +**Repeat for YUM:** + +~~~bash +PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-yum-repo \ +ARCHIVE_REPO_BUCKET_NAME=fsruby-server-edition-yum-repo-archive \ +./internal-scripts/ci-cd/archive/migrate-yum-to-archive.rb --dry-run + +PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-yum-repo \ +ARCHIVE_REPO_BUCKET_NAME=fsruby-server-edition-yum-repo-archive \ +./internal-scripts/ci-cd/archive/migrate-yum-to-archive.rb +~~~ + +### Step 3: Restart the web server + +After migration, restart the web server so Caddy picks up the new version numbers: + +~~~bash +curl -X POST https://apt.fullstaqruby.org/admin/restart_web_server \ + -H "Authorization: Bearer $ID_TOKEN" +~~~ + +Or restart the Caddy service directly via Ansible/SSH. + +### Step 4: Verify + +~~~bash +# Archive should list the archived distros +curl -s https://apt-archive.fullstaqruby.org/dists/ + +# Main repo should only contain supported distros +curl -s https://apt.fullstaqruby.org/dists/ + +# Verify state archive size decreased +gsutil ls -l gs://fsruby-server-edition-apt-repo/versions/*/state.tar.zst | tail -5 +~~~ + +## Pruning EOL Ruby versions + +After removing a Ruby version from `config.yml`, its packages persist in the Aptly state. Run the pruning scripts to remove them. + +**Dry run:** + +~~~bash +PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-apt-repo \ +./internal-scripts/ci-cd/archive/prune-apt-packages.rb --dry-run +~~~ + +The script compares packages in the Aptly state against `minor_version_packages` in `config.yml` and identifies any `fullstaq-ruby-X.Y*` packages where `X.Y` is not an active minor version. + +**Execute:** + +~~~bash +PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-apt-repo \ +./internal-scripts/ci-cd/archive/prune-apt-packages.rb +~~~ + +**Repeat for YUM:** + +~~~bash +PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-yum-repo \ +./internal-scripts/ci-cd/archive/prune-yum-packages.rb --dry-run + +PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-yum-repo \ +./internal-scripts/ci-cd/archive/prune-yum-packages.rb +~~~ + +Restart the web server after pruning (same as above). + +## Execution order + +When performing both distro archival and package pruning in the same session, always run distro archival **first**. This ensures the archive captures the full historical packages for EOL distros before any pruning happens. + + 1. `migrate-apt-to-archive.rb` + 2. `prune-apt-packages.rb` + 3. `migrate-yum-to-archive.rb` + 4. `prune-yum-packages.rb` + 5. Restart web server + +## Rollback + +The versioned bucket structure makes rollback straightforward. Each migration creates a new version — the old version is never modified. + +**Revert the main APT repo to a previous version:** + +~~~bash +# Find the pre-migration version number +gsutil cat gs://fsruby-server-edition-apt-repo/versions/latest_version.txt + +# Point back to the old version +echo -n "OLD_VERSION" | gsutil -h Content-Type:text/plain -h Cache-Control:no-store cp - gs://fsruby-server-edition-apt-repo/versions/latest_version.txt +~~~ + +**Revert the archive to a previous version:** + +~~~bash +gsutil cat gs://fsruby-server-edition-apt-repo-archive/versions/latest_version.txt + +echo -n "OLD_VERSION" | gsutil -h Content-Type:text/plain -h Cache-Control:no-store cp - gs://fsruby-server-edition-apt-repo-archive/versions/latest_version.txt +~~~ + +**Delete all archive contents** (if no users depend on it yet): + +~~~bash +gsutil -m rm -r gs://fsruby-server-edition-apt-repo-archive/versions/ +~~~ + +## How the migration scripts work + +### APT migration (`migrate-apt-to-archive.rb`) + + 1. Downloads the current Aptly state archive from the main bucket. + 2. Identifies EOL distros (published in Aptly but not in `config.yml`). + 3. Fetches the existing archive state (if any) so new distros are merged into it. + 4. Creates or extends the archive Aptly instance with EOL distro data and package pool. + 5. Publishes all archive distros (existing + newly archived). + 6. Drops the EOL distro repos from the main Aptly database. + 7. Runs `aptly db cleanup` to compact the database and reclaim pool space. + 8. Re-publishes remaining distros in the main repo. + 9. Uploads the merged archive as a new archive version (N+1). + 10. Uploads the trimmed state as a new version of the main repo. + +### YUM migration (`migrate-yum-to-archive.rb`) + + 1. Downloads the current YUM repo from the main bucket via `gsutil rsync`. + 2. Identifies EOL distro directories. + 3. Fetches the existing archive repo (if any) so new distros are merged into it. + 4. Copies EOL distro directories into the local archive copy. + 5. Uploads the merged archive as a new archive version (N+1). + 6. Removes EOL distro directories from the main local copy. + 7. Uploads the trimmed repo as a new version of the main bucket. + +### APT pruning (`prune-apt-packages.rb`) + + 1. Downloads the Aptly state. + 2. Scans all packages across all distro repos, matching `fullstaq-ruby-X.Y*` against active minor versions. + 3. Removes EOL Ruby packages using `aptly repo remove`. + 4. Compacts, re-publishes, and uploads. + +### YUM pruning (`prune-yum-packages.rb`) + + 1. Downloads the YUM repo. + 2. Deletes RPM files matching EOL Ruby versions from the filesystem. + 3. Regenerates `repodata/` with `createrepo_c` and re-signs. + 4. Uploads as a new version. diff --git a/dev-handbook/ci-cd-resumption.md b/dev-handbook/ci-cd-resumption.md index 7b7c3a0..e4078e1 100644 --- a/dev-handbook/ci-cd-resumption.md +++ b/dev-handbook/ci-cd-resumption.md @@ -8,12 +8,12 @@ In order to aleviate this problem, we implement the ability to re-run only faile Resumption support works by checking, for each CI job, whether the artifact that that job should produce, already exists. If so, then that job can be skipped. -When you re-run a CI run, Github Actions wipes all previous state (including artifacts). Therefore, we store artifacts primarily in a Google Cloud Storage bucket ([fullstaq-ruby-server-edition-ci-artifacts](https://storage.googleapis.com/fullstaq-ruby-server-edition-ci-artifacts), part of the [infrastructure](https://github.com/fullstaq-labs/fullstaq-ruby-infra)), which isn't wiped before a re-run. +When you re-run a CI run, Github Actions wipes all previous state (including artifacts). Therefore, we store artifacts primarily in a Google Cloud Storage bucket ([fsruby-server-edition-ci-artifacts](https://storage.googleapis.com/fsruby-server-edition-ci-artifacts), part of the [infrastructure](https://github.com/fullstaq-labs/fullstaq-ruby-infra)), which isn't wiped before a re-run. Here's an example artifact URL: ~~~ -gs://fullstaq-ruby-server-edition-ci-artifacts/249/rbenv-deb.tar.zst +gs://fsruby-server-edition-ci-artifacts/249/rbenv-deb.tar.zst ~~~ Artifacts are stored on a per-CI-run basis. Thus, they always contains the CI run's number. Note that the CI run number does not change even for re-runs. @@ -23,7 +23,7 @@ At the beginning of a CI run, a job named `determine_necessary_jobs` checks whic ~~~ ##### Determine whether Rbenv DEB needs to be built ##### --> Run ./.github/actions/check-artifact-exists - Checking gs://fullstaq-ruby-server-edition-ci-artifacts/249/rbenv-deb.tar.zst + Checking gs://fsruby-server-edition-ci-artifacts/249/rbenv-deb.tar.zst Artifact exists ~~~ diff --git a/internal-scripts/ci-cd/archive/migrate-apt-to-archive.rb b/internal-scripts/ci-cd/archive/migrate-apt-to-archive.rb new file mode 100755 index 0000000..815ab1a --- /dev/null +++ b/internal-scripts/ci-cd/archive/migrate-apt-to-archive.rb @@ -0,0 +1,613 @@ +#!/usr/bin/env ruby +# frozen_string_literal: true + +# Incremental migration script to move EOL distribution packages +# from the main APT repository to the archive APT repository. +# Safe to run repeatedly — merges new EOL distros into the existing archive. +# +# Usage: +# PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-apt-repo \ +# ARCHIVE_REPO_BUCKET_NAME=fsruby-server-edition-apt-repo-archive \ +# ./internal-scripts/ci-cd/archive/migrate-apt-to-archive.rb [--dry-run] [--distros centos-8,debian-9] +# +# If --distros is not specified, automatically detects EOL distros by comparing +# the Aptly state against the DEB distributions defined in config.yml. + +require_relative '../../../lib/gcloud_storage_lock' +require_relative '../../../lib/ci_workflow_support' +require_relative '../../../lib/shell_scripting_support' +require_relative '../../../lib/publishing_support' +require 'json' +require 'shellwords' +require 'tmpdir' +require 'set' +require 'fileutils' +require 'optparse' + +class MigrateAptToArchive + REPO_ORIGIN = 'Fullstaq-Ruby' + REPO_LABEL = 'Fullstaq-Ruby' + + include CiWorkflowSupport + include ShellScriptingSupport + include PublishingSupport + + def main + parse_options + require_envvar 'PRODUCTION_REPO_BUCKET_NAME' + require_envvar 'ARCHIVE_REPO_BUCKET_NAME' + + print_header 'Initializing' + load_config + create_temp_dirs + ensure_gpg_state_isolated + activate_wrappers_bin_dir + initialize_locking + initialize_aptly(@main_aptly_config_path, @main_state_path, @main_state_repo_path) + fetch_and_import_signing_key + + eol_distros = nil + version = nil + + begin + synchronize do + print_header 'Downloading main repository state' + version = get_latest_production_repo_version + if version == 0 + abort 'ERROR: No production repository exists yet' + end + fetch_main_state(version) + + print_header 'Identifying EOL distributions' + eol_distros = identify_eol_distros + if eol_distros.empty? + log_notice 'No EOL distributions found to archive' + return + end + log_notice "EOL distributions to archive: #{eol_distros.join(', ')}" + + print_header 'Fetching existing archive state (if any)' + @archive_version = get_latest_archive_version + if @archive_version > 0 + log_notice "Existing archive at version #{@archive_version}, will merge" + fetch_archive_state(@archive_version) + else + log_notice 'No existing archive — creating fresh' + end + + print_header 'Creating archive repository' + create_archive_state(eol_distros) + check_lock_health + + print_header 'Trimming main repository state' + trim_main_state(eol_distros) + check_lock_health + + if @dry_run + log_notice 'DRY RUN — not uploading changes' + print_summary(eol_distros, version) + return + end + + print_header 'Uploading archive repository' + upload_archive + check_lock_health + + print_header 'Uploading trimmed main repository' + upload_main(version) + end + + print_header 'Success!' + print_summary(eol_distros, version) if eol_distros && !eol_distros.empty? + ensure + cleanup + end + end + +private + def parse_options + @dry_run = false + @explicit_distros = nil + + OptionParser.new do |opts| + opts.banner = "Usage: #{$0} [options]" + opts.on('--dry-run', 'Do not upload changes') { @dry_run = true } + opts.on('--distros DISTROS', 'Comma-separated list of distros to archive') do |v| + @explicit_distros = v.split(',').map(&:strip) + end + end.parse! + end + + def create_temp_dirs + log_notice 'Creating temporary directories' + @temp_dir = Dir.mktmpdir('apt-archive-migration') + @wrappers_bin_dir = "#{@temp_dir}/wrappers" + + @main_aptly_config_path = "#{@temp_dir}/aptly-main.conf" + @main_state_path = "#{@temp_dir}/main-state" + @main_state_db_path = "#{@main_state_path}/db" + @main_state_repo_path = "#{@main_state_path}/repo" + @main_state_archive_path = "#{@temp_dir}/main-state.tar.zst" + + @archive_aptly_config_path = "#{@temp_dir}/aptly-archive.conf" + @archive_state_path = "#{@temp_dir}/archive-state" + @archive_state_db_path = "#{@archive_state_path}/db" + @archive_state_repo_path = "#{@archive_state_path}/repo" + @archive_state_archive_path = "#{@temp_dir}/archive-state.tar.zst" + + @signing_key_path = "#{@temp_dir}/key.gpg" + + Dir.mkdir(@wrappers_bin_dir) + [@main_state_path, @main_state_db_path, @main_state_repo_path, + @archive_state_path, @archive_state_db_path, @archive_state_repo_path].each do |dir| + FileUtils.mkdir_p(dir) + end + end + + def ensure_gpg_state_isolated + log_notice 'Creating GPG wrapper' + File.open("#{@wrappers_bin_dir}/gpg", 'w:utf-8') do |f| + f.write("#!/bin/sh\n") + f.write( + sprintf( + "exec %s --homedir %s \"$@\"\n", + Shellwords.escape(find_gpg), + Shellwords.escape(@temp_dir) + ) + ) + end + File.chmod(0755, "#{@wrappers_bin_dir}/gpg") + end + + def find_gpg + ENV['PATH'].split(':').each do |dir| + next if dir == @wrappers_bin_dir + candidate = "#{dir}/gpg" + return candidate if File.exist?(candidate) + end + abort('GPG not found') + end + + def activate_wrappers_bin_dir + ENV['PATH'] = "#{@wrappers_bin_dir}:#{ENV['PATH']}" + end + + def initialize_locking + @lock = GCloudStorageLock.new(url: lock_url) + end + + def lock_url + "gs://#{ENV['PRODUCTION_REPO_BUCKET_NAME']}/locks/apt" + end + + def synchronize(&block) + @lock.synchronize(&block) + end + + def check_lock_health + abort 'ERROR: lock is unhealthy. Aborting operation' if !@lock.healthy? + end + + def initialize_aptly(config_path, state_path, repo_path) + log_notice "Creating Aptly config: #{config_path}" + File.open(config_path, 'w:utf-8') do |f| + f.write(JSON.generate( + rootDir: state_path, + FileSystemPublishEndpoints: { + main: { + rootDir: repo_path, + linkMethod: 'symlink', + verifyMethod: 'md5' + } + } + )) + end + end + + def fetch_and_import_signing_key + log_notice 'Fetching and importing signing key' + File.open(@signing_key_path, 'wb') do |f| + f.write(fetch_signing_key) + end + @gpg_key_id = infer_gpg_key_id(@temp_dir, @signing_key_path) + log_info "Signing key ID: #{@gpg_key_id}" + import_gpg_key(@temp_dir, @signing_key_path) + end + + def fetch_main_state(version) + log_notice "Fetching main state version #{version}" + url = "gs://#{ENV['PRODUCTION_REPO_BUCKET_NAME']}/versions/#{version}/state.tar.zst" + run_bash( + sprintf('gsutil -m cp %s - | zstd -dc | tar -xC %s', + Shellwords.escape(url), + Shellwords.escape(@main_state_path)), + pipefail: true, + log_invocation: true, + check_error: true, + passthru_output: true + ) + end + + def latest_production_version_note_url + "gs://#{ENV['PRODUCTION_REPO_BUCKET_NAME']}/versions/latest_version.txt" + end + + def get_latest_archive_version + url = "gs://#{ENV['ARCHIVE_REPO_BUCKET_NAME']}/versions/latest_version.txt" + stdout_output, stderr_output, status = run_command_capture_output( + 'gsutil', 'cp', url, '-', + log_invocation: false, + check_error: false + ) + if status.success? + v = stdout_output.strip + if v =~ /\A[0-9]+\Z/ + v.to_i + else + abort("ERROR: invalid version number stored in #{url}") + end + elsif stderr_output =~ /No URLs matched/ + 0 + else + abort("ERROR: error fetching #{url}: #{stderr_output.chomp}") + end + end + + def fetch_archive_state(version) + log_notice "Fetching archive state version #{version}" + url = "gs://#{ENV['ARCHIVE_REPO_BUCKET_NAME']}/versions/#{version}/state.tar.zst" + run_bash( + sprintf('gsutil -m cp %s - | zstd -dc | tar -xC %s', + Shellwords.escape(url), + Shellwords.escape(@archive_state_path)), + pipefail: true, + log_invocation: true, + check_error: true, + passthru_output: true + ) + end + + def identify_eol_distros + if @explicit_distros + log_notice "Using explicitly specified distros: #{@explicit_distros.join(', ')}" + return @explicit_distros + end + + published = list_aptly_repos(@main_aptly_config_path) + supported = distributions + .select { |d| d[:package_format] == :DEB } + .map { |d| d[:name] } + + eol = published - supported + log_info "Published distros in Aptly state: #{published.join(', ')}" + log_info "Currently supported DEB distros: #{supported.join(', ')}" + log_info "EOL distros (published but not supported): #{eol.join(', ')}" + eol.sort + end + + def list_aptly_repos(config_path) + stdout_output, _, _ = run_command_capture_output( + 'aptly', 'repo', 'list', + "-config=#{config_path}", + '-raw', + log_invocation: true, + check_error: true + ) + stdout_output.split("\n").map(&:strip).reject(&:empty?) + end + + def list_aptly_packages(config_path, repo_name) + stdout_output, _, _ = run_command_capture_output( + 'aptly', 'repo', 'show', + "-config=#{config_path}", + '-with-packages', + repo_name, + log_invocation: false, + check_error: true + ) + stdout_output.sub!(/.*^Packages:$/m, '') + stdout_output.split("\n").map(&:strip).reject(&:empty?) + end + + def create_archive_state(eol_distros) + if @archive_version == 0 + initialize_aptly(@archive_aptly_config_path, @archive_state_path, @archive_state_repo_path) + else + # Existing archive was fetched — just initialize the Aptly config pointing at it + initialize_aptly(@archive_aptly_config_path, @archive_state_path, @archive_state_repo_path) + existing = list_aptly_repos(@archive_aptly_config_path) + log_notice "Existing archive contains distros: #{existing.join(', ')}" + end + + # Copy EOL packages from main pool into archive pool + main_pool = "#{@main_state_path}/pool" + archive_pool = "#{@archive_state_path}/pool" + if File.exist?(main_pool) + if File.exist?(archive_pool) + # Merge: copy new pool files that don't already exist (-n: no-clobber). + # Real I/O errors must surface — only no-clobber skips are benign. + run_bash( + sprintf('cp -rn %s/* %s/', + Shellwords.escape(main_pool), + Shellwords.escape(archive_pool)), + log_invocation: true, check_error: true, pipefail: false + ) + else + FileUtils.cp_r(main_pool, @archive_state_path) + end + end + + eol_distros.each do |distro| + packages = list_aptly_packages(@main_aptly_config_path, distro) + log_notice "[#{distro}] Exporting #{packages.size} packages to archive" + + # Create the archive repo for this distro + run_command( + 'aptly', 'repo', 'create', + "-config=#{@archive_aptly_config_path}", + distro, + log_invocation: true, + check_error: true + ) + + # Import packages by copying the Aptly database directory for this repo + main_repo_db = "#{@main_state_db_path}/repo/#{distro}" + if File.exist?(main_repo_db) + FileUtils.cp_r(main_repo_db, "#{@archive_state_db_path}/repo/") + end + end + + # Publish all distros in the archive (existing + newly added) + all_archive_distros = list_aptly_repos(@archive_aptly_config_path) + all_archive_distros.each do |distro| + log_notice "[#{distro}] Publishing in archive" + publish_aptly_repo(@archive_aptly_config_path, distro) + end + + # Archive the state + log_notice 'Creating archive state archive' + run_bash( + sprintf("tar -C %s -cf - . | zstd -T0 > %s", + Shellwords.escape(@archive_state_path), + Shellwords.escape(@archive_state_archive_path)), + pipefail: true, + log_invocation: true, + check_error: true, + passthru_output: true + ) + end + + def trim_main_state(eol_distros) + eol_distros.each do |distro| + log_notice "[#{distro}] Dropping from main repository" + + # Drop the publication first (if it exists) + run_command( + 'aptly', 'publish', 'drop', + "-config=#{@main_aptly_config_path}", + distro, 'filesystem:main:.', + log_invocation: true, + check_error: false + ) + + # Drop the repo + run_command( + 'aptly', 'repo', 'drop', + "-config=#{@main_aptly_config_path}", + distro, + log_invocation: true, + check_error: true + ) + end + + # Compact the database to reclaim space from dropped packages + log_notice 'Compacting main state' + run_command( + 'aptly', 'db', 'cleanup', + "-config=#{@main_aptly_config_path}", + '-verbose', + log_invocation: true, + check_error: true, + passthru_output: true + ) + + # Re-publish remaining distros + remaining = list_aptly_repos(@main_aptly_config_path) + remaining.each do |distro| + log_notice "[#{distro}] Re-publishing in main repository" + publish_aptly_repo(@main_aptly_config_path, distro) + end + + # Re-archive the trimmed state + log_notice 'Creating trimmed main state archive' + run_bash( + sprintf("tar -C %s -cf - . | zstd -T0 > %s", + Shellwords.escape(@main_state_path), + Shellwords.escape(@main_state_archive_path)), + pipefail: true, + log_invocation: true, + check_error: true, + passthru_output: true + ) + end + + def publish_aptly_repo(config_path, distro) + _, stderr_output, status = run_command_capture_output( + 'aptly', 'publish', 'repo', + '-batch', '-force-overwrite', + "-config=#{config_path}", + "-gpg-key=#{@gpg_key_id}", + "-distribution=#{distro}", + "-origin=#{REPO_ORIGIN}", + "-label=#{REPO_LABEL}", + distro, 'filesystem:main:.', + log_invocation: true, + check_error: false + ) + if !status.success? + if stderr_output =~ /unable to figure out list of architectures/ + run_command( + 'aptly', 'publish', 'repo', + '-batch', '-force-overwrite', '-architectures=all', + "-config=#{config_path}", + "-gpg-key=#{@gpg_key_id}", + "-distribution=#{distro}", + "-origin=#{REPO_ORIGIN}", + "-label=#{REPO_LABEL}", + distro, 'filesystem:main:.', + log_invocation: true, + check_error: true + ) + else + abort("ERROR publishing #{distro}: #{stderr_output.chomp}") + end + end + end + + def upload_archive + archive_bucket = ENV['ARCHIVE_REPO_BUCKET_NAME'] + new_archive_version = @archive_version + 1 + + # Upload state + log_notice "Uploading archive state as version #{new_archive_version}" + run_command( + 'gsutil', + '-h', 'Cache-Control:public', + 'cp', + @archive_state_archive_path, + "gs://#{archive_bucket}/versions/#{new_archive_version}/state.tar.zst", + log_invocation: true, + check_error: true, + passthru_output: true + ) + + # Upload published repo + log_notice 'Uploading archive repository' + run_command( + 'gsutil', '-m', + '-h', 'Cache-Control:public', + 'rsync', '-r', '-d', + @archive_state_repo_path, + "gs://#{archive_bucket}/versions/#{new_archive_version}/public", + log_invocation: true, + check_error: true, + passthru_output: true + ) + + # Create version note + run_bash( + sprintf( + 'gsutil -q -h Content-Type:text/plain -h Cache-Control:no-store cp - %s <<<%s', + Shellwords.escape("gs://#{archive_bucket}/versions/#{new_archive_version}/version.txt"), + Shellwords.escape(new_archive_version.to_s) + ), + log_invocation: true, + check_error: true, + pipefail: false + ) + + # Declare latest version + run_bash( + sprintf( + 'gsutil -q -h Content-Type:text/plain -h Cache-Control:no-store cp - %s <<<%s', + Shellwords.escape("gs://#{archive_bucket}/versions/latest_version.txt"), + Shellwords.escape(new_archive_version.to_s) + ), + log_invocation: true, + check_error: true, + pipefail: false + ) + end + + def upload_main(old_version) + new_version = old_version + 1 + bucket = ENV['PRODUCTION_REPO_BUCKET_NAME'] + + # Upload state + log_notice "Uploading trimmed main state as version #{new_version}" + run_command( + 'gsutil', + '-h', 'Cache-Control:public', + 'cp', + @main_state_archive_path, + "gs://#{bucket}/versions/#{new_version}/state.tar.zst", + log_invocation: true, + check_error: true, + passthru_output: true + ) + + # Copy previous repo version, then overwrite with trimmed version + log_notice "Copying repo version #{old_version} to #{new_version}" + run_command( + 'gsutil', '-m', + '-h', 'Cache-Control:public', + 'rsync', '-r', '-d', + "gs://#{bucket}/versions/#{old_version}/public", + "gs://#{bucket}/versions/#{new_version}/public", + log_invocation: true, + check_error: true, + passthru_output: true + ) + + log_notice "Uploading trimmed repo as version #{new_version}" + run_command( + 'gsutil', '-m', + '-h', 'Cache-Control:public', + 'rsync', '-r', '-d', + @main_state_repo_path, + "gs://#{bucket}/versions/#{new_version}/public", + log_invocation: true, + check_error: true, + passthru_output: true + ) + + # Version note + run_bash( + sprintf( + 'gsutil -q -h Content-Type:text/plain -h Cache-Control:public cp - %s <<<%s', + Shellwords.escape("gs://#{bucket}/versions/#{new_version}/version.txt"), + Shellwords.escape(new_version.to_s) + ), + log_invocation: true, + check_error: true, + pipefail: false + ) + + # Activate new version + log_notice "Activating main repo version #{new_version}" + run_bash( + sprintf( + 'gsutil -q -h Content-Type:text/plain -h Cache-Control:no-store cp - %s <<<%s', + Shellwords.escape("gs://#{bucket}/versions/latest_version.txt"), + Shellwords.escape(new_version.to_s) + ), + log_invocation: true, + check_error: true, + pipefail: false + ) + end + + def print_summary(eol_distros, old_version) + new_archive_version = @archive_version + 1 + archive_bucket = ENV['ARCHIVE_REPO_BUCKET_NAME'] + log_notice 'Migration summary' + log_info "Archived distributions: #{eol_distros.join(', ')}" + log_info "Main repo: version #{old_version} -> #{old_version + 1}" + log_info "Archive repo: version #{@archive_version} -> #{new_archive_version}" + log_info '' + log_info 'Archive APT repo URL:' + log_info " https://storage.googleapis.com/#{archive_bucket}/versions/#{new_archive_version}/public" + log_info '' + log_info 'Next steps:' + log_info ' 1. Restart the web server to pick up new archive version' + log_info ' 2. Verify archive repo: curl https://apt-archive.fullstaqruby.org/dists/' + log_info ' 3. Verify main repo still works: apt-get update on a supported distro' + end + + def cleanup + log_info "Cleaning up #{@temp_dir}" + FileUtils.remove_entry_secure(@temp_dir) + end +end + +MigrateAptToArchive.new.main diff --git a/internal-scripts/ci-cd/archive/migrate-yum-to-archive.rb b/internal-scripts/ci-cd/archive/migrate-yum-to-archive.rb new file mode 100755 index 0000000..e5b85ca --- /dev/null +++ b/internal-scripts/ci-cd/archive/migrate-yum-to-archive.rb @@ -0,0 +1,350 @@ +#!/usr/bin/env ruby +# frozen_string_literal: true + +# Incremental migration script to move EOL distribution packages +# from the main YUM repository to the archive YUM repository. +# Safe to run repeatedly — merges new EOL distros into the existing archive. +# +# Usage: +# PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-yum-repo \ +# ARCHIVE_REPO_BUCKET_NAME=fsruby-server-edition-yum-repo-archive \ +# ./internal-scripts/ci-cd/archive/migrate-yum-to-archive.rb [--dry-run] [--distros centos-8] +# +# If --distros is not specified, automatically detects EOL distros by comparing +# the repo contents against the RPM distributions defined in config.yml. + +require_relative '../../../lib/gcloud_storage_lock' +require_relative '../../../lib/ci_workflow_support' +require_relative '../../../lib/shell_scripting_support' +require_relative '../../../lib/publishing_support' +require 'shellwords' +require 'tmpdir' +require 'fileutils' +require 'optparse' + +class MigrateYumToArchive + include CiWorkflowSupport + include ShellScriptingSupport + include PublishingSupport + + def main + parse_options + require_envvar 'PRODUCTION_REPO_BUCKET_NAME' + require_envvar 'ARCHIVE_REPO_BUCKET_NAME' + + print_header 'Initializing' + load_config + create_temp_dirs + initialize_locking + fetch_and_import_signing_key + + eol_distros = nil + version = nil + + begin + synchronize do + print_header 'Downloading main repository' + version = get_latest_production_repo_version + if version == 0 + abort 'ERROR: No production repository exists yet' + end + fetch_main_repo(version) + + print_header 'Identifying EOL distributions' + eol_distros = identify_eol_distros + if eol_distros.empty? + log_notice 'No EOL distributions found to archive' + return + end + log_notice "EOL distributions to archive: #{eol_distros.join(', ')}" + + print_header 'Fetching existing archive (if any)' + @archive_version = get_latest_archive_version + if @archive_version > 0 + log_notice "Existing archive at version #{@archive_version}, will merge" + fetch_archive_repo(@archive_version) + else + log_notice 'No existing archive — creating fresh' + @archive_repo_path = "#{@temp_dir}/archive-repo" + Dir.mkdir(@archive_repo_path) + end + + if @dry_run + log_notice 'DRY RUN — not uploading changes' + print_summary(eol_distros, version) + return + end + + print_header 'Uploading EOL distros to archive' + upload_archive(eol_distros) + check_lock_health + + print_header 'Removing EOL distros from main repository' + remove_from_main(eol_distros, version) + end + + print_header 'Success!' + print_summary(eol_distros, version) if eol_distros && !eol_distros.empty? + ensure + cleanup + end + end + +private + def parse_options + @dry_run = false + @explicit_distros = nil + + OptionParser.new do |opts| + opts.banner = "Usage: #{$0} [options]" + opts.on('--dry-run', 'Do not upload changes') { @dry_run = true } + opts.on('--distros DISTROS', 'Comma-separated list of distros to archive') do |v| + @explicit_distros = v.split(',').map(&:strip) + end + end.parse! + end + + def create_temp_dirs + @temp_dir = Dir.mktmpdir('yum-archive-migration') + @local_repo_path = "#{@temp_dir}/repo" + @signing_key_path = "#{@temp_dir}/key.gpg" + Dir.mkdir(@local_repo_path) + end + + def initialize_locking + @lock = GCloudStorageLock.new(url: lock_url) + end + + def lock_url + "gs://#{ENV['PRODUCTION_REPO_BUCKET_NAME']}/locks/yum" + end + + def synchronize(&block) + @lock.synchronize(&block) + end + + def check_lock_health + abort 'ERROR: lock is unhealthy. Aborting operation' if !@lock.healthy? + end + + def fetch_and_import_signing_key + log_notice 'Fetching and importing signing key' + File.open(@signing_key_path, 'wb') do |f| + f.write(fetch_signing_key) + end + @gpg_key_id = infer_gpg_key_id(@temp_dir, @signing_key_path) + log_info "Signing key ID: #{@gpg_key_id}" + import_gpg_key(@temp_dir, @signing_key_path) + end + + def latest_production_version_note_url + "gs://#{ENV['PRODUCTION_REPO_BUCKET_NAME']}/versions/latest_version.txt" + end + + def get_latest_archive_version + url = "gs://#{ENV['ARCHIVE_REPO_BUCKET_NAME']}/versions/latest_version.txt" + stdout_output, stderr_output, status = run_command_capture_output( + 'gsutil', 'cp', url, '-', + log_invocation: false, + check_error: false + ) + if status.success? + v = stdout_output.strip + if v =~ /\A[0-9]+\Z/ + v.to_i + else + abort("ERROR: invalid version number stored in #{url}") + end + elsif stderr_output =~ /No URLs matched/ + 0 + else + abort("ERROR: error fetching #{url}: #{stderr_output.chomp}") + end + end + + def fetch_archive_repo(version) + @archive_repo_path = "#{@temp_dir}/archive-repo" + Dir.mkdir(@archive_repo_path) + log_notice "Fetching archive repo version #{version}" + run_command( + 'gsutil', '-m', 'rsync', '-r', + "gs://#{ENV['ARCHIVE_REPO_BUCKET_NAME']}/versions/#{version}/public", + @archive_repo_path, + log_invocation: true, + check_error: true, + passthru_output: true + ) + end + + def fetch_main_repo(version) + log_notice "Fetching main repo version #{version}" + run_command( + 'gsutil', '-m', 'rsync', '-r', + "gs://#{ENV['PRODUCTION_REPO_BUCKET_NAME']}/versions/#{version}/public", + @local_repo_path, + log_invocation: true, + check_error: true, + passthru_output: true + ) + end + + def identify_eol_distros + if @explicit_distros + log_notice "Using explicitly specified distros: #{@explicit_distros.join(', ')}" + return @explicit_distros + end + + # List distro directories in the downloaded repo + published = Dir.entries(@local_repo_path) + .select { |e| File.directory?("#{@local_repo_path}/#{e}") } + .reject { |e| e.start_with?('.') } + + supported = distributions + .select { |d| d[:package_format] == :RPM } + .map { |d| d[:name] } + + eol = published - supported + log_info "Published distros in YUM repo: #{published.join(', ')}" + log_info "Currently supported RPM distros: #{supported.join(', ')}" + log_info "EOL distros (published but not supported): #{eol.join(', ')}" + eol.sort + end + + def upload_archive(eol_distros) + archive_bucket = ENV['ARCHIVE_REPO_BUCKET_NAME'] + new_archive_version = @archive_version + 1 + + # Copy EOL distro directories into the local archive repo + eol_distros.each do |distro| + src = "#{@local_repo_path}/#{distro}" + dst = "#{@archive_repo_path}/#{distro}" + log_notice "[#{distro}] Copying to archive staging area" + FileUtils.cp_r(src, dst) + end + + # Upload merged archive repo + log_notice "Uploading archive repo as version #{new_archive_version}" + run_command( + 'gsutil', '-m', + '-h', 'Cache-Control:public', + 'rsync', '-r', '-d', + @archive_repo_path, + "gs://#{archive_bucket}/versions/#{new_archive_version}/public", + log_invocation: true, + check_error: true, + passthru_output: true + ) + + # Create version note + run_bash( + sprintf( + 'gsutil -q -h Content-Type:text/plain -h Cache-Control:no-store cp - %s <<<%s', + Shellwords.escape("gs://#{archive_bucket}/versions/#{new_archive_version}/version.txt"), + Shellwords.escape(new_archive_version.to_s) + ), + log_invocation: true, + check_error: true, + pipefail: false + ) + + # Declare latest version + run_bash( + sprintf( + 'gsutil -q -h Content-Type:text/plain -h Cache-Control:no-store cp - %s <<<%s', + Shellwords.escape("gs://#{archive_bucket}/versions/latest_version.txt"), + Shellwords.escape(new_archive_version.to_s) + ), + log_invocation: true, + check_error: true, + pipefail: false + ) + end + + def remove_from_main(eol_distros, old_version) + new_version = old_version + 1 + bucket = ENV['PRODUCTION_REPO_BUCKET_NAME'] + + # Remove EOL distro directories from local copy + eol_distros.each do |distro| + distro_path = "#{@local_repo_path}/#{distro}" + if File.exist?(distro_path) + log_notice "[#{distro}] Removing from local repo" + FileUtils.rm_rf(distro_path) + end + end + + # Upload trimmed repo as new version + log_notice "Copying repo version #{old_version} to #{new_version}" + run_command( + 'gsutil', '-m', + '-h', 'Cache-Control:public', + 'rsync', '-r', '-d', + "gs://#{bucket}/versions/#{old_version}/public", + "gs://#{bucket}/versions/#{new_version}/public", + log_invocation: true, + check_error: true, + passthru_output: true + ) + + log_notice "Uploading trimmed repo as version #{new_version}" + run_command( + 'gsutil', '-m', + '-h', 'Cache-Control:public', + 'rsync', '-r', '-d', + @local_repo_path, + "gs://#{bucket}/versions/#{new_version}/public", + log_invocation: true, + check_error: true, + passthru_output: true + ) + + # Version note + run_bash( + sprintf( + 'gsutil -q -h Content-Type:text/plain -h Cache-Control:public cp - %s <<<%s', + Shellwords.escape("gs://#{bucket}/versions/#{new_version}/version.txt"), + Shellwords.escape(new_version.to_s) + ), + log_invocation: true, + check_error: true, + pipefail: false + ) + + # Activate + log_notice "Activating main repo version #{new_version}" + run_bash( + sprintf( + 'gsutil -q -h Content-Type:text/plain -h Cache-Control:no-store cp - %s <<<%s', + Shellwords.escape("gs://#{bucket}/versions/latest_version.txt"), + Shellwords.escape(new_version.to_s) + ), + log_invocation: true, + check_error: true, + pipefail: false + ) + end + + def print_summary(eol_distros, old_version) + new_archive_version = @archive_version + 1 + archive_bucket = ENV['ARCHIVE_REPO_BUCKET_NAME'] + log_notice 'Migration summary' + log_info "Archived distributions: #{eol_distros.join(', ')}" + log_info "Main repo: version #{old_version} -> #{old_version + 1}" + log_info "Archive repo: version #{@archive_version} -> #{new_archive_version}" + log_info '' + log_info 'Archive YUM repo URL:' + log_info " https://storage.googleapis.com/#{archive_bucket}/versions/#{new_archive_version}/public" + log_info '' + log_info 'Next steps:' + log_info ' 1. Restart the web server to pick up new archive version' + log_info ' 2. Verify archive repo: curl https://yum-archive.fullstaqruby.org/' + log_info ' 3. Verify main repo still works on a supported distro' + end + + def cleanup + log_info "Cleaning up #{@temp_dir}" + FileUtils.remove_entry_secure(@temp_dir) + end +end + +MigrateYumToArchive.new.main diff --git a/internal-scripts/ci-cd/archive/prune-apt-packages.rb b/internal-scripts/ci-cd/archive/prune-apt-packages.rb new file mode 100755 index 0000000..db709eb --- /dev/null +++ b/internal-scripts/ci-cd/archive/prune-apt-packages.rb @@ -0,0 +1,390 @@ +#!/usr/bin/env ruby +# frozen_string_literal: true + +# Prune EOL Ruby version packages from the APT repository state. +# +# Removes packages for Ruby versions no longer in config.yml from all +# distro repos in the Aptly state, then compacts and re-uploads. +# +# Usage: +# PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-apt-repo \ +# ./internal-scripts/ci-cd/archive/prune-apt-packages.rb [--dry-run] +# +# Automatically detects which Ruby versions are EOL by comparing +# packages in the Aptly state against minor_version_packages in config.yml. + +require_relative '../../../lib/gcloud_storage_lock' +require_relative '../../../lib/ci_workflow_support' +require_relative '../../../lib/shell_scripting_support' +require_relative '../../../lib/publishing_support' +require 'json' +require 'shellwords' +require 'tmpdir' +require 'set' +require 'fileutils' +require 'optparse' + +class PruneAptPackages + REPO_ORIGIN = 'Fullstaq-Ruby' + REPO_LABEL = 'Fullstaq-Ruby' + + # Matches: fullstaq-ruby-3.1, fullstaq-ruby-3.1-jemalloc, fullstaq-ruby-3.1.7, etc. + # Does NOT match: fullstaq-ruby-common, fullstaq-rbenv + RUBY_PACKAGE_PATTERN = /\Afullstaq-ruby-(\d+\.\d+)/ + + include CiWorkflowSupport + include ShellScriptingSupport + include PublishingSupport + + def main + parse_options + require_envvar 'PRODUCTION_REPO_BUCKET_NAME' + + print_header 'Initializing' + load_config + create_temp_dirs + ensure_gpg_state_isolated + activate_wrappers_bin_dir + initialize_locking + initialize_aptly + fetch_and_import_signing_key + + total_pruned = 0 + version = nil + repos = nil + + begin + synchronize do + print_header 'Downloading repository state' + version = get_latest_production_repo_version + if version == 0 + abort 'ERROR: No production repository exists yet' + end + fetch_state(version) + + print_header 'Identifying EOL Ruby versions' + active_minors = active_ruby_minor_versions + log_info "Active Ruby minor versions: #{active_minors.join(', ')}" + + print_header 'Scanning packages' + repos = list_aptly_repos + repos.each do |distro| + pruned = prune_eol_packages_from_repo(distro, active_minors) + total_pruned += pruned + end + + if total_pruned == 0 + log_notice 'No EOL Ruby packages found to prune' + return + end + + log_notice "Total packages to prune: #{total_pruned}" + + if @dry_run + log_notice 'DRY RUN — not uploading changes' + return + end + + print_header 'Compacting state' + compact_state + check_lock_health + + print_header 'Re-publishing repository' + repos.each do |distro| + publish_repo(distro) + end + check_lock_health + + print_header 'Archiving and uploading state' + archive_state + upload_state(version + 1) + upload_repo(version + 1, version) + create_version_note(version + 1) + declare_latest_version(version + 1) + end + + print_header 'Success!' + if total_pruned > 0 && !@dry_run + log_info "Pruned #{total_pruned} packages across #{repos.size} repos" + log_info "Main repo: version #{version} -> #{version + 1}" + end + ensure + cleanup + end + end + +private + def parse_options + @dry_run = false + OptionParser.new do |opts| + opts.banner = "Usage: #{$0} [options]" + opts.on('--dry-run', 'Do not upload changes') { @dry_run = true } + end.parse! + end + + def create_temp_dirs + @temp_dir = Dir.mktmpdir('apt-prune') + @wrappers_bin_dir = "#{@temp_dir}/wrappers" + @aptly_config_path = "#{@temp_dir}/aptly.conf" + @signing_key_path = "#{@temp_dir}/key.gpg" + @local_state_path = "#{@temp_dir}/state" + @local_state_db_path = "#{@local_state_path}/db" + @local_state_repo_path = "#{@local_state_path}/repo" + @local_state_archive_path = "#{@temp_dir}/state.tar.zst" + + Dir.mkdir(@wrappers_bin_dir) + FileUtils.mkdir_p(@local_state_db_path) + FileUtils.mkdir_p(@local_state_repo_path) + end + + def ensure_gpg_state_isolated + File.open("#{@wrappers_bin_dir}/gpg", 'w:utf-8') do |f| + f.write("#!/bin/sh\n") + f.write( + sprintf("exec %s --homedir %s \"$@\"\n", + Shellwords.escape(find_gpg), + Shellwords.escape(@temp_dir)) + ) + end + File.chmod(0755, "#{@wrappers_bin_dir}/gpg") + end + + def find_gpg + ENV['PATH'].split(':').each do |dir| + next if dir == @wrappers_bin_dir + candidate = "#{dir}/gpg" + return candidate if File.exist?(candidate) + end + abort('GPG not found') + end + + def activate_wrappers_bin_dir + ENV['PATH'] = "#{@wrappers_bin_dir}:#{ENV['PATH']}" + end + + def initialize_locking + @lock = GCloudStorageLock.new(url: lock_url) + end + + def lock_url + "gs://#{ENV['PRODUCTION_REPO_BUCKET_NAME']}/locks/apt" + end + + def synchronize(&block) + @lock.synchronize(&block) + end + + def check_lock_health + abort 'ERROR: lock is unhealthy. Aborting operation' if !@lock.healthy? + end + + def initialize_aptly + File.open(@aptly_config_path, 'w:utf-8') do |f| + f.write(JSON.generate( + rootDir: @local_state_path, + FileSystemPublishEndpoints: { + main: { + rootDir: @local_state_repo_path, + linkMethod: 'symlink', + verifyMethod: 'md5' + } + } + )) + end + end + + def fetch_and_import_signing_key + log_notice 'Fetching and importing signing key' + File.open(@signing_key_path, 'wb') do |f| + f.write(fetch_signing_key) + end + @gpg_key_id = infer_gpg_key_id(@temp_dir, @signing_key_path) + log_info "Signing key ID: #{@gpg_key_id}" + import_gpg_key(@temp_dir, @signing_key_path) + end + + def latest_production_version_note_url + "gs://#{ENV['PRODUCTION_REPO_BUCKET_NAME']}/versions/latest_version.txt" + end + + def fetch_state(version) + log_notice "Fetching state version #{version}" + run_bash( + sprintf('gsutil -m cp %s - | zstd -dc | tar -xC %s', + Shellwords.escape("gs://#{ENV['PRODUCTION_REPO_BUCKET_NAME']}/versions/#{version}/state.tar.zst"), + Shellwords.escape(@local_state_path)), + pipefail: true, + log_invocation: true, + check_error: true, + passthru_output: true + ) + end + + def active_ruby_minor_versions + config['ruby']['minor_version_packages'].map { |p| p['minor_version'] } + end + + def list_aptly_repos + stdout_output, _, _ = run_command_capture_output( + 'aptly', 'repo', 'list', + "-config=#{@aptly_config_path}", '-raw', + log_invocation: true, check_error: true + ) + stdout_output.split("\n").map(&:strip).reject(&:empty?) + end + + def list_packages(distro) + stdout_output, _, _ = run_command_capture_output( + 'aptly', 'repo', 'show', + "-config=#{@aptly_config_path}", + '-with-packages', distro, + log_invocation: false, check_error: true + ) + stdout_output.sub!(/.*^Packages:$/m, '') + stdout_output.split("\n").map(&:strip).reject(&:empty?) + end + + def prune_eol_packages_from_repo(distro, active_minors) + packages = list_packages(distro) + eol_packages = packages.select do |pkg| + if pkg =~ RUBY_PACKAGE_PATTERN + minor = $1 + !active_minors.include?(minor) + else + false + end + end + + if eol_packages.empty? + log_info "[#{distro}] No EOL packages found" + return 0 + end + + log_notice "[#{distro}] Pruning #{eol_packages.size} EOL packages (of #{packages.size} total)" + eol_packages.each { |pkg| log_info " #{YELLOW}PRUNE#{RESET} #{pkg}" } + + if !@dry_run + eol_packages.each_slice(50) do |batch| + query = batch.join(' | ') + run_command( + 'aptly', 'repo', 'remove', + "-config=#{@aptly_config_path}", + distro, query, + log_invocation: false, + check_error: true + ) + end + end + + eol_packages.size + end + + def compact_state + run_command( + 'aptly', 'db', 'cleanup', + "-config=#{@aptly_config_path}", '-verbose', + log_invocation: true, check_error: true, passthru_output: true + ) + end + + def publish_repo(distro) + log_notice "[#{distro}] Publishing" + _, stderr_output, status = run_command_capture_output( + 'aptly', 'publish', 'repo', + '-batch', '-force-overwrite', + "-config=#{@aptly_config_path}", + "-gpg-key=#{@gpg_key_id}", + "-distribution=#{distro}", + "-origin=#{REPO_ORIGIN}", + "-label=#{REPO_LABEL}", + distro, 'filesystem:main:.', + log_invocation: true, check_error: false + ) + if !status.success? + if stderr_output =~ /unable to figure out list of architectures/ + run_command( + 'aptly', 'publish', 'repo', + '-batch', '-force-overwrite', '-architectures=all', + "-config=#{@aptly_config_path}", + "-gpg-key=#{@gpg_key_id}", + "-distribution=#{distro}", + "-origin=#{REPO_ORIGIN}", + "-label=#{REPO_LABEL}", + distro, 'filesystem:main:.', + log_invocation: true, check_error: true + ) + else + abort("ERROR publishing #{distro}: #{stderr_output.chomp}") + end + end + end + + def archive_state + log_notice 'Creating state archive' + run_bash( + sprintf("tar -C %s -cf - . | zstd -T0 > %s", + Shellwords.escape(@local_state_path), + Shellwords.escape(@local_state_archive_path)), + pipefail: true, + log_invocation: true, check_error: true, passthru_output: true + ) + end + + def upload_state(version) + bucket = ENV['PRODUCTION_REPO_BUCKET_NAME'] + run_command( + 'gsutil', '-h', 'Cache-Control:public', 'cp', + @local_state_archive_path, + "gs://#{bucket}/versions/#{version}/state.tar.zst", + log_invocation: true, check_error: true, passthru_output: true + ) + end + + def upload_repo(version, old_version) + bucket = ENV['PRODUCTION_REPO_BUCKET_NAME'] + + log_notice "Copying repo version #{old_version} to #{version}" + run_command( + 'gsutil', '-m', '-h', 'Cache-Control:public', + 'rsync', '-r', '-d', + "gs://#{bucket}/versions/#{old_version}/public", + "gs://#{bucket}/versions/#{version}/public", + log_invocation: true, check_error: true, passthru_output: true + ) + + log_notice "Uploading pruned repo as version #{version}" + run_command( + 'gsutil', '-m', '-h', 'Cache-Control:public', + 'rsync', '-r', '-d', + @local_state_repo_path, + "gs://#{bucket}/versions/#{version}/public", + log_invocation: true, check_error: true, passthru_output: true + ) + end + + def create_version_note(version) + run_bash( + sprintf('gsutil -q -h Content-Type:text/plain -h Cache-Control:public cp - %s <<<%s', + Shellwords.escape("gs://#{ENV['PRODUCTION_REPO_BUCKET_NAME']}/versions/#{version}/version.txt"), + Shellwords.escape(version.to_s)), + log_invocation: true, check_error: true, pipefail: false + ) + end + + def declare_latest_version(version) + log_notice "Activating version #{version}" + run_bash( + sprintf('gsutil -q -h Content-Type:text/plain -h Cache-Control:no-store cp - %s <<<%s', + Shellwords.escape("gs://#{ENV['PRODUCTION_REPO_BUCKET_NAME']}/versions/latest_version.txt"), + Shellwords.escape(version.to_s)), + log_invocation: true, check_error: true, pipefail: false + ) + end + + def cleanup + log_info "Cleaning up #{@temp_dir}" + FileUtils.remove_entry_secure(@temp_dir) + end +end + +PruneAptPackages.new.main diff --git a/internal-scripts/ci-cd/archive/prune-yum-packages.rb b/internal-scripts/ci-cd/archive/prune-yum-packages.rb new file mode 100755 index 0000000..ab45fe2 --- /dev/null +++ b/internal-scripts/ci-cd/archive/prune-yum-packages.rb @@ -0,0 +1,246 @@ +#!/usr/bin/env ruby +# frozen_string_literal: true + +# Prune EOL Ruby version packages from the YUM repository. +# +# Removes RPM files for Ruby versions no longer in config.yml from all +# distro/arch directories, regenerates repodata, and re-uploads. +# +# Usage: +# PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-yum-repo \ +# ./internal-scripts/ci-cd/archive/prune-yum-packages.rb [--dry-run] + +require_relative '../../../lib/gcloud_storage_lock' +require_relative '../../../lib/ci_workflow_support' +require_relative '../../../lib/shell_scripting_support' +require_relative '../../../lib/publishing_support' +require 'shellwords' +require 'tmpdir' +require 'fileutils' +require 'optparse' + +class PruneYumPackages + # Matches: fullstaq-ruby-3.1.7-jemalloc-rev1-centos8.x86_64.rpm + RUBY_RPM_PATTERN = /\Afullstaq-ruby-(\d+\.\d+)/ + + include CiWorkflowSupport + include ShellScriptingSupport + include PublishingSupport + + def main + parse_options + require_envvar 'PRODUCTION_REPO_BUCKET_NAME' + + print_header 'Initializing' + load_config + create_temp_dirs + initialize_locking + pull_utility_image_if_not_exists + fetch_and_import_signing_key + + total_pruned = 0 + version = nil + + begin + synchronize do + print_header 'Downloading repository' + version = get_latest_production_repo_version + if version == 0 + abort 'ERROR: No production repository exists yet' + end + fetch_repo(version) + + print_header 'Identifying EOL Ruby versions' + active_minors = active_ruby_minor_versions + log_info "Active Ruby minor versions: #{active_minors.join(', ')}" + + print_header 'Scanning and pruning packages' + affected_dirs = [] + + Dir.glob("#{@local_repo_path}/*/*").each do |arch_dir| + next unless File.directory?(arch_dir) + distro = File.basename(File.dirname(arch_dir)) + arch = File.basename(arch_dir) + + rpms = Dir.glob("#{arch_dir}/fullstaq-ruby-*.rpm") + eol_rpms = rpms.select do |rpm| + basename = File.basename(rpm) + if basename =~ RUBY_RPM_PATTERN + !active_minors.include?($1) + else + false + end + end + + next if eol_rpms.empty? + + log_notice "[#{distro}/#{arch}] Pruning #{eol_rpms.size} EOL packages (of #{rpms.size} total)" + eol_rpms.each { |rpm| log_info " #{YELLOW}PRUNE#{RESET} #{File.basename(rpm)}" } + total_pruned += eol_rpms.size + + if !@dry_run + eol_rpms.each { |rpm| File.delete(rpm) } + affected_dirs << arch_dir + end + end + + if total_pruned == 0 + log_notice 'No EOL Ruby packages found to prune' + return + end + + log_notice "Total packages pruned: #{total_pruned}" + + if @dry_run + log_notice 'DRY RUN — not uploading changes' + return + end + + print_header 'Regenerating repo metadata' + affected_dirs.each do |dir| + invoke_createrepo(dir) + sign_repo(dir) + end + check_lock_health + + print_header 'Uploading pruned repository' + upload_repo(version + 1, version) + create_version_note(version + 1) + declare_latest_version(version + 1) + end + + print_header 'Success!' + if total_pruned > 0 && !@dry_run + log_info "Pruned #{total_pruned} packages" + log_info "Main repo: version #{version} -> #{version + 1}" + end + ensure + cleanup + end + end + +private + def parse_options + @dry_run = false + OptionParser.new do |opts| + opts.banner = "Usage: #{$0} [options]" + opts.on('--dry-run', 'Do not upload changes') { @dry_run = true } + end.parse! + end + + def create_temp_dirs + @temp_dir = Dir.mktmpdir('yum-prune') + @local_repo_path = "#{@temp_dir}/repo" + @signing_key_path = "#{@temp_dir}/key.gpg" + Dir.mkdir(@local_repo_path) + end + + def initialize_locking + @lock = GCloudStorageLock.new(url: lock_url) + end + + def lock_url + "gs://#{ENV['PRODUCTION_REPO_BUCKET_NAME']}/locks/yum" + end + + def synchronize(&block) + @lock.synchronize(&block) + end + + def check_lock_health + abort 'ERROR: lock is unhealthy. Aborting operation' if !@lock.healthy? + end + + def fetch_and_import_signing_key + log_notice 'Fetching and importing signing key' + File.open(@signing_key_path, 'wb') do |f| + f.write(fetch_signing_key) + end + @gpg_key_id = infer_gpg_key_id(@temp_dir, @signing_key_path) + import_gpg_key(@temp_dir, @signing_key_path) + end + + def latest_production_version_note_url + "gs://#{ENV['PRODUCTION_REPO_BUCKET_NAME']}/versions/latest_version.txt" + end + + def fetch_repo(version) + run_command( + 'gsutil', '-m', 'rsync', '-r', + "gs://#{ENV['PRODUCTION_REPO_BUCKET_NAME']}/versions/#{version}/public", + @local_repo_path, + log_invocation: true, check_error: true, passthru_output: true + ) + end + + def active_ruby_minor_versions + config['ruby']['minor_version_packages'].map { |p| p['minor_version'] } + end + + def invoke_createrepo(dir) + run_command( + 'docker', 'run', '--rm', + '-v', "#{dir}:/input:delegated", + '--user', "#{Process.uid}:#{Process.gid}", + utility_image_name, + 'createrepo_c', '--update', '/input', + log_invocation: true, check_error: true + ) + end + + def sign_repo(path) + run_command( + 'gpg', "--homedir=#{@temp_dir}", "--local-user=#{@gpg_key_id}", + '--batch', '--yes', '--detach-sign', '--armor', + "#{path}/repodata/repomd.xml", + log_invocation: true, check_error: true + ) + end + + def upload_repo(version, old_version) + bucket = ENV['PRODUCTION_REPO_BUCKET_NAME'] + + log_notice "Copying repo version #{old_version} to #{version}" + run_command( + 'gsutil', '-m', '-h', 'Cache-Control:public', + 'rsync', '-r', '-d', + "gs://#{bucket}/versions/#{old_version}/public", + "gs://#{bucket}/versions/#{version}/public", + log_invocation: true, check_error: true, passthru_output: true + ) + + log_notice "Uploading pruned repo as version #{version}" + run_command( + 'gsutil', '-m', '-h', 'Cache-Control:public', + 'rsync', '-r', '-d', + @local_repo_path, + "gs://#{bucket}/versions/#{version}/public", + log_invocation: true, check_error: true, passthru_output: true + ) + end + + def create_version_note(version) + run_bash( + sprintf('gsutil -q -h Content-Type:text/plain -h Cache-Control:public cp - %s <<<%s', + Shellwords.escape("gs://#{ENV['PRODUCTION_REPO_BUCKET_NAME']}/versions/#{version}/version.txt"), + Shellwords.escape(version.to_s)), + log_invocation: true, check_error: true, pipefail: false + ) + end + + def declare_latest_version(version) + run_bash( + sprintf('gsutil -q -h Content-Type:text/plain -h Cache-Control:no-store cp - %s <<<%s', + Shellwords.escape("gs://#{ENV['PRODUCTION_REPO_BUCKET_NAME']}/versions/latest_version.txt"), + Shellwords.escape(version.to_s)), + log_invocation: true, check_error: true, pipefail: false + ) + end + + def cleanup + log_info "Cleaning up #{@temp_dir}" + FileUtils.remove_entry_secure(@temp_dir) + end +end + +PruneYumPackages.new.main