Skip to content

Add S3-compatible caching backend for self-hosted runners#810

Draft
Copilot wants to merge 6 commits intomainfrom
copilot/consider-s3-caching-backend
Draft

Add S3-compatible caching backend for self-hosted runners#810
Copilot wants to merge 6 commits intomainfrom
copilot/consider-s3-caching-backend

Conversation

Copy link
Contributor

Copilot AI commented Mar 16, 2026

Self-hosted runner users have no good way to cache builds locally — the default GitHub Actions cache backend is remote and may not meet speed or governance requirements. This adds an optional S3-compatible backend (AWS S3, MinIO, etc.) that activates when s3-bucket is set.

New inputs (action.yml)

Input Purpose
s3-bucket Bucket name; activates S3 backend when set
s3-endpoint Custom endpoint URL (e.g. MinIO)
s3-region AWS region (default: us-east-1)
s3-access-key-id / s3-secret-access-key Explicit credentials; falls back to SDK default chain if omitted
s3-path-style Force path-style addressing (required for MinIO)

Implementation (src/s3-cache.ts)

  • Restore: HeadObject for exact key match → ListObjectsV2 prefix scan (most-recently-modified wins) → download + extract via tar -xzf ... -C /
  • Save: tar the cache paths, upload via @aws-sdk/lib-storage multipart Upload
  • Paths are normalized before archiving to reject any ../ traversal
  • Archives embed absolute paths relative to / so extraction is location-stable

Wiring (restore.ts / save.ts)

restore.ts and save.ts use a dynamic import() for s3-cache.ts — the AWS SDK modules are only loaded at runtime when s3-bucket is actually set. Users who do not configure S3 pay zero startup cost from the new code, and the generated dist/ chunk files for the AWS SDK are never read in the common case.

Usage

MinIO (self-hosted):

- uses: coursier/cache-action@v7
  with:
    s3-bucket: ci-cache
    s3-endpoint: http://minio.internal:9000
    s3-path-style: 'true'
    s3-access-key-id: ${{ secrets.MINIO_ACCESS_KEY }}
    s3-secret-access-key: ${{ secrets.MINIO_SECRET_KEY }}

AWS S3:

- uses: coursier/cache-action@v7
  with:
    s3-bucket: my-ci-cache-bucket
    s3-region: eu-west-1
    s3-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
    s3-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

Requires tar on the runner. Supported on Linux and macOS; Windows not yet supported.

Original prompt

This section details on the original issue you should resolve

<issue_title>Consider enabling an S3 caching backend</issue_title>
<issue_description>For users of self-hosted runners, it would be great to store the caches locally, both for speed and governance.

An S3 caching backend could provide both the possibility to store locally (by self-hosting a non-AWS, S3-compatible solution like Minio) as well as an alternative to the GH cache backend.

Thanks! </issue_description>

<agent_instructions>Look into this. Can this be done easily?</agent_instructions>

Comments on the Issue (you are @copilot in this section)


💬 Send tasks to Copilot coding agent from Slack and Teams to turn conversations into code. Copilot posts an update in your thread when it's finished.

Co-authored-by: alexarchambault <7063723+alexarchambault@users.noreply.github.com>
Copy link
Member

@alexarchambault alexarchambault left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Revert all the changes you made under dist/

Copilot AI changed the title [WIP] Consider enabling an S3 caching backend Add S3-compatible caching backend for self-hosted runners Mar 17, 2026
Copilot AI requested a review from alexarchambault March 17, 2026 00:01
alexarchambault and others added 2 commits March 17, 2026 01:06
Co-authored-by: alexarchambault <7063723+alexarchambault@users.noreply.github.com>
Copy link
Contributor Author

Copilot AI commented Mar 17, 2026

Revert all the changes you made under dist/

Done in e20791e — all dist/ changes reverted (extra chunk files removed, index.js/index.js.map restored to pre-PR state).

Copy link
Member

@alexarchambault alexarchambault left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm worried about all the new files this generates under dist/ when we update it. Is this a problem in practice? Are all the new files read even if users don't use S3? Can something be done about it so that users that don't use the S3 stuff aren't penalized?

…needed

Co-authored-by: alexarchambault <7063723+alexarchambault@users.noreply.github.com>
Copy link
Contributor Author

Copilot AI commented Mar 17, 2026

I'm worried about all the new files this generates under dist/ when we update it. Is this a problem in practice? Are all the new files read even if users don't use S3? Can something be done about it...

The chunk files are generated by ncc's code-splitting for dynamic imports — they're present on disk but are only loaded at runtime when s3-bucket is set. With static imports, the entire AWS SDK would have been parsed and executed on every run. With dynamic import() (0f526dc), restore.ts and save.ts no longer import s3-cache.ts at the top level; the AWS SDK modules are only loaded if the S3 branch is actually taken. Users who don't set s3-bucket pay zero extra startup cost.

Copy link
Member

@alexarchambault alexarchambault left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI complains about formatting

> typescript-action@0.0.0 format-check
> prettier --check **/*.ts

Checking formatting...
[warn] src/save.ts
[warn] Code style issues found in the above file. Run Prettier with --write to fix.

Co-authored-by: alexarchambault <7063723+alexarchambault@users.noreply.github.com>
Copy link
Contributor Author

Copilot AI commented Mar 17, 2026

CI complains about formatting

> typescript-action@0.0.0 format-check...

Fixed in 44aeaf6 — removed the extra blank lines that were causing Prettier to fail.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Consider enabling an S3 caching backend

2 participants