diff --git a/README.md b/README.md index 9f2331b..fe021e6 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,18 @@ # pulp-rust -A Pulp plugin to support hosting your own Rust package registry. +A Pulp plugin to support hosting your own Rust/Cargo package registry. + +> **Tech Preview**: This project is currently in tech preview. APIs, behaviors, and data models are subject to change, including breaking changes, without prior notice. + +## Features + +- Use Pulp as a pull-through cache for crates.io or any Cargo sparse registry +- Host a private Cargo registry for internal crates +- Implements the [Cargo sparse registry protocol](https://doc.rust-lang.org/cargo/reference/registry-index.html#sparse-index) for compatibility with standard Cargo tooling +- Download crates on-demand to reduce disk usage +- Every operation creates a restorable snapshot with Versioned Repositories +- Host content either locally or on S3/Azure/GCP +- De-duplication of all saved content For more information, please see the [documentation](docs/index.md) or the [Pulp project page](https://pulpproject.org/). diff --git a/docs/index.md b/docs/index.md index 3dffbea..2053d5c 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,12 +1,20 @@ -# Welcome to Pulp +# Welcome to Pulp Rust -The `` plugin extends pulpcore to support hosting packages. -This plugin is a part of the Pulp Project, and assumes some familiarity with the -[pulpcore documentation](site:pulpcore/). +The `rust` plugin extends [pulpcore](site:pulpcore/) to support hosting Rust/Cargo package registries. This plugin is a part of the [Pulp Project](site:/), and assumes some familiarity with the [pulpcore documentation](site:pulpcore/). -If you are just getting started, we recommend: +!!! warning "Tech Preview" + This plugin is currently in **tech preview**. APIs, behaviors, and data models are subject to change, including breaking changes, without prior notice. -- [Getting Started with ](site:pulp_rust/docs/tutorials/getting-started.md), - for a starting out with a common use case. +If you are just getting started, we recommend getting to know the [basic workflows](site:pulp_rust/docs/user/guides/pull-through-cache/). +See the [REST API documentation](site:pulp_rust/restapi/) for detailed endpoint reference. +## Features + +- [Use Pulp as a pull-through cache](site:pulp_rust/docs/user/guides/pull-through-cache/) for crates.io or any Cargo sparse registry +- [Host a private Cargo registry](site:pulp_rust/docs/user/guides/private-registry/) for internal crates +- Implements the [Cargo sparse registry protocol](https://doc.rust-lang.org/cargo/reference/registry-index.html#sparse-index) for compatibility with standard Cargo tooling +- Download crates on-demand to reduce disk usage +- Every operation creates a restorable snapshot with Versioned Repositories +- Host content either locally or on S3/Azure/GCP +- De-duplication of all saved content diff --git a/docs/user/guides/_SUMMARY.md b/docs/user/guides/_SUMMARY.md new file mode 100644 index 0000000..a7048f6 --- /dev/null +++ b/docs/user/guides/_SUMMARY.md @@ -0,0 +1,2 @@ +* [Pull-Through Cache](pull-through-cache.md) +* [Host a Private Registry](private-registry.md) diff --git a/docs/user/guides/private-registry.md b/docs/user/guides/private-registry.md new file mode 100644 index 0000000..9e836ac --- /dev/null +++ b/docs/user/guides/private-registry.md @@ -0,0 +1,131 @@ +# Host a Private Cargo Registry + +This guide walks you through setting up Pulp as a private Cargo registry for hosting internal +crates. This is useful for organizations that need to distribute proprietary or internal-only +Rust packages. + +!!! note + Package publishing support (`cargo publish`) is not yet available but is planned for an + upcoming release. In the meantime, content can be uploaded through the Pulp REST API. + +## Create a Repository + +```bash +pulp rust repository create --name my-crates +``` + +## Create a Distribution + +A distribution makes the repository's content available to Cargo over HTTP. + +```bash +pulp rust distribution create \ + --name my-crates \ + --base-path my-crates \ + --repository my-crates +``` + +Your private registry is now served at `http:///pulp/cargo/my-crates/`. + +## Configure Cargo + +Add the private registry to your Cargo configuration. Create or edit `~/.cargo/config.toml`: + +```toml +[registries.my-crates] +index = "sparse+http:///pulp/cargo/my-crates/" +``` + +### Using the Private Registry as a Dependency Source + +To depend on crates from your private registry, specify the registry in your `Cargo.toml`: + +```toml +[dependencies] +my-internal-lib = { version = "1.0", registry = "my-crates" } +``` + +### Setting the Default Registry + +You can set your private registry as the default for `cargo publish` and other registry commands +so you don't need to pass `--registry` every time: + +```toml +[registry] +default = "my-crates" +``` + +This affects commands like `cargo publish`, `cargo yank`, and `cargo owner`. It does **not** +change where dependencies are resolved from — that is controlled by source replacement (below). + +!!! tip + Setting a default registry is recommended for organizations with private crates. Without it, + running `cargo publish` without `--registry` will publish to crates.io by default, which could + accidentally leak proprietary code to the public registry. + +### Replacing crates.io Entirely + +If you want all crate lookups to go through your private registry (for example, in an air-gapped +environment), you can replace the default source: + +```toml +[source.crates-io] +replace-with = "my-crates" + +[source.my-crates] +registry = "sparse+http:///pulp/cargo/my-crates/" +``` + +This redirects all dependency resolution — including transitive dependencies — through your +private registry. Any crate not present in the registry will fail to resolve. + +## Combining with Pull-Through Caching + +If you need both private crates and public crates.io dependencies, we recommend keeping them as +**separate registries** rather than mixing them into one. This avoids +[dependency confusion](https://medium.com/@alex.birsan/dependency-confusion-4a5d60fec610) attacks, +where a malicious package on a public registry could impersonate a private dependency. + +```bash +# Set up a separate pull-through cache for crates.io +pulp rust remote create --name crates-io --url "sparse+https://index.crates.io/" --policy on_demand +pulp rust repository create --name crates-io-cache --remote crates-io --retain-repo-versions 1 +pulp rust distribution create \ + --name crates-io-cache \ + --base-path crates-io-cache \ + --repository crates-io-cache \ + --remote crates-io +``` + +Then configure Cargo to use both registries, with crates.io going through the cache and private +crates resolved from your private registry: + +```toml +[registries.my-crates] +index = "sparse+http:///pulp/cargo/my-crates/" + +[source.crates-io] +replace-with = "crates-io-cache" + +[source.crates-io-cache] +registry = "sparse+http:///pulp/cargo/crates-io-cache/" +``` + +```toml +[dependencies] +serde = "1.0" # resolved from crates-io-cache +my-internal-lib = { version = "1.0", registry = "my-crates" } # resolved from private registry +``` + +!!! warning + Avoid adding a public remote (such as crates.io) to a private registry's distribution. Mixing + public and private packages in a single registry index creates a risk of dependency confusion + attacks, where an attacker publishes a crate on the public registry with the same name as one + of your private crates. + +## Further Reading + +- [Cargo registries configuration](https://doc.rust-lang.org/cargo/reference/registries.html) -- configuring alternate registries in Cargo +- [Cargo source replacement](https://doc.rust-lang.org/cargo/reference/source-replacement.html) -- replacing crates.io with an alternate source +- [Cargo config reference](https://doc.rust-lang.org/cargo/reference/config.html) -- full reference for `.cargo/config.toml` +- [Specifying dependencies](https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html#specifying-dependencies-from-other-registries) -- using dependencies from alternate registries diff --git a/docs/user/guides/pull-through-cache.md b/docs/user/guides/pull-through-cache.md new file mode 100644 index 0000000..d0587e0 --- /dev/null +++ b/docs/user/guides/pull-through-cache.md @@ -0,0 +1,109 @@ +# Use Pulp as a Pull-Through Cache + +This guide walks you through setting up Pulp as a pull-through cache for an upstream Cargo registry +such as [crates.io](https://crates.io). Cargo will fetch crates through Pulp, which transparently +downloads and stores them on first access. Subsequent requests are served directly from Pulp's +storage backend. + +## Create a Remote + +A remote tells Pulp where to fetch crates from. The URL should point to the sparse index of +the upstream registry. + +```bash +pulp rust remote create --name crates-io --url "sparse+https://index.crates.io/" --policy on_demand +``` + +The `--policy` flag controls how content is stored: + +| Policy | Behavior | +|-------------|---------------------------------------------------------------------------------------------| +| `on_demand` | Downloads and caches crate files on first request. Metadata is saved to the Pulp database. | +| `streamed` | Streams crate files to the client without saving them locally. | + +For a pull-through cache, `on_demand` is recommended so that crates are retained for future +requests. `streamed` is primarily useful in cases where you want a proxy in between clients and +the public registry for e.g. access control - which is not so common in the Rust ecosystem. + +## Create a Repository + +Repositories store and organize content in Pulp. When using pull-through caching, crates are +automatically added to the repository as they are downloaded. + +```bash +pulp rust repository create --name crates-io-cache --remote crates-io --retain-repo-versions 1 +``` + +!!! tip + Setting `--retain-repo-versions 1` is recommended for pull-through caches. Each new crate + download creates a new repository version, and without this setting the number of versions + will grow unboundedly. + +## Create a Distribution + +A distribution makes the repository available to Cargo over HTTP. The `--base-path` determines the +URL path where the registry is served. + +```bash +pulp rust distribution create \ + --name crates-io-cache \ + --base-path crates-io-cache \ + --repository crates-io-cache \ + --remote crates-io +``` + +!!! note + The remote must be set on both the repository and the distribution. The distribution's remote + controls the proxy fallback for the sparse index, while the repository's remote is used for + pull-through content storage. + +Your registry is now available at `http:///pulp/cargo/crates-io-cache/`. + +## Configure Cargo + +To use your Pulp instance as a registry, add it to your Cargo configuration. Create or edit +`~/.cargo/config.toml`: + +```toml +[registries.pulp] +index = "sparse+http:///pulp/cargo/crates-io-cache/" + +[source.crates-io] +replace-with = "pulp" + +[source.pulp] +registry = "pulp" +``` + +This tells Cargo to route all crate downloads through Pulp instead of directly to crates.io. + +!!! tip + You can also set this per-project by placing a `.cargo/config.toml` file in your project root. + +Now any `cargo build`, `cargo add`, or `cargo install` command will fetch crates through Pulp: + +```bash +cargo add serde --features derive +cargo build +``` + +The first time a crate is requested, Pulp fetches it from the upstream registry and caches it +locally. All subsequent requests for the same crate version are served from Pulp's storage backend. + +## Verify Cached Content + +You can inspect what content has been cached in Pulp: + +```bash +# List cached crates +pulp rust content list --limit 10 + +# Check repository versions +pulp rust repository version list --repository crates-io-cache +``` + +## Further Reading + +- [Cargo registries configuration](https://doc.rust-lang.org/cargo/reference/registries.html) -- configuring alternate registries in Cargo +- [Cargo source replacement](https://doc.rust-lang.org/cargo/reference/source-replacement.html) -- replacing crates.io with an alternate source +- [Cargo config reference](https://doc.rust-lang.org/cargo/reference/config.html) -- full reference for `.cargo/config.toml`