chore: merge new changes from ipfs/kubo master#2
Open
alvin-reyes wants to merge 826 commits intoIPFSR:masterfrom
Open
chore: merge new changes from ipfs/kubo master#2alvin-reyes wants to merge 826 commits intoIPFSR:masterfrom
alvin-reyes wants to merge 826 commits intoIPFSR:masterfrom
Conversation
* rpc: don't reuse object during decode Retaining the object during the loop will make fields such as `Name` stick between iterations. This patch decodes into a new struct each iteration, assuring we don't retain values from other pins. * rpc: use the Detailed option during request
* reprovide sweep draft * update reprovider dep * go mod tidy * fix provider type * change router type * dual reprovider * revert to provider.System * back to start * SweepingReprovider test * fix nil pointer deref * noop provider for nil dht * disabled initial network estimation * another iteration * suppress missing self addrs err * silence empty rt err on lan dht * comments * new attempt at integrating * reverting changes in core/node/libp2p/routing.go * removing SweepingProvider * make reprovider optional * add noop reprovider * update KeyChanFunc type alias * restore boxo KeyChanFunc * fix missing KeyChanFunc * test(sharness): PARALLEL=1 and timeout 30m running sequentially to see where timeout occurs * initialize MHStore * revert workflow debug * config * config docs * merged IpfsNode provider and reprovider * move Provider interface to from kad-dht to node * moved Provider interface from kad-dht to kubo/core/node * mod_tidy * Add Clear to Provider interface * use latest kad-dht commit * make linter happy * updated boxo provide interface * boxo PR fix * using latest kad-dht commit * use latest boxo release * fix fx * fx cyclic deps * fix merge issues * extended tests * don't provide LAN DHT * docs * restore dual dht provider * don't start provider before it is online * address linter * dual/provider fix * add delay in provider tests for dht bootstrap * add OfflineDelay parameter to config * remove increase number of workers in test * improved keystore gc process * fix: replace incorrect logger import in coreapi replaced github.com/labstack/gommon/log with the standard github.com/ipfs/go-log/v2 logger used throughout kubo. removed unused labstack dependency from go.mod files. * fix: remove duplicate WithDefault call in provider config * fix: use correct option method for burst workers * fix: improve error messages for experimental sweeping provider updated error messages to clearly indicate when commands are unavailable due to experimental sweeping provider being enabled via Reprovider.Sweep.Enabled=true * docs: remove obsolete KeyStoreGCInterval config removed from config.md as option no longer exists (removed in b540fba) updated keystore description to reflect gc happens at reprovide interval * docs: add TODO placeholder changelog for experimental sweeping DHT provider using v0.38-TODO.md name to avoid merge conflicts with master branch and allow CI tests to run. will be renamed to v0.38.md once config migration is added to the PR * fix: provideKeysRec go routine * clear keystore on close * fix: datastore prefix * fix: improve error handling in provideKeysRec - close errCh channel to distinguish between nil and pending errors - check for pending errors when provided.New closes - handle context cancellation during error send - prevent race condition where errors could be silently lost this ensures DAG walk errors are always propagated correctly * address gammazero's review * rename BurstProvider to LegacyProvider * use latest provider/keystore * boxo: make mfs StartProviding async * bump boxo * chore: update boxo to f2b4e12fb9a8ac138ccb82aae3b51ec51d9f631c - updated boxo dependency to specified commit - updated go.mod and go.sum files across all modules * use latest kad-dht/boxo * Buffered SweepingProvider wrapper * use latest kad-dht commit * allow no DHT router * use latest kad-dht & boxo --------- Co-authored-by: Marcin Rataj <lidel@lidel.org> Co-authored-by: gammazero <11790789+gammazero@users.noreply.github.com>
…o Provide.DHT (#10951) * refactor: consolidate Provider/Reprovider into unified Provide config - merge Provider and Reprovider configs into single Provide section - add fs-repo-17-to-18 migration for config consolidation - improve migration ergonomics with common package utilities - convert deprecated "flat" strategy to "all" during migration - improve Provide docs * docs: add total_provide_count metric guidance - document how to monitor provide success rates via prometheus metrics - add performance comparison section to changelog - explain how to evaluate sweep vs legacy provider effectiveness * fix: add OpenTelemetry meter provider for metrics - set up meter provider with Prometheus exporter in daemon - enables metrics from external libs like go-libp2p-kad-dht - fixes missing total_provide_count_total when SweepEnabled=true - update docs to reflect actual metric names --------- Co-authored-by: gammazero <11790789+gammazero@users.noreply.github.com> Co-authored-by: guillaumemichel <guillaume@michel.id> Co-authored-by: Daniel Norman <1992255+2color@users.noreply.github.com> Co-authored-by: Hector Sanjuan <code@hector.link>
Bumps [github.com/go-viper/mapstructure/v2](https://github.com/go-viper/mapstructure) from 2.2.1 to 2.4.0. - [Release notes](https://github.com/go-viper/mapstructure/releases) - [Changelog](https://github.com/go-viper/mapstructure/blob/main/CHANGELOG.md) - [Commits](go-viper/mapstructure@v2.2.1...v2.4.0) --- updated-dependencies: - dependency-name: github.com/go-viper/mapstructure/v2 dependency-version: 2.4.0 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* fix: use CheckIfPinnedWithType for pin ls with names updates to use CheckIfPinnedWithType method from ipfs/boxo#1035, enabling efficient pin name retrieval for 'ipfs pin ls <cid> --names' - uses new CheckIfPinnedWithType from boxo for type-specific pin checks - pin names are now returned when listing specific CIDs with --names flag * test: add CLI tests for pin ls with names tests cover: - pin ls with specific CIDs returning names - pin ls without CID listing all pins with names - pin ls with --type and --names combinations - JSON output with and without names - pin update preserving names - error cases (invalid CID, unpinned CID) * docs: add pin name improvements to v0.38 changelog covers fix for ipfs pin ls --names with specific CIDs and RPC pin name leak fix * fix(rpc): support pin names in Add() passes the Name field from PinAddSettings to the API request adds test to verify pin names work via RPC * test: add coverage for pin names functionality - test special characters, unicode, long names - test concurrent operations - test persistence across daemon restarts - test garbage collection preservation - fix indirect pin test logic * chore: boxo@main with boxo#1039 * fix(pin): improve pin ls robustness and validation - add nil check for n.Pinning with early fail-fast validation - use pin.StringToMode() for consistent type validation - add edge case tests for invalid types and unpinned CIDs
* fix: prevent --flush=false in 'ipfs files rm' command the 'ipfs files rm' command always flushes for safety to ensure data integrity. this change adds an explicit error when users try to pass --flush=false, improving ux and preventing confusion. related to #10842 * fix: add MFS cache size limit to prevent unbounded growth - add Internal.MFSAutoflushThreshold config (experimental) - directories auto-flush when cache exceeds threshold with --flush=false - prevents high memory usage issue from #10842 - default: 256 entries per directory (matching HAMT shard size) - set to 0 to restore old behavior (risky, may cause errors) Closes #10842
* fix(webui): show helpful errors for incompatible configurations - show error when Gateway.NoFetch=true and WebUI is not available locally - show error when Gateway.DeserializedResponses=false (incompatible) - add tests for both error scenarios * chore(webui): update to v4.9.0 https://github.com/ipfs/ipfs-webui/releases/tag/v4.9.0 * docs: add WebUI v4.9.0 update to v0.38 changelog - highlight new diagnostics screen for troubleshooting - include screenshots of key features in table format - add local access URL for WebUI - update TOC with new sections
preserve private use characters as specified in libp2p/specs#491 enforce 128 rune limit on untrusted peer data
Bumps [actions/setup-node](https://github.com/actions/setup-node) from 4 to 5. - [Release notes](https://github.com/actions/setup-node/releases) - [Commits](actions/setup-node@v4...v5) --- updated-dependencies: - dependency-name: actions/setup-node dependency-version: '5' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [actions/setup-go](https://github.com/actions/setup-go) from 5 to 6. - [Release notes](https://github.com/actions/setup-go/releases) - [Commits](actions/setup-go@v5...v6) --- updated-dependencies: - dependency-name: actions/setup-go dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [hadolint/hadolint-action](https://github.com/hadolint/hadolint-action) from 3.1.0 to 3.2.0. - [Release notes](https://github.com/hadolint/hadolint-action/releases) - [Changelog](https://github.com/hadolint/hadolint-action/blob/master/.releaserc) - [Commits](hadolint/hadolint-action@v3.1.0...v3.2.0) --- updated-dependencies: - dependency-name: hadolint/hadolint-action dependency-version: 3.2.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 5.5.0 to 5.5.1. - [Release notes](https://github.com/codecov/codecov-action/releases) - [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md) - [Commits](codecov/codecov-action@fdcc847...5a10915) --- updated-dependencies: - dependency-name: codecov/codecov-action dependency-version: 5.5.1 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* ci: optimize build workflows - use go version from go.mod instead of hardcoding - group platforms by OS for parallel builds - remove legacy try-build targets * fix: checkout before setup-go in all workflows setup-go needs go.mod to be present, so checkout must happen first * chore: remove deprecated // +build syntax go 1.17+ uses //go:build, the old syntax is no longer needed * simplify: remove nofuse tag from CI workflows - workflows now rely on platform build constraints - keep make nofuse target for manual builds - remove unused appveyor.yml * ci: remove legacy travis variable and fix gateway-conformance - remove TRAVIS env variable from 4 workflows - fix gateway-conformance checkout path to match working-directory - replace deprecated cache-go-action with built-in setup-go caching
* docs: improve slow reprovide warning messages simplify warning text and provide actionable solutions in order of preference * feat(config): add validation for Provide.DHT settings - validate interval doesn't exceed DHT record validity (48h) - validate worker counts and other parameters are within valid ranges - improve slow reprovide warning messages to reference config parameter - add tests for all validation cases * docs: add reprovide cycle visualization shows traffic patterns of legacy vs sweep vs accelerated DHT
* feat: allow custom http provide when offline * refactor: improve offline HTTP provider handling and tests - fixed comment/function name mismatch - added mock server test for HTTP provide success - clarified test names for offline scenarios * test: simplify single-node provider tests use h.NewNode().Init() instead of NewNodes(1) for cleaner test setup * fix: allow SweepingProvider to work with HTTP-only routing when no DHT is available but HTTP routers are configured for providing, return NoopProvider instead of failing. this allows the daemon to start and HTTP-based providing to work through the routing system. moved HTTP provider detection to config package as HasHTTPProviderConfigured() for better code organization and reusability. this fix is important as SweepingProvider will become the new default in the future. --------- Co-authored-by: Marcin Rataj <lidel@lidel.org>
Bumps [actions/github-script](https://github.com/actions/github-script) from 7 to 8. - [Release notes](https://github.com/actions/github-script/releases) - [Commits](actions/github-script@v7...v8) --- updated-dependencies: - dependency-name: actions/github-script dependency-version: '8' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* fix: SweepingProvider shouldn't error when missing DHT * fix: prevent panic when SweepingProvider has no DHT when SweepingProvider is enabled but no DHT is available (e.g., Routing.Type=none), the daemon would panic with a nil pointer dereference in ResettableKeystore.ResetCids. this fix: - returns NoopProvider when no DHT implementation is available - skips keystore initialization for NoopProvider to avoid unnecessary operations - allows nodes to run without DHT when using HTTP-only routing or offline mode the panic occurred because initKeyStore tried to access a nil keystore when SweepingProvider returned nil for the keystore parameter. by checking if the provider is NoopProvider and skipping keystore operations, we avoid the panic while maintaining correct behavior for all other provider types. cc #10974 #10975 --------- Co-authored-by: Marcin Rataj <lidel@lidel.org>
- bump version to v0.39.0-dev - add v0.39 changelog template - update CHANGELOG.md with v0.39 link
Bumps [hadolint/hadolint-action](https://github.com/hadolint/hadolint-action) from 3.2.0 to 3.3.0. - [Release notes](https://github.com/hadolint/hadolint-action/releases) - [Changelog](https://github.com/hadolint/hadolint-action/blob/master/.releaserc) - [Commits](hadolint/hadolint-action@v3.2.0...v3.3.0) --- updated-dependencies: - dependency-name: hadolint/hadolint-action dependency-version: 3.3.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* fix: SweepingProvider slow start #10979 * don't purge keystore * feat: add INFO logging for provider keystore sync log start/completion of async keystore sync with strategy --------- Co-authored-by: Marcin Rataj <lidel@lidel.org>
adds validation to ensure pin names don't exceed 255 bytes across all commands that accept pin names. this prevents issues with filesystem limitations and improves compatibility. affected commands: - ipfs pin add --name - ipfs add --pin-name - ipfs pin ls --name (filter) - ipfs pin remote add --name - ipfs pin remote ls --name (filter) - ipfs pin remote rm --name (filter)
* Filestore: provide Filestore nodes When strategy is set to "all" (the blockstore does all the providing when a block is written), no providing was happening to Filestore blocks that were not written to the underlying blockstore (so, the DAG leaves, as they live in the filesystem directly). This fixes that. * docs: clarify filestore and urlstore fix in changelog both filestore (local file references) and urlstore (HTTP/HTTPS URL references) blocks are now properly provided shortly after initial add
* fix: add MFS operation limit for --flush=false adds a global counter that tracks consecutive MFS operations performed with --flush=false and fails with clear error after limit is reached. this prevents unbounded memory growth while avoiding the data corruption risks of auto-flushing. - adds Internal.MFSNoFlushLimit config - operations fail with actionable error at limit - counter resets on successful flush or any --flush=true operation - operations with --flush=true reset and don't count this commit removes automatic flush from #10971 and instead errors to encourage users of --flush=false to develop a habit of calling 'ipfs files flush' periodically. boxo will no longer auto-flush (ipfs/boxo#1041) to avoid corruption issues, and kubo applies the limit to 'ipfs files' commands instead. closes #10842 * test: add tests for MFSNoFlushLimit tests verify the new Internal.MFSNoFlushLimit config option: - default limit of 256 operations - custom limit configuration - counter reset on flush=true - counter reset on explicit flush command - limit=0 disables the feature - multiple MFS command types count towards limit * docs: explain why MFS operations fail instead of auto-flushing addresses feedback from #10985 (review) - clarify that automatic flushing at limit was considered but rejected - explain the data corruption risks of auto-flushing - guide users who want auto-flush to use --flush=true (default) - document benefits of explicit failure for batch operations
Co-authored-by: Marcin Rataj <lidel@lidel.org>
- update boxo to v0.34.1-0.20250926171300-4c0aa3a121fb - update go-libp2p-kad-dht to v0.34.1-0.20250926161957-861573b39723 - update changelog to reference webui v4.9.1
Bumps [hadolint/hadolint-action](https://github.com/hadolint/hadolint-action) from 3.2.0 to 3.3.0. - [Release notes](https://github.com/hadolint/hadolint-action/releases) - [Changelog](https://github.com/hadolint/hadolint-action/blob/master/.releaserc) - [Commits](hadolint/hadolint-action@v3.2.0...v3.3.0) --- updated-dependencies: - dependency-name: hadolint/hadolint-action dependency-version: 3.3.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (cherry picked from commit 46d438f)
adds validation to ensure pin names don't exceed 255 bytes across all commands that accept pin names. this prevents issues with filesystem limitations and improves compatibility. affected commands: - ipfs pin add --name - ipfs add --pin-name - ipfs pin ls --name (filter) - ipfs pin remote add --name - ipfs pin remote ls --name (filter) - ipfs pin remote rm --name (filter) (cherry picked from commit 1107ac4)
Merge release v0.40.1
- core/corehttp/webui.go: bump WebUIPath CID to v4.12.0, add v4.11.1 to historical list - docs/changelogs/v0.41.md: add webui improvements highlight (ipv6 geolocation, peers screen optimizations)
Bumps [docker/login-action](https://github.com/docker/login-action) from 3 to 4. - [Release notes](https://github.com/docker/login-action/releases) - [Commits](docker/login-action@v3...v4) --- updated-dependencies: - dependency-name: docker/login-action dependency-version: '4' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* chore(deps): bump actions/upload-artifact from 6 to 7 Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 6 to 7. - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](actions/upload-artifact@v6...v7) --- updated-dependencies: - dependency-name: actions/upload-artifact dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * chore(deps): bump actions/download-artifact from 7 to 8 --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Marcin Rataj <lidel@lidel.org>
- point to canonical URLs instead of github PR links - only updated active docs and v0.40 changelog - left old changelogs and unmerged IPIPs as-is
…in the ipfs-ecosystem group (#11219) * chore(deps): bump github.com/ipfs/go-ipld-legacy Bumps the ipfs-ecosystem group with 1 update: [github.com/ipfs/go-ipld-legacy](https://github.com/ipfs/go-ipld-legacy). Updates `github.com/ipfs/go-ipld-legacy` from 0.2.2 to 0.3.0 - [Release notes](https://github.com/ipfs/go-ipld-legacy/releases) - [Commits](ipfs/go-ipld-legacy@v0.2.2...v0.3.0) --- updated-dependencies: - dependency-name: github.com/ipfs/go-ipld-legacy dependency-version: 0.3.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: ipfs-ecosystem ... Signed-off-by: dependabot[bot] <support@github.com> * chore: run make mod_tidy --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>
Bumps [ipfs/gateway-conformance](https://github.com/ipfs/gateway-conformance) from 0.11 to 0.12. - [Release notes](https://github.com/ipfs/gateway-conformance/releases) - [Changelog](https://github.com/ipfs/gateway-conformance/blob/main/CHANGELOG.md) - [Commits](ipfs/gateway-conformance@v0.11...v0.12) --- updated-dependencies: - dependency-name: ipfs/gateway-conformance dependency-version: '0.12' dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 6 to 7. - [Release notes](https://github.com/docker/build-push-action/releases) - [Commits](docker/build-push-action@v6...v7) --- updated-dependencies: - dependency-name: docker/build-push-action dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action) from 3 to 4. - [Release notes](https://github.com/docker/setup-buildx-action/releases) - [Commits](docker/setup-buildx-action@v3...v4) --- updated-dependencies: - dependency-name: docker/setup-buildx-action dependency-version: '4' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [docker/setup-qemu-action](https://github.com/docker/setup-qemu-action) from 3 to 4. - [Release notes](https://github.com/docker/setup-qemu-action/releases) - [Commits](docker/setup-qemu-action@v3...v4) --- updated-dependencies: - dependency-name: docker/setup-qemu-action dependency-version: '4' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* test: increase GetClosestPeers DHT timeout from 2m to 5m passing runs finish in 8-48s, but WAN DHT bootstrap occasionally fails entirely in CI (~4.3% flake rate on master), hitting the 2m ceiling. bumping to 5m gives more retry attempts without affecting passing runs. * test: fix flaky TestThreeLeggedCatTransfer - add 3-minute context timeout instead of unbounded context.Background(), so failures produce a meaningful error instead of burning 10 minutes - reduce data from 100 MB to 1 MB; the test verifies three-legged DHT discovery + bitswap transfer, not bulk throughput - explicitly provide root CID to DHT before catter fetches, eliminating the race between async reprovider and immediate Get
* fix: validate --max-hamt-fanout CLI flag per UnixFS spec the CLI flag bypassed the config validation in ValidateImportConfig, allowing spec-noncompliant values (e.g. 3 or 999999) to be silently accepted. validation now happens in the options layer, covering both CLI and programmatic API usage. * fix: simplify HAMT fanout constraint phrasing Co-authored-by: Guillaume Michel <15075495+guillaumemichel@users.noreply.github.com>
…#11238) * fix(core/commands/pin): return error if listing an invalid, but known, pin type * test: add cli test for pin ls with known but non-listable type Covers the case where --type=internal passes boxo's StringToMode validation but is rejected by options.Pin.Ls.Type, which previously caused a panic instead of returning an error. --------- Co-authored-by: Marcin Rataj <lidel@lidel.org>
* fix: resolve provider addresses dynamically for HTTP routing HTTP routing provider records now use addresses resolved at provide-time instead of static config values captured at daemon startup. This fixes nodes with default 0.0.0.0 swarm addresses sending unresolved addresses to HTTP routers. - routing/delegated.go: add AddrFunc to ExtraHTTPParams, use WithProviderInfoFunc when set - core/node/libp2p/routingopt.go: add httpRouterAddrFunc that prefers AutoNAT V2 confirmed reachable addrs, falls back to static config; Addresses.Announce is a full static override, AppendAnnounce is appended to dynamic/fallback results - bump boxo to ipfs/boxo#1115 (WithProviderInfoFunc support) Fixes #11213 * refactor: remove redundant static addrs from HTTP routing Now that AddrFunc is always set by all callers, the static Addrs field and httpAddrsFromConfig are dead weight. - routing/delegated.go: remove Addrs from ExtraHTTPParams and addrs param from ConstructHTTPRouter, remove unused createAddrInfo - core/node/libp2p/routingopt.go: remove httpAddrsFromConfig, drop redundant httpAddrsFromConfig calls from all callers - core/node/libp2p/routingopt_test.go: remove TestHttpAddrsFromConfig (covered by TestHttpRouterAddrFunc), convert to table-driven tests * fix: always append AppendAnnounce addrs in HTTP routing AppendAnnounce was silently dropped when Announce was set, breaking the documented behavior that AppendAnnounce is always appended. * refactor: add compile-time assertion for confirmedAddrsHost * fix: harden parseMultiaddrs and prevent static addr slice aliasing - log error on invalid multiaddr strings instead of silently dropping - return cloned slice in static Announce path to prevent caller mutation * perf: reduce allocations in httpRouterAddrFunc closure - precompute fallback result (both parts immutable, no per-call concat) - cache type assertion at construction time - skip concat when AppendAnnounce is empty - return static Announce slice directly (boxo already copies) * chore: update boxo to main with ipfs/boxo#1115 merged * docs: clarify changelog entry for custom HTTP routing provider addresses
* fix(provider): purge keystore datastore after reset * changelog * use MapDatastore if no datastore is configured * bump kad-dht to latest commit * purge orphaned keystore migration * bump kad-dht * use main datastore for keystore "meta" store * add provider/keystore/0 and /1 to ipfs diag command mount keystore datastores to /provider/keystore/0 and /1 so that they are included in the ipfs diag datastore command * fix(provider): reject unexpected keystore suffix to prevent stray deletions destroyDs calls os.RemoveAll with a suffix from the upstream library. If suffix were ever ".." or empty, this could delete wrong directories. Validate that suffix is "0" or "1" in both createDs and destroyDs. * fix(provider): close opened datastores when mounting partially fails If opening datastore "0" succeeds but "1" fails, MountKeystoreDatastores returned an error without closing "0". * fix(provider): defer batch creation in orphan purge until keys are found Avoids allocating a datastore batch when no orphaned keys exist. * fix(provider): warn on unrecognized datastore wrapper types findRootDatastoreSpec silently returns wrapper specs it doesn't know about. If a plugin adds a wrapper with a "child" field, openDatastoreAt gets the wrapper instead of the leaf backend and fails confusingly. Log a warning so operators can spot the issue. * docs: document keystore migration behavior on upgrade and downgrade - explain why context.Background() is used in the migration code - add changelog note about the provide cycle restarting on upgrade - add downgrade caveat about orphaned provider-keystore directory * chore(deps): bump go-libp2p-kad-dht to latest keystore factory commit * fix(provider): harden keystore migration and spec handling - chunk orphan purge into 4096-key batches to bound memory and match existing batching patterns in the same file - cancel the purge context via fx.Lifecycle OnStop so SIGINT during startup does not block indefinitely - deep-copy slices in copySpec (not just maps) so the function matches its documented "deep-copy" contract - return nil from findRootDatastoreSpec when no "/" mount exists, so callers fall back to in-memory instead of passing a mount-type spec to openDatastoreAt - rename local variable to avoid shadowing the mount package import * test(provider): add migration purge test and diag datastore put command - add `ipfs diag datastore put` subcommand for writing arbitrary key-value pairs to the datastore (offline, experimental) - add DatastorePut harness helper for CLI tests - add TestProviderKeystoreMigrationPurge: seeds orphaned keystore keys via `put`, starts the daemon to trigger migration, verifies the orphaned keys are purged and provider-keystore/ dir is created - add put/get roundtrip test for diag datastore * chore(deps): bump go-libp2p-kad-dht to 1bede74b8246 * fix(provider): log keystore datastore create and destroy operations * docs: rewrite provider keystore changelog to focus on user impact * bump kad-dht@master --------- Co-authored-by: Marcin Rataj <lidel@lidel.org>
* feat(cmd): add 'ipfs cid inspect' command
Adds a new subcommand to inspect and display detailed CID information
including version, multibase encoding, multicodec, and multihash
components. Also shows equivalent CIDv0/CIDv1 representations.
Example output:
CID: bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi
Version: 1
Multibase: base32 (b)
Multicodec: dag-pb (0x70)
Multihash: sha2-256 (0x12)
Length: 32 bytes
Digest: c3c4733ec8affd06cf9e9ff50ffc6bcd2ec85a6170004bb709669c31de94391a
CIDv0: QmbWqxBEKC3P8tqsKc98xmWNzrzDtRLMiMPL8wBuTGsMnR
CIDv1: bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi
Supports --enc=json for machine-readable output.
Fixes #11205
* refactor: tidy up CidInspectRes struct and add '/cid/inspect' route to tests
This commit refines the CidInspectRes struct for better readability and consistency. Additionally, it includes the '/cid/inspect' route in the command tests to ensure comprehensive coverage of the new CID inspection functionality.
* docs: update changelog for v0.42 to include new `ipfs cid inspect` command
Added a section highlighting the new `ipfs cid inspect <cid>` command, detailing its functionality to display comprehensive CID information, including version, encoding, and hash details. The command supports machine-readable output and operates offline.
* feat(cmd): improve ipfs cid inspect
- multibase: always shown (implicit for CIDv0), prefix as string
- multicodec/multihash: annotated (implicit) for CIDv0
- digest: uppercase hex with 0x prefix
- cidV0: empty in JSON when not possible, text encoder explains why
- cidV1: base36 for libp2p-key codec, base32 otherwise
- errors: ErrorMsg kept for HTTP RPC API, text encoder returns non-zero exit
- PeerID fallback: helpful hint with equivalent CID on invalid input
- unknown codec/hash: graceful "unknown" label
- stdin support via .EnableStdin()
- inspect listed first in subcommands, cid format points to inspect
- cli tests for all cases including JSON, PeerID, unknown codec
* chore: move cid inspect changelog to v0.41
- digest: bare lowercase hex (no 0x prefix), matching sha256sum
---------
Co-authored-by: Marcin Rataj <lidel@lidel.org>
`ipfs key export` was using `os.Create` (0o666 pre-umask, typically 0o644) making exported private keys world-readable on multi-user systems. Use `os.OpenFile` with 0o600 to match the restrictive permissions the keystore itself uses for key files.
* fix(object): validate UnixFS type in patch add-link Reject adding named links to non-directory nodes in `object patch add-link`, which previously produced invalid DAGs silently. - reject UnixFS File/Symlink/etc nodes (only Directory and HAMTShard support named links per the UnixFS spec) - reject non-UnixFS dag-pb nodes (no UnixFS metadata to validate) - add `--allow-non-unixfs` flag to bypass both checks - pass `allow-non-unixfs` in client/rpc when SkipUnixFSValidation is set - test all three node types: bare dag-pb, UnixFS File, UnixFS Directory - reproduce the exact data-loss scenario from #7190 Fixes: #7190 * fix(object): reject HAMTShard in patch add-link dagutils.Editor operates at the dag-pb level and does not update HAMT bitfields, so mutating HAMTShard nodes produces corrupt DAGs. - reject HAMTShard in add-link (was incorrectly allowed) - update help text to note dag-pb limitations and suggest ipfs files - add HAMT test cases to sharness and API tests - expect full error strings in all validation tests - update changelog to cover all rejected node types * fix(object): validate UnixFS type in patch rm-link Same issue as add-link: dagutils.Editor operates at the dag-pb level and cannot update UnixFS metadata, so mutating non-Directory nodes produces corrupt DAGs. - add UnixFS validation to rm-link (Directory allowed, all else rejected) - add --allow-non-unixfs flag to rm-link command - add ObjectRmLinkSettings/ObjectRmLinkOption types - update ObjectAPI.RmLink interface to accept options - pass allow-non-unixfs in client/rpc - update rm-link help text to note dag-pb limitations - add rm-link validation tests for all four node types
Use request-scoped otelhttp.Labeler instead, which is the recommended replacement per upstream otelhttp v0.67.0. - add withMetricLabels helper wrapping inner handler - migrate all 4 call sites in commands.go and gateway.go
* test(cli): add CARv2 import over HTTP API test Regression test for #9361. Imports a CARv2 fixture via the daemon (online mode) and verifies the blocks are accessible. Currently fails with "operation not supported" due to the multipart reader not supporting seeking. * fix(cmd): support CARv2 import over HTTP API Strip the io.Seeker interface from the file before passing it to go-car's NewBlockReader. Over the HTTP API the underlying reader is a multipart stream that cannot seek, but boxo's ReaderFile advertises io.Seeker and returns ErrNotSupported at runtime. Hiding the interface lets go-car fall back to forward-only reading. Fixes #9361 --------- Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>
Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 5.5.2 to 5.5.3. - [Release notes](https://github.com/codecov/codecov-action/releases) - [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md) - [Commits](codecov/codecov-action@671740a...1af5884) --- updated-dependencies: - dependency-name: codecov/codecov-action dependency-version: 5.5.3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>
) * fix(MFS): fix deadlock, attrs, caching * unmount ipns and mfs in mount tests; allow offline * set attrs Uid, Gid, and Valid for readonly and /ipns * doc: update changelog * fix(fuse): maximize kernel cache for immutable /ipfs paths /ipfs content is addressed by CID and never changes, so kernel attribute caching is safe and avoids unnecessary FUSE round-trips. Also sets uid/gid on Root.Attr for consistency. * docs: move FUSE changelog to v0.41 highlights * fix(fuse): make IPNS fsync a no-op Calling fsync on a file opened through /ipns deadlocks and eventually panics, taking down the entire IPNS mount. The Fsync handler called mfs.File.Flush(), which tries to open a second write descriptor on the same file. Only one write descriptor can exist at a time (desclock is exclusive), and the first one from Open is still held. The new one blocks forever waiting for the lock. After the FUSE timeout, Release tries to close the original descriptor and hits a nil pointer panic in DagModifier.Sync. Make Fsync a no-op, matching the MFS mount. Data gets flushed when the file is closed. Also improve the MFS Fsync comment to explain the same constraint. * fix(fuse): set uid/gid on IPNS symlinks The "local" symlink in /ipns showed uid=0 gid=0 (root) while directories and files showed the daemon's uid/gid. Set uid/gid and disable attr caching to match other mutable IPNS nodes. Also add TODO comments across all three FUSE mounts for using Mode and Mtime from UnixFS records when present, and for wiring IPNS record TTL into attr cache duration. * fix(fuse): return empty listing for empty directories IPNS Directory.ReadDirAll and readonly Node.ReadDirAll returned ENOENT when a directory had no children. An empty directory still exists, it just has nothing in it. Return an empty slice instead. MFS already handles this correctly. The readonly Root.ReadDirAll correctly returns EPERM (you can't list all of /ipfs). The IPNS Root.ReadDirAll always has entries (peer keys), so it was never affected. This matters for /ipfs because empty directories are valid content-addressed objects (e.g. QmUNLLsPACCz1vLxQVkXqqLX5R1X345qqfHbsf67hvA3Nn is a well-known CID of an empty UnixFS directory). * fix(fuse): always sync MFS writes to root on close The Sync flag was computed as `req.Flags|fuse.OpenSync > 0` (bitwise OR), which is always true because fuse.OpenSync is non-zero. Replace with an explicit `true` to match the IPNS mount and make the intent clear: FUSE writes must always propagate to the MFS root when the file is closed, regardless of whether the caller set O_SYNC. * docs: update FUSE changelog for new fixes * test(fuse): add empty directory listing tests Verify that listing an empty directory returns an empty result instead of an error, for all three FUSE mounts: - /mfs: empty root + empty subdirectory - /ipns: empty peer directory + empty subdirectory - /ipfs: empty UnixFS directory added to the DAG * test(fuse): add append and byte-at-a-time write tests for MFS IPNS had TestAppendFile and TestMultiWrite but MFS did not. Add matching tests to cover appending to an existing file and writing one byte at a time. * ci(fuse): add dedicated FUSE test job with auto-detection Add a fuse-tests CI job that installs fuse3, sets TEST_FUSE=1, and runs FUSE unit tests. Previously these tests were compiled out by the nofuse build tag (set when TEST_FUSE=0 in the unit-tests job). Introduce fuse/fusetest package with shared test helpers: - SkipUnlessFUSE: respects TEST_FUSE env var (0=skip, 1=run) with auto-detection fallback that checks for fusermount in PATH - MountError: fatals when TEST_FUSE=1 (CI expects FUSE to work), skips when auto-detecting (local dev without FUSE) Replace the old ci.NoFuse() (checked TEST_NO_FUSE, a dead env var nobody set) and per-file maybeSkipFuseTests wrappers. On Linux, bazil.org/fuse hardcodes "fusermount" but modern distros only ship "fusermount3". The CI job creates a symlink; the auto-detect gives a helpful skip message when only fusermount3 is found locally. * fix(fuse): handle EINTR on close in IPNS concurrent write test TestConcurrentWrites was flaky because Go's goroutine preemption signal (SIGURG) can interrupt the FUSE FLUSH inside close(), returning EINTR. The write itself already succeeded and the kernel will still send RELEASE to the daemon, so the data is safe. Replace os.WriteFile with explicit open/write/close so we can ignore EINTR on close while still catching real errors. * fix(fuse): resolve bare file CIDs on /ipfs mount Accessing a file by its CID at the /ipfs FUSE mount root returned ENOENT because ProtoNodeConverter cannot handle UnixFS file ADLs. Decode dag-pb blocks directly from bytes instead. Closes #9044 * fix(fuse): fix same-directory rename on /mfs Renaming a file within the same MFS directory left the source behind. The directory's entry cache was re-synced before the old name was removed. Unlink the source before AddChild to match the working IPNS pattern. * test(fuse): add mixed dag-pb/raw directory test Covers the scenario from #9044: a directory with both dag-pb and raw-leaf children read through the /ipfs FUSE mount. * test(fuse): remove redundant testing.Short() checks SkipUnlessFUSE(t) already handles skipping via TEST_FUSE. The testing.Short() guard was a second skip gate that served no purpose since FUSE tests only run under make test_fuse. * fix(fuse): get DAG node before unlinking source in rename Move GetNode() before Unlink() in both mfs and ipns Rename so that a GetNode() failure does not leave the source entry already removed. Also add FUSE test instructions to AGENTS.md. * ci: skip fuse3 install when fusermount exists Self-hosted runners persist state, so after the first run fuse3 and the symlink are already in place. Skip apt-get update and install entirely when fusermount is in PATH. --------- Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com> Co-authored-by: Marcin Rataj <lidel@lidel.org>
Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 5.5.3 to 6.0.0. - [Release notes](https://github.com/codecov/codecov-action/releases) - [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md) - [Commits](codecov/codecov-action@1af5884...57e3a13) --- updated-dependencies: - dependency-name: codecov/codecov-action dependency-version: 6.0.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* fix(fuse): persist IPNS writes across restarts The IPNS FUSE mount's MFS republisher calls Name.Publish to persist changes, but checkPublishAllowed blocks all publishes while the mount is active. This means writes through the FUSE mount are silently dropped and lost on daemon restart. Add a context key so the mount's internal publishes bypass the guard while manual `ipfs name publish` from CLI/RPC remains blocked. - core/coreiface/name.go: context key and helpers for mount publish - core/coreapi: checkPublishAllowed checks context before blocking - fuse/ipns: ipnsPubFunc marks its context as mount-internal - fuse/ipns: tests set node.Mounts.Ipns to exercise the guard Fixes #2168 * docs: add IPNS FUSE persistence fix to v0.41 changelog * refactor(fuse): move publish bypass to internal package Move the FUSE mount publish context key from the public coreiface package to internal/fusemount, preventing external consumers from bypassing the publish guard. - internal/fusemount/context.go: new internal package with context helpers - core/coreiface/name.go: remove exported ContextWithMountPublish / IsMountPublish - core/coreapi/coreapi.go: use fusemount.IsPublish for the guard check - fuse/ipns/ipns_unix.go: use fusemount.ContextWithPublish to tag context
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
19 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.