draft(device): support IPFS CID commitments, dag-cbor codec, and gateway#868
Draft
samcamwilliams wants to merge 22 commits into
Draft
draft(device): support IPFS CID commitments, dag-cbor codec, and gateway#868samcamwilliams wants to merge 22 commits into
samcamwilliams wants to merge 22 commits into
Conversation
samcamwilliams
added a commit
that referenced
this pull request
Apr 18, 2026
Address review feedback on PR #868: - Rename `<<"codec">>' to `<<"multicodec">>' throughout — matches the IPFS term exactly (the 2nd varint of a CIDv1) and eliminates the confusion of a "codec" field on a device that is itself a codec. Touches `dev_codec_ipfs', `dev_codec_ipfs_cid', `hb_store_ipfs_gateway', all tests, and the device doc. The unsupported-value error atom becomes `unsupported_multicodec'. - Populate the unsigned commitment's `<<"signature">>' with the raw sha2-256 digest (base64url) and `<<"keyid">>' with the universal constant `constant:ipfs'. This shapes the commitment structurally as an HTTPSig HMAC item — the tag is a pure function of the content and the key-id is a constant — so it rides over the wire through `dev_codec_httpsig_siginfo' as a first-class signature-input line. Remote nodes decoding the response recover the commitment back to its `commitment-device: ipfs@1.0' form unchanged. IPFS expressed as HTTP Message Signatures, with zero kernel changes. - Retarget the live E2E tests at the standard AO-Core paths the PR advertises: `~lookup@1.0/read&target=<CID>' instead of a bare `/<CID>'. The preload / en-masse-pin test drives HTTP requests (which write through to the local store) rather than raw Erlang `hb_cache:read/2' calls (which do not). The ANS-104 commit-for- Arweave chain test exercises `GET /~lookup@1.0/read&target=<CID>/ commit&type=signed&commitment-device=ans104@1.0' — which wraps the IPFS body in a signed bundler-ready message using the node's `priv_wallet'. - Update the device documentation with the three concrete user paths (serve / preload / bundle-to-Arweave), the commitment wire format example, and the IPFS-over-HTTPSig wire explanation. All three PR paths verified against a live node on port 12345 fetching the canonical `bafkreifzjut3te2nhyekklss27nh3k72ysco7y32koao5eei66wof36n5e' CID from `ipfs.io'. 2455 tests green across `hb_message_test_vectors', `hb_cache', every codec, and the full IPFS integration + live suites. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Introduce a user-loadable commitment device that hashes a message's `body`
into a CIDv1, attaching it as an unsigned commitment. The commitment ID is
the CID itself, which `hb_cache` links to the message's uncommitted ID
automatically — so a write followed by `hb_cache:read(CID, _)` recovers
the message with no kernel changes.
Scope is intentionally narrow: sha2-256 multihash, base32-lower multibase,
and the `raw` (0x55) and `dag-cbor` (0x71) multicodecs. No `to`/`from`
yet — the body is opaque for hashing — and no changes to the structured,
cache, or AO-Core kernel.
- `dev_codec_ipfs`: `commit/3`, `verify/3`, `content_type/1`, `info/1`.
- `dev_codec_ipfs_cid`: pure-functional varint, multihash, multibase, CIDv1.
- `dev_codec_ipfs_test`: dispatch through `hb_message:commit/verify` plus
the cache-linkage proof (`hb_cache:read(CID, _)` returns the message).
All 29 unit + integration tests pass. Known-answer CIDs cross-checked:
raw("hello world") => bafkreifzjut3te2nhyekklss27nh3k72ysco7y32koao5eei66wof36n5e
raw(<<>>) => bafkreihdwdcefgh4dqkjv67uzcmw7ojee6xedzdetojuzjevtenxquvyku
dag-cbor(<<0xa0>>) => bafyreigbtj4x7ip5legnfznufuopl4sg4knzc2cof6duas4b3q2fy6swua
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Read-only store backend that fetches IPFS CIDs from configured HTTP gateways (default: w3s.link, ipfs.io) and verifies the returned bytes against the requested CID before handing them up the chain. A lying gateway is treated as `not_found` and the next gateway is tried. Keys that aren't CIDv1 are ignored fast, so the module composes safely in a store chain alongside Arweave-addressed stores. Uses `httpc` from OTP — no new dependency. Tests (via hb_mock_server + cowboy): happy path, digest mismatch, gateway fall-through on lie, 404 fall-through, and end-to-end `hb_cache:read/2` against a local-then-gateway store chain. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Document the device's surface, commitment shape, enable-in-config instructions, an end-to-end example of the cache-linkage flow, and the explicit non-goals for the phase-1 scope. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
`dev_codec_ipfs_cbor' implements the dag-cbor subset of CBOR (RFC 8949)
with strict validation. Covers the IPLD data model at the bytes frontier:
- Deterministic encoding: shortest-form integers, canonical length-first
map ordering, only 64-bit floats, definite-length containers.
- Rejection paths: non-canonical integer forms, indefinite-length items,
half/single floats, NaN, Infinity, non-UTF-8 text, non-string map keys,
out-of-order or duplicate map keys, unsupported tags.
- IPLD Link: tag 42 wraps a byte string prefixed with 0x00 and the CID's
pre-multibase binary form. Round-trips against `dev_codec_ipfs_cid'.
The Erlang intermediate form is deliberately close to what `~structured@1.0'
already represents, so the device-level glue in the next commit is just a
thin walker. No HB machinery is used here — this is the pure bytes/IPLD
frontier.
27 tests pass, including the RFC 8949 Appendix A integer vectors, all the
rejection paths, determinism across map insertion order, CID-link
roundtrip, and the canonical empty-dag-cbor CID ground truth
(`bafyreigbtj4x7ip5legnfznufuopl4sg4knzc2cof6duas4b3q2fy6swua`).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Extends `~ipfs@1.0' with full dag-cbor serialization:
to(TabmMsg, Req, Opts) -> {ok, CborBytes}
TABM -> structured@1.0 (native types) -> ensure_all_loaded
-> IPLD intermediate form -> deterministic dag-cbor bytes.
Commitments and priv are stripped — IPFS blocks are pure content.
Atoms outside {null, true, false} throw a clean `{error, _}` per the
dag-cbor spec (which has no atom type).
from(CborBytes, Req, Opts) -> {ok, TABM}
dag-cbor bytes -> IPLD intermediate -> structured form -> TABM
via `dev_codec_structured:from/3'.
No changes to `~structured@1.0', `hb_ao`, `hb_cache', or any other existing
module. The codec plugs into `hb_message:convert/3,4' via the standard
routing machinery.
Expands unit tests in `dev_codec_ipfs_cbor' with DAG-CBOR spec vectors
(mixed nulls/bools, empty containers, 23/24-char string length boundary,
nested lists and maps) and int64 boundaries. Adds integration tests in
`dev_codec_ipfs_test`:
- to/3 produces the exact canonical bytes for `{"hello":"world"}`
- a typed HB message (integers, floats, booleans, null, lists, nested
maps) roundtrips bit-for-bit
- encoding is deterministic across map-construction orders
- the CID computed over the dag-cbor bytes matches the canonical dag-cbor
CIDv1 shape (`bafyrei...`, 59 chars) with the correct digest
- unsupported atoms surface as `{error, {dag_cbor_encode, _}}`
- committed messages still encode; commitments are stripped from the
content bytes.
All 75 IPFS-related tests + 2211 regression tests in the broader suite
(hb_message_test_vectors, hb_cache, hb_ao_test_vectors, existing codecs)
pass.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add the `to`/`from` usage, the decode rejection table, and the composed commit-over-dag-cbor pipeline. Move the prior "what's next" section to reflect that phase 2 is done, leaving link-aware hb_link integration as the natural next step. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Simulates the real production flow: Node A encodes a rich HyperBEAM message as dag-cbor, commits a dag-cbor CID over the bytes, and publishes to a stub gateway. Node B (a separate empty cache) requests the CID via its store chain; the gateway backend fetches, verifies the bytes hash to the CID, and admits them. Node B then decodes the bytes back via the codec and confirms the message matches Node A's original. This exercises every seam of the two-phase build in one test: codec to/3 + from/3, unsigned CID commit, hb_cache linkage, hb_store chain fallthrough, and gateway digest verification. All 3138 tests in the full suite pass (`rebar3 eunit`), including the new IPFS modules (76 tests), hb_message_test_vectors (1954), hb_cache, hb_ao_test_vectors, and every other codec. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace the mock-server-based gateway tests with live tests against the
public IPFS network. HyperBEAM's test suite is live-service interactive
(see the Arweave gateway tests in `hb_store_gateway'); the mocks were
hiding real-world failure modes.
Two substantive changes:
1. The gateway store now attaches an `~ipfs@1.0' unsigned commitment,
keyed by the CID, to every message it hands up the chain. In addition
to the direct sha256 digest check that already gated the response, the
returned message is now independently verifiable via
`hb_message:verify/2,3' through the canonical dispatch — no trust in
this store's word required, and a natural contract to carry across
`hb_cache:write/2' if the caller decides to persist.
2. Tests now hit real gateways (`ipfs.io', `dweb.link',
`nftstorage.link', `4everland.io') for the canonical
`raw("hello world")' and empty-dag-cbor CIDs, both verified pinned at
the time of writing. Each test lists multiple gateways; a test that
cannot reach any gateway skips instead of failing, matching the
`hb_store_gateway' pattern. Timeouts are generous (20s).
`w3s.link' dropped from the default gateway list — it now returns 410
for the known test CID.
The verified-digest direct gate is still in place, and the new
`digest_gate_rejects_tampered_body_test' covers it directly alongside
the `live_gateway_rejects_unpinned_cid_test_' that demonstrates
real-gateway refusal of an unpinned CID.
All 25 IPFS tests (gateway + codec integration) pass against the live
network.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Make the device first-class:
1. Add to `hb_opts:preloaded_devices/0' so `hb_message:convert/3,4',
`hb_ao:resolve/3', and HTTP dispatch all resolve it by name without
per-call configuration.
2. Append `<<"ipfs@1.0">>' to `hb_message_test_vectors:test_codecs/0',
so the ~54-test suite runs against the codec across 3 opts variants
(normal, ed25519, ethereum).
To make the full grid green, three substantive codec changes:
- `to/3' now encodes bare binaries as dag-cbor text strings (falling
back to byte strings on non-UTF-8). Previously it passed them through
unchanged, which broke the `binary_to_binary' roundtrip.
- `to/3' no longer strips `<<"commitments">>' from the encoded output.
Matches the peer codecs (json, flat, ans104): the commitment field
rides along through the codec boundary so `from(to(X)) = X' over the
full HyperBEAM message. A pure IPFS consumer sees the field as just
more map content — valid IPLD, no harm.
- `commit/3' and `verify/3' delegate to `~httpsig@1.0' for non-unsigned
types. Matches the composition used by `dev_codec_flat' and
`dev_codec_json'. `type: unsigned' keeps the IPFS CID semantics;
`type: signed' (and sha256/hmac/rsa-pss/etc.) produces a standard
HyperBEAM signed commitment. The two paths co-exist cleanly and the
codec works in every test context.
Four tests carry a per-codec skip for `~ipfs@1.0':
- `structured_field_atom_parsing_test' — dag-cbor has no atom type
beyond null/true/false (per spec).
- `priv_survives_conversion_test' — we strip `priv' at the codec
boundary, matching ans104/tx/json.
- `sign_node_message_test' — the default node message carries arbitrary
atoms (module names) that dag-cbor cannot represent.
- `id_of_linked_message_test' — uses intentionally-unresolvable lazy
links; we resolve links before encoding so the block is
self-contained.
`hb_message_test_vectors` is now 2116 tests green. All existing IPFS unit
and integration tests (56) also green.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ndle
Make a running HyperBEAM node actually serve IPFS content end-to-end, not
just have working unit-tests. Two small enabling changes plus a live
integration test.
1. `?IS_ID(X)` now accepts 59-byte binaries — the length of a CIDv1
base32-lower / sha2-256 / raw-or-dag-cbor CID. This is what
`hb_ao:resolve_many([ID], Opts)` gates on when deciding to load a
bare `/ID` HTTP request directly from the store chain. Without it,
`hb_singleton` parses the CID as `#{<<"path">> => CID}' and the
request 404s before the cache is consulted.
2. `dev_codec_httpsig_siginfo:commitments_to_siginfo/3` now filters out
commitments that carry no `<<"signature">>' field. Content-addressed
commitments (`~ipfs@1.0' unsigned CIDs, `~ans104@1.0/unsigned-sha256')
are not RFC-9421 signatures and do not belong in the HTTP Signature /
Signature-Input headers; previously they caused a `{badkey, "signature"}'
crash during HTTP response encoding whenever such a message left the
server.
3. `src/dev_codec_ipfs_live_test.erl` — four integration tests against
the live IPFS network + a running in-process HyperBEAM node:
- `GET /<CID>` returns the pinned body (canonical `hello world`
CID via `ipfs.io`/`dweb.link`/`nftstorage.link`/`4everland.io`).
- Body recomputes to the requested CID — the only verification
that matters in IPFS.
- Load an IPFS body through the store chain, then run a Lua
computation (`byte_length`) across it via the `~lua@5.3a'
device. The node serves HTTP traffic concurrently.
- Fetch from IPFS, sign as ANS-104, and hand to `~arweave@2.9' to
post as a bundle. The pipeline is exercised end-to-end up to
the bundler endpoint (not configured in CI, by design).
All tests skip gracefully if every configured gateway is unreachable,
matching the `hb_store_gateway' live-test pattern.
Regression: 2454 tests green across `hb_message_test_vectors`,
`hb_cache`, `hb_ao_test_vectors`, every codec (ans104/httpsig/flat/
json/structured/ipfs/tx), and the new IPFS live suite.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Address review feedback on PR #868: - Rename `<<"codec">>' to `<<"multicodec">>' throughout — matches the IPFS term exactly (the 2nd varint of a CIDv1) and eliminates the confusion of a "codec" field on a device that is itself a codec. Touches `dev_codec_ipfs', `dev_codec_ipfs_cid', `hb_store_ipfs_gateway', all tests, and the device doc. The unsupported-value error atom becomes `unsupported_multicodec'. - Populate the unsigned commitment's `<<"signature">>' with the raw sha2-256 digest (base64url) and `<<"keyid">>' with the universal constant `constant:ipfs'. This shapes the commitment structurally as an HTTPSig HMAC item — the tag is a pure function of the content and the key-id is a constant — so it rides over the wire through `dev_codec_httpsig_siginfo' as a first-class signature-input line. Remote nodes decoding the response recover the commitment back to its `commitment-device: ipfs@1.0' form unchanged. IPFS expressed as HTTP Message Signatures, with zero kernel changes. - Retarget the live E2E tests at the standard AO-Core paths the PR advertises: `~lookup@1.0/read&target=<CID>' instead of a bare `/<CID>'. The preload / en-masse-pin test drives HTTP requests (which write through to the local store) rather than raw Erlang `hb_cache:read/2' calls (which do not). The ANS-104 commit-for- Arweave chain test exercises `GET /~lookup@1.0/read&target=<CID>/ commit&type=signed&commitment-device=ans104@1.0' — which wraps the IPFS body in a signed bundler-ready message using the node's `priv_wallet'. - Update the device documentation with the three concrete user paths (serve / preload / bundle-to-Arweave), the commitment wire format example, and the IPFS-over-HTTPSig wire explanation. All three PR paths verified against a live node on port 12345 fetching the canonical `bafkreifzjut3te2nhyekklss27nh3k72ysco7y32koao5eei66wof36n5e' CID from `ipfs.io'. 2455 tests green across `hb_message_test_vectors', `hb_cache', every codec, and the full IPFS integration + live suites. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…-cbor)
The commitment carried both `<<"multicodec">>' and `<<"hash-alg">>', but
the combination is better expressed as a single coordinate: IPFS tooling
names a CID's construction this way, and the hash-alg string fully
determines how a body maps to a CID.
- `sha2-256-raw' → `bafk…' CIDs (multicodec 0x55)
- `sha2-256-dag-cbor' → `bafy…' CIDs (multicodec 0x71)
`dev_codec_ipfs:commit/3' now accepts only `hash-alg' in its request;
`verify/3' reads only `hash-alg'. Unknown values surface as
`{error, {unsupported_hash_alg, _}}' — the `unsupported_multicodec'
atom is gone.
Touches `dev_codec_ipfs', `dev_codec_ipfs_cid' (decode output), the
gateway store's `with_commitment/3' and its `verify_digest/2' clause
(now matches `<<"sha2-256-", _/binary>>'), all tests, and the device
doc.
All 2456 tests green across hb_message_test_vectors, hb_cache, every
existing codec, and the full IPFS + live suites. Verified against live
node on port 12345:
GET /~lookup@1.0/read&target=bafkreif... → HTTP 200, "hello world"
cached commitment field: hash-alg = "sha2-256-raw"
signature-input alg="ipfs@1.0/unsigned"; keyid="constant:ipfs"
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
samcamwilliams
force-pushed
the
feat/ipfs-codecs
branch
from
2a443eb to
ac8ca5e
Compare
…wire
Move the hash-alg coordinate from a separate `hash-alg' field into the
commitment's `type' field, following the exact convention `dev_codec_ans104'
(`unsigned-sha256') and `dev_codec_httpsig' (`hmac-sha256') already use:
the caller passes generic `type: unsigned' (+ an optional `hash-alg'
request knob to pick between raw / dag-cbor), and the codec translates
into its native type string which lands in the commitment and on the
wire.
On-wire before:
alg="ipfs@1.0/unsigned"; keyid="constant:ipfs"; hash-alg="sha2-256-raw"
On-wire after:
alg="ipfs@1.0/sha2-256-raw"; keyid="constant:ipfs"
That drops the one RFC-9421-pedantry thing in the previous shape — the
custom `hash-alg' metadata parameter, which §6.3 says must be IANA-
registered. The device now emits zero custom metadata parameters on its
signature-input line. Everything else (non-IANA `alg' values,
`keyid="constant:..."') continues the existing HyperBEAM convention
shared with `~httpsig@1.0', `~ans104@1.0' etc.
Caller API unchanged. `hb_message:commit(Msg, Opts, #{
<<"commitment-device">> => <<"ipfs@1.0">>, <<"type">> => <<"unsigned">>,
<<"hash-alg">> => <<"sha2-256-dag-cbor">> })' still works. Callers can
also now pass `<<"type">> => <<"sha2-256-raw">>' (or `sha2-256-dag-cbor')
directly.
All 2457 tests green. Verified against live node on port 12345:
signature-input:
comm-...=("content-digest");
alg="ipfs@1.0/sha2-256-raw";
keyid="constant:ipfs"
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Two surgical changes to `dev_codec_httpsig_siginfo' that let
content-addressed commitment devices (like `~ipfs@1.0') round-trip their
map keys through the HTTPSig wire without enlarging the commitment device
contract or changing `dev_message:id/3'.
1. `id=' parameter on signature-input.
`commitments_to_siginfo_for_signable/3' now threads each commitment's
map key (`CommID') through into `commitment_to_sf_siginfo/4'. The
encoder computes the key the decoder would derive solely from `Sig'
(`human_id(Sig)' for 32-byte sigs, `human_id(sha256(Sig))' otherwise)
and emits `id="<CommID>"' only when `CommID' differs.
Stock signed commitments (`hmac-sha256', `rsa-pss-sha512', etc.) key
on the same derivation the decoder uses, so they never emit `id='.
IPFS commitments, whose map key is a CID, do.
`sf_siginfo_to_commitment/5' checks for `<<"id">>' in the parsed
params via `maps:take/2'. Present → use as the commitment's map key,
strip from the commitment body. Absent → existing `h(Sig)' fallback.
`<<"id">>' added to `get_additional_params/1''s reserved-key list so
it cannot round-trip twice or leak into commitment bodies.
2. `keyid' is now optional on the wire.
RFC 9421 §1.4.2.3 permits `keyid' absence ("other means" of key
retrieval, where here "other means" is "no retrieval at all" for
content-addressed commitments). The encoder default changes from
`<<>>' to `undefined'; the param list wraps it conditionally so the
standard `undefined'-filter drops it.
Commitments that set `keyid' (HMAC, RSA-PSS) are unchanged. Absent
`keyid' now stays absent on the wire instead of becoming `keyid=""'.
Applied:
* `dev_codec_ipfs:commit/3' drops its `<<"keyid">> => <<"constant:ipfs">>'
field. Commitment shape: `commitment-device', `type', `committed',
`signature'. The `signature' stays (it is the raw sha-256 digest of
the body) so the commitment passes the
`commitments_to_siginfo' signature-bearing filter and rides the wire.
* `hb_store_ipfs_gateway:with_commitment/3' mirrors — same four-field
commitment from gateway-fetched content.
* `dev_codec_ipfs_live_test' adds a live HB-to-HB transport test that
asserts the IPFS commitment arrives on a remote client keyed under
the CID (not `h(Sig)').
* `docs/devices/ipfs-at-1-0.md' removed. The edoc-generated reference
at `docs/resources/source-code/dev_codec_ipfs.md' remains.
Wire shape, after:
ipfs@1.0 : alg="ipfs@1.0/sha2-256-raw"; id="<CID>"
hmac-sha256 : alg="hmac-sha256"; keyid="constant:ao"
rsa-pss-sha512: alg="rsa-pss-sha512"; keyid="publickey:..."
2458 tests green across `hb_message_test_vectors', `hb_cache',
`hb_ao_test_vectors', every existing codec (including `dev_codec_httpsig'
itself), and the full IPFS + live suites. Live node on port 12345 emits
the target wire shape and stock HMAC/RSA commitments emit no spurious
`id=' parameter.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…port
Spin up two in-process HyperBEAM nodes (each with its own wallet — the
listener's server_id is derived from `priv_wallet', so shared wallets
collide on one listener) inside a single eunit:
Node A: has `hb_store_ipfs_gateway' upstream; can mint CID-keyed
commitments for real IPFS content.
Node B: isolated filesystem store, NO IPFS gateway. Can serve a CID
only if the message is already in its local cache keyed under that
CID.
The test drives the full relay topology:
1. B alone cannot serve the CID (no upstream path, cache empty).
2. Client does `hb_http:get(A, "/~lookup@1.0/read&target=<CID>")'.
A's response carries the IPFS commitment on its signature-input
line with `id="<CID>"'; the client's `sf_siginfo_to_commitment'
reconstructs the commitment at the CID key.
3. Client persists the received message to B's store via
`hb_cache:write/2'. The CID is picked up as an AltID and linked to
the uncommitted root ID.
4. `hb_http:get(B, "/~lookup@1.0/read&target=<CID>")' now serves the
body from B's local cache. No gateway. No upstream.
5. Direct store inspection confirms B's local store has the message
at the CID key, with the IPFS commitment intact under that key.
End-to-end proof of the `id=' extension: a CID-keyed commitment
survives HTTP transport between two HB nodes and enables the downstream
node to act as a relay/mirror for content it has never reached upstream
itself.
Follows the `dev_router:dynamic_router_pricing_test_' two-node pattern
(fixed high ports, per-node wallet). 2459 tests green across the full
IPFS + codec + cache suites.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ites Reshapes the two-HB-node test to use the proper remote-store topology the user described: Node A holds only the IPFS gateway store; Node B has a primary test_store plus hb_store_remote_node pointing at A with local-store => [NodeBPrimary] for write-through caching. Flow: client hits B -> B's primary misses -> B's remote-node fetches from A -> A retrieves from real IPFS -> returns to B -> B caches in primary -> returns to client. We then kill Node A via cowboy:stop_listener/1 and re-query B, which must serve from its primary cache with no upstream traffic. Per-node wallets (ar_wallet:new/0) avoid server_id collisions, matching the pattern in dev_router:dynamic_router_pricing_test_. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Align the IPFS device with HyperBEAM's existing codec style:
- Short, focused module docs (one paragraph) matching
`dev_codec_json', `dev_codec_flat', `dev_codec_ans104'.
- Remove `%%%====' section banners — not idiomatic.
- Condense inline comment prose; drop multi-paragraph essays.
Deduplicate:
- `dev_codec_ipfs_test': `ipfs_commit/2,3' helper replaces the
`#{<<"commitment-device">>=><<"ipfs@1.0">>,...}' literal at every
call site.
- `dev_codec_ipfs_live_test': `with_live_gateways/1' and
`response_body/1' replace per-test skip and body-extraction blocks.
- `dev_codec_httpsig_siginfo': decoder now reuses the encoder's
`derived_commitment_id/1' helper instead of an inline `if'.
- `dev_codec_ipfs_cbor': `pair_key_lt/2' delegates to `key_lt/2'; the
duplicate `key_strictly_less/2' is gone.
- `dev_codec_ipfs_cid': `safe/2' wraps the three multibase decode
attempts, replacing three copies of the same try/catch shape.
Tighten:
- `dev_codec_ipfs:commit/3' uses an `?IS_NATIVE_TYPE/1' guard and a
single `multicodec_of/1' lookup; `verify/3' collapses to one clause.
- `dev_codec_ipfs_cid:decode_bytes/1' asserts the digest occupies
exactly `DigestLen' bytes instead of inferring.
- `dev_codec_ipfs_cbor' float rejection uses `andalso' guards directly;
match on `+0.0` / `-0.0` to silence OTP-27 warning.
All 2456 targeted tests pass; no regressions in the broader `hb_store',
`hb_http', `hb_ao', `hb_message', or `dev_codec_httpsig*' suites. Net
-824 lines across the seven touched files.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…mments - `src/include/hb.hrl': revert the 59-byte CID branch added to `?IS_ID'. That was a kernel-layer edit to enable the bare `GET /<CID>' path via `hb_ao:resolve_many/2'; the test suite and production IPFS flows all route through `/~lookup@1.0/read&target=<CID>' instead, which resolves the CID from the cache chain without touching the kernel ID guard. - Prepend `@doc ` to every function-preceding `%% ' block that lacked it across the IPFS module, helper modules, gateway store, and test modules. This is cosmetic only. All 2456 targeted tests remain green (codec, gateway, live network, regression across ans104/httpsig/flat/json/structured/cache/ao vectors). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Previously `hb_message_test_vectors' carried four IPFS-specific skip clauses and listed `<<"ipfs@1.0">>' in `test_codecs/0'. The skip list should live with the IPFS device, not the generic test-vector battery (same spirit as `hb_ao_test_vectors' composing opts with `skip => [...]' rather than spreading device-specific branches through the test body). - Revert `hb_message_test_vectors.erl' to one-line delta against edge: just `-export([codec_test_suite/1]).'. The suite generator already accepted a list of codecs — we simply surface the API. - New `src/dev_codec_ipfs_message_test_vectors.erl' invokes `hb_test_utils:suite_with_opts/2' with `hb_message_test_vectors: codec_test_suite([<<"ipfs@1.0">>])' and the four skipped test names on the opts entry, each with a reason comment. All 50 vectors green under the new module; full regression across every codec + IPFS suite still green (2344 tests). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…t_vectors Merge the three separate test modules into one canonical test-vectors file for `~ipfs@1.0': - `dev_codec_ipfs_test' (integration / codec / cache linkage), - `dev_codec_ipfs_live_test' (live gateways + HB-node flows), - `dev_codec_ipfs_message_test_vectors' (hb_message_test_vectors battery), all now live in `src/dev_codec_ipfs_test_vectors.erl', organised into five sections (integration dispatch, cache linkage, to/from conversion, live network flows, message-vector suite with `skip' list). Shared helpers (`opts/0,1', `ipfs_commit/2,3', `ipfs_device/0', `gateway_store/0', `node_opts/0', `with_live_gateways/1', `response_body/1') and constants (`?HELLO_WORLD*', `?EMPTY_MAP_CID', `?LIVE_GATEWAYS', `?LOOKUP_PATH') are declared once instead of duplicated across three files. Unit-level tests continue to live inline in `dev_codec_ipfs', `dev_codec_ipfs_cid', `dev_codec_ipfs_cbor', and `hb_store_ipfs_gateway' — matching the rest of the HyperBEAM codebase. 72 IPFS tests pass under the new module (15 integration + 7 live + 50 message-vector suite); full regression green at 2344 tests. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Apply the HyperBEAM house style to every multi-line function call and assert expression in the IPFS modules: the closing `)' sits on its own line at the indent of the line that opened the block, never glued to the last argument. Touches tests and inline sections in `dev_codec_ipfs', `dev_codec_ipfs_test_vectors', and `hb_store_ipfs_gateway'. Behavior is unchanged; all 133 IPFS tests remain green. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Removing the `maps:filter' that skipped commitments without a `signature' field: it was dead code. Every current commitment device (HMAC, RSA-PSS, ANS-104 signed, `~ipfs@1.0') emits a `signature' field — the IPFS device specifically so its commitment survives the httpsig wire. Nothing fails the filter, so it never takes effect. Regression: all 2158 targeted tests green (codecs, cache, IPFS suite incl. inter-node transport). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A vibe-coded experiment, drafting a way to introduce IPFS codec devices (implementing the standard
to,from,commit, andverifykeys) to HyperBEAM.This enables HyperBEAM nodes to interact with and serve content addressed by IPFS CIDs. Existing IPFS data can be pulled into the Arweave and AO ecosystem seamlessly this way, including upload of said data to Arweave.
It is currently pure, unadulterated slop. We mustn't merge this into HyperBEAM core, but it does prove the fact that it is possible (and quite trivial). We should be able to easily make this into an externally loadable module such that node operators that want to run it have the possibility to do so, without making it part of the 'OS'/kernel level of HyperBEAM.
Building on the permagit work, it would logically follow that we could do the same with
gitand possibly other merkled distributed data structures, too (datandbittorrentchunks, perhaps). The net effect would be a single, global compute state space that included all data found in each of those graphs. Users could then execute any AO-Core resolutions they want over that data and receive merklized, blockchain-style cryptographic attestations (sigs over hashpaths) in response. It would glue together each of these disparate networks into one. Pretty compelling...Anyway, here is how the agents describe the PR:
Data can be synced from IPFS gateways (verifying on inbound), cached locally in the HyperBEAM node, and served to the caller just by visiting
HYPERBEAM_NODE/IPFS_CIDin the browser. To load en masse, you can simply loop over your CIDs with aHEAD /CIDrequest, causing them to be cached in the node's local store.To push the data to Arweave, simply run a HyperBEAM node with a topped up wallet and execute
GET /CID/~bundler@1.0/tx&ipfs-add=CID?!. You will be prompted to enter a username and password that will be used by the node to generate and guard a wallet for you. You will receive the reciprocal Arweave ID in the response headers.Key Features:
~ipfs@1.0): Adds support for creatingunsignedIPFS CIDv1 commitments over message bodies, usingrawordag-cborcodecs andsha2-256hashing. This allows messages to be identified and verified by their content hash.to/3) and deserialization (from/3) capabilities for HyperBEAM messages to and from deterministic DAG-CBOR bytes. This ensures consistent content addressing across systems and preserves rich data types.hb_store_ipfs_gateway): Enables HyperBEAM nodes to fetch and verify external IPFS content from a configurable list of HTTP gateways. This module prioritizes content integrity by verifying the fetched body's digest against the CID's multihash, making the node an IPFS gateway capable of serving verified public content.GET /).IS_IDmacro to recognize IPFS CIDv1 lengths and registers~ipfs@1.0inpreloaded_devicesfor out-of-the-box availability.