fix: pieces can be removed from a data-set #253

SgtPooki · 2025-11-19T18:26:11Z

Add rm command to CLI. This command will remove a piece from a data set.

Usage

filecoin-pin rm --piece <cid> --data-set <id> --wait-for-confirmation

Options

--piece <cid>: The CID of the piece to remove.
--data-set <id>: The ID of the data set to remove the piece from.
--wait-for-confirmation: Wait for transaction confirmation before exiting.

~~still in draft because data-set show and rm are conflicting somewhere. See https://filecoinproject.slack.com/archives/C095WFA0QK1/p1763576428763319~~

Copilot

Pull Request Overview

This PR adds a new rm command to the CLI that enables users to remove pieces from data sets via the Synapse SDK. The implementation follows established patterns from similar commands (add, import) and provides both CLI and library usage patterns through a flexible API design.

Key changes:

New rm command with options for piece CID, data set ID, and optional transaction confirmation
Core removePiece function supporting both CLI (with dataSetId) and library (with StorageContext) usage patterns
Progress event system for tracking transaction submission and confirmation
Package export for the new core/piece module

Reviewed Changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
`src/commands/rm.ts`	Defines the CLI command interface with piece, data-set, and wait-for-confirmation options
`src/rm/types.ts`	TypeScript interfaces for rm command options and results
`src/rm/remove-piece.ts`	CLI orchestration layer with input validation, Synapse initialization, and progress tracking
`src/rm/index.ts`	Export file for rm module
`src/core/piece/remove-piece.ts`	Reusable core function for removing pieces with flexible API supporting both CLI and library usage
`src/core/piece/index.ts`	Export file for core piece module
`src/cli.ts`	Registers the new rm command with the main CLI program
`package.json`	Adds package export entry for the new core/piece module

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

SgtPooki · 2025-11-24T16:21:03Z

Need to look into why we're still calling PDP server for datasets and piece info.. we should use warmstorage for source of truth. discussed in slack: https://filecoinproject.slack.com/archives/C095WFA0QK1/p1763984382967049?thread_ts=1763576428.763319&cid=C095WFA0QK1

Copilot

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/commands/rm.ts

src/core/data-set/get-data-set-pieces.ts

src/commands/rm.ts

BigLep · 2025-11-25T19:34:38Z

@SgtPooki : I assume you'll request review from Rod once tests are passing?

SgtPooki · 2025-11-25T19:39:41Z

there are certain circumstances (should not be possible, but we should show to users) where piece could be known on chain, but not on SP, or vice versa.. some changes have been added to display this. See slack thread for more details

status types

/**
 * Status of the piece, e.g. "pending removal", "active", "orphaned"
 *
 * - "pending-removal": the piece is scheduled for deletion, but still showing on chain
 * - "active": the piece is active, onchain and known by the provider
 * - "onchain-orphaned": the piece is not known by the provider, but still on chain
 * - "offchain-orphaned": the piece is known by the provider, but not on chain
 *
 * The orphaned states should not happen, but have been observed and should be logged and displayed to the user.
 */
export enum PieceStatus {
  ACTIVE = 'ACTIVE',
  PENDING_REMOVAL = 'PENDING_REMOVAL',
  ONCHAIN_ORPHANED = 'ONCHAIN_ORPHANED',
  OFFCHAIN_ORPHANED = 'OFFCHAIN_ORPHANED',
}

output

┌  Filecoin Onchain Cloud Data Set Details for #48
│
◇  ━━━ Data Set ━━━
│
│  Network: calibration
│  Client address: 0x2B3d0b827BAded638279C637d7ef21a266994c9E
│
│  #48
│    Status: live
│    CDN add-on: disabled
│
│    Provider
│      ID: 2
│      Address: 0xbCdf1bdc1a97D071a5a8EF03F1F05225b6E2a1Ba
│      Name: ezpdpz-calib2
│      Description: PDP Calibration node based in the UK
│      Service URL: https://calib2.ezpdpz.net
│      Active: yes
│      Location: C=GB;ST=Gloucestershire;L=Cheltenham
│
│    Metadata
│      source: "filecoin-pin"
│      withIPFSIndexing: ""
│
│    Payment
│      PDP rail ID: 89
│      Payer: 0x2B3d0b827BAded638279C637d7ef21a266994c9E
│      Payee: 0xbCdf1bdc1a97D071a5a8EF03F1F05225b6E2a1Ba
│
│    Pieces
│      Total pieces: 9
│      Total size: 900.0 B
│      Unique PieceCIDs: 6
│      Unique IPFS Root CIDs: 6
│
│      #1 (active)
│        PieceCID: bafkzcibcdub2fsyawhoxetnwlwxajaqovmapo6b2tgkuw2uwhnjwdqfo2zt6wmq
│        Size: 225.0 B
│        Metadata
│          ipfsRootCID: "bafybeidifi6iaamwjrohdnsnumattjcraz6rm46mbfzpphkbtgnse2wd54"
│
│      #2 (active)
│        PieceCID: bafkzcibcdub3mf2pi3y523askafb774xgzz5rwma2qgjpi2tgacxevdozo2huja
│        Size: 225.0 B
│        Metadata
│          ipfsRootCID: "bafybeids6gixmju6uwwji332pa7mg2hej2ecojoqppa5let6whxvuw474u"
│
│      #3 (active)
│        PieceCID: bafkzcibcdubu56fv7yais7htphs3j6q6n53sl3osyrwsc45a6zuvv2amdj5qopi
│        Size: 225.0 B
│        Metadata
│          ipfsRootCID: "bafybeieltmtc5xuye5h4jfhjpyujrcloyll5fjdsuop4ulvx6raekxl6zy"
│
│      #4 (offchain orphaned)
│        PieceCID: bafkzcibcdubzlvmhmqgjksynuephe32tz7letfuct6o6ielczcsjkifrpfllobq
│        Size: unknown
│        Metadata
│          ipfsRootCID: "bafybeig5utgrmrfqsvvhq42zopcxcf7cytcm4mpck6suaovlhzjt2hyxxu"
│
│      #5 (onchain orphaned)
│        PieceCID: bafkzcibcdubxifgdf7nmryydlkqbmo5q7a5r7wirrousepemrrkx55hb644awby
│        Size: 225.0 B
│        Metadata
│          ipfsRootCID: "bafybeidjstuje7huztr4mjwbqslwqkt4q4clypi76ijzzkwxvp5ngxzxra"
│
│      #6 (offchain orphaned)
│        PieceCID: bafkzcibcdubuavvokuyremde4qn5lh24rkscmx6onybphbhoeq6ijhwc2fdbcly
│        Size: unknown
│        Metadata
│          ipfsRootCID: "bafybeig577ua2iyo2cyhciu4vcgotxbj3cxsjnmbhbkgzak24ruvtchbxq"
│
│
└  Data set inspection complete

SgtPooki · 2025-11-25T20:38:58Z

@SgtPooki : I assume you'll request review from Rod once tests are passing?

@BigLep so, the tests are failing here because this depends on synapse-sdk PR: FilOzone/synapse-sdk#464

I can move to draft if we want to wait on that

SgtPooki · 2025-12-02T14:30:06Z

FilOzone/synapse-sdk#464 was just merged, so i'll update this PR when a new version of synapse-sdk is released

Copilot

Pull request overview

Copilot reviewed 14 out of 15 changed files in this pull request and generated 8 comments.

Files not reviewed (1)

upload-action/package-lock.json: Language not supported

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/core/data-set/get-data-set-pieces.ts

src/core/piece/remove-piece.ts

src/rm/remove-piece.ts

src/core/piece/remove-piece.ts

src/core/data-set/get-data-set-pieces.ts

SgtPooki

self review

src/core/piece/remove-piece.ts

src/core/data-set/get-detailed-data-set.ts

src/rm/remove-piece.ts

SgtPooki · 2025-12-04T14:56:35Z

src/core/piece/remove-piece.ts

+ * @returns Transaction hash of the removal operation
+ * @throws Error if storage context is not bound to a Data Set
+ */
+export async function removePiece(pieceCid: string, options: RemovePieceOptions): Promise<`0x${string}` | string> {


do we want removePieceWithDataSetId and removePieceWithStorageContext helper functions or other better names?

nah, in fact, you only seem to be using the dataSetId variant of this so maybe it doesn't need the context form of it? or you could do the opposite and require a context?

rvagg · 2025-12-05T03:17:04Z

src/core/data-set/get-data-set-pieces.ts

+  try {
+    const warmStorage = await WarmStorageService.create(synapse.getProvider(), synapse.getWarmStorageAddress())
+    const pdpVerifier = new PDPVerifier(synapse.getProvider(), warmStorage.getPDPVerifierAddress())
+    scheduledRemovals = await pdpVerifier.getScheduledRemovals(storageContext.dataSetId)


in this instance, maybe we should look at adding getScheduledRemovals to StorageContext, this seems like it's going to be a common-enough operation for anyone who wants to do anything similar to this

I should have done that in the synapse-sdk PR.. 🤦

rvagg · 2025-12-05T03:20:26Z

src/core/data-set/get-data-set-pieces.ts

+        pieceCid: pieceCid.toString(),
+        status,
+      }
+      if (status === PieceStatus.ONCHAIN_ORPHANED) {


ooof, how often are we seeing these instances? I guess we're dealing with filecoin-project/curio#815 still, maybe eventually we can trust the two to be in sync but this is probably a good sanity check to do here.

I think, in the spirit of providing an example of usage, you should document what's going on here at the top of this block, or pull it out into a separate function that does the reconciliation / checking so it's much more clear what's going on and why.

tbh, its not common, but there is a thread in slack where Kubuxu was looking into it, and I believe he opened filecoin-project/curio#825 to help with further investigation.

ACK on pulling the code out and documenting more about what's happening.

rvagg · 2025-12-05T03:26:19Z

src/rm/remove-piece.ts

+    })
+
+    // Display results
+    log.spinnerSection('Results', [


oh, I missed spinnerSection, the cli-logger is getting pretty fancy 👌

🙇 I added that when I needed to get consistent spacing with the payments status output!

fix: pieces can be removed from a data-set

93d32ae

SgtPooki linked an issue Nov 19, 2025 that may be closed by this pull request

Add rm --piece $pieceCid command #8

Open

FilOzzy added team/filecoin-pin "Filecoin Pin" project is a stakeholder for this work. team/fs-wg FOC working group is a stakeholder for this work, and thus wants to track it on their project board. labels Nov 19, 2025

FilOzzy added this to FS Nov 19, 2025

github-project-automation bot moved this to 📌 Triage in FS Nov 19, 2025

SgtPooki mentioned this pull request Nov 19, 2025

cleanup: cli command patterns #254

Open

6 tasks

SgtPooki requested a review from Copilot November 19, 2025 18:31

Copilot started reviewing on behalf of SgtPooki November 19, 2025 18:31 View session

Copilot finished reviewing on behalf of SgtPooki November 19, 2025 18:35

Copilot AI reviewed Nov 19, 2025

View reviewed changes

SgtPooki mentioned this pull request Nov 19, 2025

Expose pending piece state in PDP API filecoin-project/curio#808

Closed

10 tasks

Merge branch 'master' into 8-add-rm---piece-piececid-command

9abd9ea

SgtPooki mentioned this pull request Nov 24, 2025

fix: add getScheduledRemovals method FilOzone/synapse-sdk#464

Merged

2 tasks

SgtPooki self-assigned this Nov 25, 2025

SgtPooki added 2 commits November 25, 2025 11:26

fix: handle pieces pending removal

09a2f91

fix: display piece onchain vs pdpserver status

e7e2fc4

SgtPooki marked this pull request as ready for review November 25, 2025 17:29

SgtPooki requested a review from Copilot November 25, 2025 17:29

Copilot started reviewing on behalf of SgtPooki November 25, 2025 17:30 View session

Copilot finished reviewing on behalf of SgtPooki November 25, 2025 17:34

Copilot AI reviewed Nov 25, 2025

View reviewed changes

src/commands/rm.ts Outdated Show resolved Hide resolved

src/core/data-set/get-data-set-pieces.ts Outdated Show resolved Hide resolved

src/commands/rm.ts Show resolved Hide resolved

fix: rm cmd requires --piece and --data-set

2c4b84f

BigLep moved this from 📌 Triage to ⌨️ In Progress in FS Nov 25, 2025

fix: fix default to orphaned piece status

ffd4d01

entuziaz mentioned this pull request Nov 27, 2025

refactor: align add command with clean CLI separation pattern #273

Merged

SgtPooki added 4 commits December 4, 2025 08:46

Merge branch 'master' into 8-add-rm---piece-piececid-command

d139558

deps: upgrade to latest synapse

9069e21

fix: use new createStorageContext signature in rm

28c0883

test: fix mock merge issue

2d9fc7f

SgtPooki requested a review from Copilot December 4, 2025 13:53

Copilot started reviewing on behalf of SgtPooki December 4, 2025 13:53 View session

Copilot finished reviewing on behalf of SgtPooki December 4, 2025 13:57

Copilot AI reviewed Dec 4, 2025

View reviewed changes

SgtPooki added 4 commits December 4, 2025 09:06

chore: remove unnecessary as string casting

01f6e6c

fix: logger is no longer required for createStorageContext

b60eae6

test: remove-piece is tested

6fcb9d6

test: assert runRmPiece happy path

362cca6

SgtPooki commented Dec 4, 2025

View reviewed changes

SgtPooki added 6 commits December 4, 2025 09:58

chore: remove outdated comment

6e372f6

Update src/core/data-set/get-detailed-data-set.ts

e77693f

Update src/rm/remove-piece.ts

414aec3

Update src/rm/remove-piece.ts

2198e51

Update src/rm/remove-piece.ts

a66dd7a

chore: update jsdoc comments in runRmPiece

797def7

SgtPooki requested a review from rvagg December 4, 2025 15:05

rvagg reviewed Dec 5, 2025

View reviewed changes

fix: pieces can be removed from a data-set #253

Are you sure you want to change the base?

fix: pieces can be removed from a data-set #253

Uh oh!

Conversation

SgtPooki commented Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Usage

Options

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

SgtPooki commented Nov 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

BigLep commented Nov 25, 2025

Uh oh!

SgtPooki commented Nov 25, 2025

status types

output

Uh oh!

SgtPooki commented Nov 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SgtPooki commented Dec 2, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SgtPooki left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

SgtPooki commented Nov 19, 2025 •

edited

Loading

SgtPooki commented Nov 24, 2025 •

edited

Loading

SgtPooki commented Nov 25, 2025 •

edited

Loading