[Project] GA durability (2 copies; 1 with endorsed SP)

[BigLep]

This is a tracking issue for all the work to fully land our minimum-bar durability story for GA to make sure we don't lose track of any of the details. This tracks the project work across multiple repos. We're sticking this in filecoin-services, because we don't currently have a better over-arching place.

The expectation is that Synapse will store two copies by default:

• One copy to an "endorsed" SP (which is also an "approved" SP)
• One copy to another "approved" SP

Tasks

Contracts

• (Endorsement Providers Contract) Create contract with deployment script: Endorsed Providers Contract #382
• (Endorsement Providers Contract) Deploy contract with multisig to calibration and mainnet

Synapse

• (Endorsement Providers Contract) Use endorsement providers contract: feat: Endorsements Service synapse-sdk#553
• (@rvagg) Design desired API for exposing multiple copies. Happening in https://www.notion.so/filecoindev/Multi-copy-Synapse-Upload-2bddc41950c1803d9fb6c8830c8a7a28
• GA DURABILITY: Multi-copy upload via SP-to-SP pull synapse-sdk#494
• Simplify provider & data set resolution logic & API synapse-sdk#495
• Do a release so that filecoin-pin can use it
• (M4.1) Update synapse-react to support multi sp upload flow synapse-sdk#646

Curio

• feat!(ServiceProviderRegistry): Hex Capabilities filecoin-project/curio#816
• pdpv0: Add SP-to-SP piece fetch endpoint filecoin-project/curio#828
• fix(pdp): register pulled pieces in pdp_piecerefs for service ownership filecoin-project/curio#943
• pdpv0: passive telemetry on SP performance filecoin-project/curio#1093

foc-devnet

• (Endorsement Providers Contract) Add endorsements contract to devnet foc-devnet#41
• (Endorsement Providers Contract) Ability to set an "endorsed SP" foc-devnet#50
• Inter-SP communication setup foc-devnet#39
• (M4.1) Integrate 2-copy test validation into foc-devnet: feat: CI overhaul for nightlies and PRs foc-devnet#90

User Documentation

• Synapse docs cover durability story.
  • For example, pull the relevant bits from feat(storage): multi-copy upload with store->pull->commit flow synapse-sdk#593 (comment)

SP Community

• (M4.1) Endorsement Certificates) Onboard at least X SPs with endorsement certificates

Synapse Demo App

• (M4.1) Do 2 copies by default. (This depends on Update synapse-react to support multi sp upload flow synapse-sdk#646)

filecoin-pin

• Do 2 copies by default in in filecoin-pin CLI: feat: upgrade to synapse-sdk 0.38 and synapse-core 0.2 filecoin-project/filecoin-pin#343
• (M4.2) Do 2 copies in filecoin-pin-webiste: feat: upgrade to filecoin-pin 0.18 with multi-copy upload UI filecoin-project/filecoin-pin-website#154

Dealbot

• (M4.2) Add pull flow testing to validate SP piece retrieval from external sources dealbot#300

Operations

• (Endorsement Certificates) Runbook/process for how to endorse an SP
  • Happening in https://www.notion.so/filecoindev/FOC-Operational-Excellence-2b7dc41950c1802aa432fff8ecb801cc
• [Tracking issue] Sponsored repair #454

Notes

Past Issues and Docs

https://www.notion.so/filecoindev/GTM-Replication-280dc41950c180c78684d3eda77c3164
FilOzone/synapse-sdk#275
#250

Followup Work

• Discussion about further improvements to our initial GA durability story is being explored in Track SP failures across uploads for smarter provider selection synapse-sdk#627

[BigLep]

@rjan90 @rvagg : let me know if this has been covered elsewhere, but I wanted to make sure we have a place that covers all the details for fully landing our "GA durability" story. Fully completing this touches quite a few ares if I understand correctly and wanted to make none of the details are lost. Feel free to close, correct, or add to it.

[BigLep]

Design desired API for exposing multiple copies is happening in https://www.notion.so/filecoindev/Multi-copy-Synapse-Upload-2bddc41950c1803d9fb6c8830c8a7a28

[rvagg]

Added curio and synapse issues above