CLI Commands for State Management (#573)

[anjor]

Summary

• Adds comprehensive CLI commands for state management: singularity state list/get/stats/repair
• Implements CSV/JSON export functionality with automatic timestamps
• Provides manual recovery/repair operations with dry-run mode and audit logging
• Supports advanced filtering by deal ID, state, provider, client, and time range

Changes

• New CLI Commands: Complete state management suite under singularity state
• Export System: Pluggable CSV/JSON export with custom output paths
• Recovery Operations: Manual state repair with safety features
• Comprehensive Testing: Unit tests and integration validation
• Documentation: Complete user guide with examples and use cases

Test Plan

• All new CLI commands build and execute successfully
• Export functionality tested with both CSV and JSON formats
• Recovery operations validated with dry-run mode
• Integration with existing state tracking services verified

Resolves #573

[lanzafame]

@anjor this a draft pr... Can we get this to a level worth reviewing please? Documentation written and in the correct place, and unit and integration tests written. Also make sure remove any ai slop.

[Sankara-Jefferson]

@anjor — I'll review it first thing in the morning. I got sidetracked with some UI deployment work.

[Sankara-Jefferson]

Can we consider moving the state command (and its subcommands) under the existing deal command group?

singularity deal state list
singularity deal state get <deal-id>
singularity deal state repair ...

[anjor]

@Sankara-Jefferson is this what you had in mind?

[Sankara-Jefferson]

I can no longer see it:

jeffersonsankara@Jeffersons-MacBook-Pro singularity % ./singularity -h     
2025-07-31T12:41:03.700-0700	INFO	auto-deal	dataprep/autodeal.go:44	Auto-deal service initialized
NAME:
   singularity - A tool for large-scale clients with PB-scale data onboarding to Filecoin network

USAGE:
   singularity [global options] command [command options]

DESCRIPTION:
   Database Backend Support:
     Singularity supports multiple database backend: sqlite3, postgres, mysql5.7+
     Use '--database-connection-string' or $DATABASE_CONNECTION_STRING to specify the database connection string.
       Example for postgres  - postgres://user:pass@example.com:5432/dbname
       Example for mysql     - mysql://user:pass@tcp(localhost:3306)/dbname?parseTime=true
       Example for sqlite3   - sqlite:/absolute/path/to/database.db
                   or        - sqlite:relative/path/to/database.db

   Network Support:
     Default settings in Singularity are for Mainnet. You may set below environment values for other network:
       For Calibration network:
         * Set LOTUS_API to https://api.calibration.node.glif.io/rpc/v1
         * Set MARKET_DEAL_URL to https://marketdeals-calibration.s3.amazonaws.com/StateMarketDeals.json.zst
         * Set LOTUS_TEST to 1
       For all other networks:
         * Set LOTUS_API to your network's Lotus API endpoint
         * Set MARKET_DEAL_URL to empty string
         * Set LOTUS_TEST to 0 or 1 based on whether the network address starts with 'f' or 't'
       Switching between different networks with the same database instance is not recommended.

   Logging:
     Singularity uses go-log for logging and can be controlled by below environment variables:
       * GOLOG_LOG_LEVEL  - example values: debug, info, warn, error, dpanic, panic, fatal
       * GOLOG_LOG_FMT    - example values: color, nocolor, json
       * More details can be found at https://github.com/ipfs/go-log

   Upgrading:
     Within each minor version upgrade, use "singularity admin init" to upgrade the database schema.
     For major version upgrade, please refer to the release notes for upgrade instructions.


COMMANDS:
   onboard     Complete data onboarding workflow (storage → preparation → scanning → deal creation)
   version, v  Print version information
   help, h     Shows a list of commands or help for one command
   Daemons:
     run  run different singularity components
   Operations:
     admin                        Admin commands
     deal                         Replication / Deal making management
     deal-schedule-template, dst  Deal schedule template management
     wallet                       Wallet management
     storage                      Create and manage storage system connections
     prep                         Create and manage dataset preparations
     error                        Error log management
   Utility:
     ez-prep      Prepare a dataset from a local path
     download     Download a CAR file from the metadata API
     extract-car  Extract folders or files from a folder of CAR files to a local directory

GLOBAL OPTIONS:
   --database-connection-string value  Connection string to the database (default: sqlite:./singularity.db) [$DATABASE_CONNECTION_STRING]
   --help, -h                          show help
   --json                              Enable JSON output (default: false)
   --verbose                           Enable verbose output. This will print more columns for the result as well as full error trace (default: false)

   Lotus

   --lotus-api value    Lotus RPC API endpoint (default: "https://api.node.glif.io/rpc/v1") [$LOTUS_API]
   --lotus-test         Whether the runtime environment is using Testnet. (default: false) [$LOTUS_TEST]
   --lotus-token value  Lotus RPC API token [$LOTUS_TOKEN]

jeffersonsankara@Jeffersons-MacBook-Pro singularity % 

jeffersonsankara@Jeffersons-MacBook-Pro singularity % ./singularity deal -h
2025-07-31T12:42:30.115-0700	INFO	auto-deal	dataprep/autodeal.go:44	Auto-deal service initialized
NAME:
   singularity deal - Replication / Deal making management

USAGE:
   singularity deal command [command options] [arguments...]

COMMANDS:
   schedule     Schedule deals
   send-manual  Send a manual deal proposal to boost or legacy market
   list         List all deals
   help, h      Shows a list of commands or help for one command

OPTIONS:
   --help, -h  show help
jeffersonsankara@Jeffersons-MacBook-Pro singularity % 

[anjor]

@Sankara-Jefferson it's under deal now as you suggested.

./singularity deal state -h
NAME:
   singularity deal state - Deal state management and monitoring

USAGE:
   singularity deal state command [command options] [arguments...]

DESCRIPTION:
   Comprehensive deal state management tools including:
   - View and filter deal state changes
   - Export state history to CSV/JSON
   - Manual recovery and repair operations
   - State change statistics and analytics

COMMANDS:
   list, ls  List deal state changes with optional filtering and pagination
   get       Get state changes for a specific deal
   stats     Get statistics about deal state changes
   repair    Manual recovery and repair commands for deal state management
   help, h   Shows a list of commands or help for one command

OPTIONS:
   --help, -h  show help

[Sankara-Jefferson]

@anjor yes, that works for me. It looks better. Thei is ready for merging, @ianconsolata.

[ianconsolata]

LGTM, only one minor change, and some questions. @lanzafame Can you do another review and let us know if @anjor has addressed all your concerns?

[parkan]

@ianconsolata please let me know if you want me to rebase this on main, I would probably actually still close this and redo in smaller chunks