Skip to content

Merge Strategy: infra/refactor → main (v0.1.3 + 16k lines across 109 files) #8

Open
Open
Merge Strategy: infra/refactor → main (v0.1.3 + 16k lines across 109 files)#8
Labels
enhancementNew feature or request
@bryanchriswhite

Description

@bryanchriswhite
bryanchriswhite
opened on Nov 18, 2025

Merge Strategy: infra/refactor → main

Overview

This issue tracks the comprehensive merge strategy for bringing the infra/refactor branch (65 commits, 109 files changed) back into main. The branch contains significant architectural improvements and new features built on top of v0.1.3.

Summary Statistics

  • Files Changed: 109 files
  • Lines Added: 16,903
  • Lines Removed: 296
  • Net Change: +16,607 lines
  • Commits: 65 commits since v0.1.3
  • New Infrastructure Files: 66 files (k8s/, oauth-broker/, pinshare-ui/)

Critical Backwards Incompatible Changes

🚨 1. API Versioning (High Severity)

Breaking Change: All API routes now require /api/v1 prefix

Old Route New Route
GET /files GET /api/v1/files
GET /files/{sha256} GET /api/v1/files/{sha256}
GET /p2p/status GET /api/v1/p2p/status

Impact: All API clients must update endpoint URLs

Related Commits:

  • 166b1ad - Implement /api/v1 versioning for all API endpoints
  • 74e5ac3 - Standardize all API routes to use /api prefix

Migration Required: Yes - Update all API client code

🚨 2. Docker Architecture (High Severity)

Breaking Change: IPFS daemon separated from PinShare container

Old Behavior:

  • Single container with bundled IPFS daemon
  • Entrypoint starts IPFS, waits 60s, then starts PinShare
  • IPFS data volume mounted at /data/ipfs

New Behavior:

  • PinShare connects to external IPFS API via IPFS_API env var
  • No bundled IPFS daemon
  • Requires separate IPFS container/daemon

Impact: Existing Docker deployments will break without migration

Migration Required: Yes - Deploy separate IPFS container

Example Migration:

# New deployment approach
docker run -d --name ipfs ipfs/kubo
docker run -d --name pinshare \
  -e IPFS_API=http://ipfs:5001 \
  pinshare:latest

Major Architectural Changes

1. Infrastructure & Deployment ⚙️

Complete Kubernetes deployment structure:

  • 27 Kubernetes manifest files with Kustomize overlays (dev/production)
  • Tiltfile for local Kubernetes development with live reload
  • Docker Compose configuration for full stack
  • SOPS-encrypted secrets management
  • IPFS now runs as separate StatefulSet

Files: k8s/, Tiltfile, docker-compose.yml, .sops.yaml

2. React Web UI 🎨

Complete modern web interface:

  • React + Vite + TailwindCSS + React Router
  • Pages: Browse, Network, Network Graph, Upload Status, Google Drive Import
  • Playwright E2E tests
  • Production deployment scripts (Nginx + S3/CloudFront ready)
  • 21 files, 7,809 lines

Files: pinshare-ui/ directory

Backwards Compatible: ✅ Yes (additive, API remains unchanged)

3. OAuth Broker Microservice 🔐

Standalone OAuth service for production:

  • Handles Google OAuth flow where redirect URLs can't point to localhost
  • Session management with expiry
  • PostMessage-based token forwarding to UI
  • Separate Go module with GitHub Actions workflow

Files: oauth-broker/ directory, .github/workflows/oauth-broker.yml

Backwards Compatible: ✅ Yes (optional service for Google Drive import)

4. Database Layer 💾

SQLite integration for persistent state:

  • User management (Google OAuth users)
  • Import job tracking with status
  • Encrypted token storage (AES-256)
  • Database migrations
  • Structured schema with foreign keys and indexes

Files: internal/db/db.go, internal/db/jobs.go, internal/db/users.go

New Dependency: github.com/mattn/go-sqlite3

Backwards Compatible: ✅ Yes (additive, doesn't affect metadata.json)

5. Job Queue System 🔄

Background job processing:

  • Worker pool-based queue
  • Concurrent job processing with configurable workers
  • Context-based cancellation
  • Job lifecycle tracking

Files: internal/jobs/queue.go, internal/jobs/import_job.go

Backwards Compatible: ✅ Yes (internal component)

New Features (All Backwards Compatible)

1. Google Drive Import 📂

Complete Google Drive integration:

  • OAuth 2.0 authentication with PKCE
  • Google Drive API client
  • File browser with folder navigation
  • Background job queue for imports
  • Database tracking of import jobs
  • Integration with existing upload pipeline
  • Comprehensive documentation

Files: internal/api/gdrive_api.go, internal/gdrive/, UI components, docs/GOOGLE_DRIVE_IMPORT.md

New Dependencies: golang.org/x/oauth2, google.golang.org/api

Key Commits:

  • d4e96a5 - Implement Google Drive Import Feature
  • 0988441 - Fix Google Drive OAuth integration and add centralized OAuth broker
  • 481f71d - Persist imported Google Drive files to metadata.json

2. Upload Status Tracking 📊

Real-time upload progress monitoring:

  • Status manager with concurrent upload tracking
  • Progress stages: validating → hashing → scanning → uploading → storing → completed
  • Context-based cancellation support
  • API endpoints and UI page

Endpoints:

  • GET /api/v1/upload-status - All uploads
  • GET /api/v1/upload-status/active - Active uploads
  • POST /api/v1/upload-status/{fileName}/cancel - Cancel upload

Files: internal/store/upload_status.go, UI components

3. FileName Metadata Field 📝

Original filename tracking:

  • New fileName field in BaseMetadata
  • Captured during file scan
  • Optional field (backwards compatible)
  • UI displays filenames instead of SHA256/CID

Files: internal/store/store.go, OpenAPI spec

4. Network Visualization 🕸️

Interactive P2P network graph:

  • Force-directed graph using react-force-graph-2d
  • Real-time peer connections
  • Node details on hover/click

Files: pinshare-ui/src/pages/NetworkGraph.jsx

5. Additional Improvements

  • CORS configuration via PS_ALLOWED_ORIGINS env var
  • IPFS pinning endpoint: POST /api/v1/ipfs/pin/{cid}
  • Health endpoint: GET /health

Proposed Merge Strategy (10-11 Independent PRs)

Phase 1: Foundation (Breaking Changes First) ⚠️

PR #1: Docker Refactor - IPFS Separation

  • Priority: Critical (Breaking)
  • Files: ~4 files (Dockerfile, docker-compose.yml, docs)
  • Includes: Migration guide for Docker users
  • Dependencies: None
  • Effort: 1 day

PR #2: API Versioning + CORS

  • Priority: Critical (Breaking)
  • Files: ~10 files (routing, API handlers)
  • Includes: /api/v1 prefix, CORS middleware, health endpoint
  • Dependencies: None
  • Effort: 1-2 days
  • Note: Consider backwards compatibility shim for one release

PR #3: FileName Field Addition

  • Priority: Medium
  • Files: ~5 files (metadata, OpenAPI spec, store)
  • Dependencies: None
  • Effort: 0.5 days

PR #4: Database Layer

  • Priority: High (Foundation for features)
  • Files: 3 files (internal/db/)
  • Includes: SQLite, migrations, user/job tables
  • Dependencies: None
  • Effort: 1 day

PR #5: Job Queue System

  • Priority: High (Foundation for features)
  • Files: 2 files (internal/jobs/)
  • Includes: Worker pool, background processing
  • Dependencies: None
  • Effort: 0.5 days

Phase 2: Core Features 🚀

PR #6: Upload Status Tracking

PR #7: Google Drive Import (Backend)

PR #8: OAuth Broker Service

Phase 3: User Experience 🎨

PR #9: Web UI

PR #10: Kubernetes Deployment

  • Priority: Medium
  • Files: 34 files (k8s/, Tiltfile)
  • Includes: K8s manifests, Kustomize overlays, Tilt, SOPS secrets
  • Dependencies: None (infrastructure)
  • Effort: 2-3 days

Phase 4: Polish ✨

PR #11: Documentation

  • Priority: Low
  • Files: 4 files (docs/)
  • Includes: Architecture docs, Google Drive guide, README updates
  • Dependencies: All previous PRs (should reference them)
  • Effort: 0.5 days

Risk Assessment

🔴 High Risk

  1. API Versioning - Breaking change affects all API consumers
  2. Docker Refactor - Breaking change affects all Docker deployments

Mitigation:

  • Comprehensive migration guides with examples
  • Consider deprecation warnings in v0.1.4 if possible
  • Test migration path thoroughly
  • Clear communication in release notes

🟡 Medium Risk

  1. Google Drive Import - Complex OAuth flow, external API dependencies
  2. Kubernetes Deployment - Complex infrastructure configuration

Mitigation:

  • Thorough testing of OAuth flows (success, failure, token refresh)
  • Test K8s on multiple clusters (kind, minikube, cloud)
  • Feature flags for Google Drive import
  • Comprehensive documentation

🟢 Low Risk

  1. Web UI - Additive, well-isolated component
  2. Upload Status - Small scope, well-defined
  3. Documentation - No code risk

Testing Requirements

Per-PR Testing Checklist

PR #1 (Docker):

  • Docker Compose stack startup with separate IPFS
  • IPFS connectivity verification
  • Volume persistence
  • Multi-platform builds (amd64, arm64)

PR #2 (API):

  • All endpoints accessible at new routes
  • CORS with different origins
  • Health endpoint responds correctly

PR #7 (Google Drive):

  • OAuth flow (success, failure, token refresh)
  • Large file imports (>100MB)
  • Network interruptions during download
  • Concurrent job processing
  • Error handling (quota limits, permission errors)

PR #9 (Web UI):

  • Playwright E2E tests pass
  • Browser compatibility (Chrome, Firefox, Safari)
  • Mobile responsiveness

PR #10 (Kubernetes):

  • Local cluster deployment (kind/minikube)
  • Cloud cluster deployment
  • Secret encryption/decryption with SOPS
  • Tilt hot-reload functionality

Migration Checklist (v0.1.3 → New Version)

✅ Critical (Required for All Users)

  • Update API client code to use /api/v1 prefix for all endpoints
  • Reconfigure Docker deployment for separate IPFS container
  • Set IPFS_API environment variable pointing to IPFS daemon
  • Update docker-compose or orchestration scripts
  • Test migration in staging environment before production

🔧 Optional (For New Features)

  • Configure Google OAuth credentials (GOOGLE_CLIENT_ID, GOOGLE_CLIENT_SECRET, GOOGLE_REDIRECT_URL)
  • Generate and set PS_ENCRYPTION_KEY (32 bytes for AES-256)
  • Configure PS_ALLOWED_ORIGINS for CORS
  • Deploy OAuth broker for production Google Drive import
  • Deploy web UI (default port 5174)
  • Set PS_DATABASE_FILE, PS_TEMP_DOWNLOAD_DIR, PS_MAX_CONCURRENT_JOBS

☸️ Kubernetes Users

  • Apply new K8s manifests from k8s/base/
  • Configure Kustomize overlay for environment
  • Set up SOPS encryption for secrets
  • Update Tiltfile for local development context

New Environment Variables

Required for Core Functionality

IPFS_API=http://localhost:5001  # External IPFS API endpoint

Optional for Features

# Google Drive Import
GOOGLE_CLIENT_ID=your-client-id
GOOGLE_CLIENT_SECRET=your-client-secret
GOOGLE_REDIRECT_URL=http://localhost:8080/api/v1/auth/google/callback
PS_ENCRYPTION_KEY=your-32-byte-aes-key

# CORS
PS_ALLOWED_ORIGINS=http://localhost:5174,https://pinshare.example.com

# Database & Jobs
PS_DATABASE_FILE=pinshare.db
PS_TEMP_DOWNLOAD_DIR=./tmp/gdrive
PS_MAX_CONCURRENT_JOBS=3
PS_MAX_FILE_SIZE_MB=1024

New External Dependencies

Go Modules

  • github.com/mattn/go-sqlite3 - SQLite database
  • golang.org/x/oauth2 - OAuth 2.0 client
  • google.golang.org/api - Google Drive API
  • github.com/google/uuid - UUID generation

Infrastructure

  • Separate IPFS daemon/container (required)
  • Kubernetes cluster (optional)
  • SOPS for secrets (optional)
  • OAuth broker service (optional for production)

Next Steps

  1. Review this proposal - Validate the grouping and merge order
  2. Create tracking issues - One issue per PR group
  3. Prioritize PRs - Determine which to tackle first
  4. Create feature branches - Extract each group into clean branches
  5. Draft PRs - Start with Phase 1 (breaking changes)
  6. Migration guides - Write detailed upgrade documentation
  7. Release planning - Decide version numbering (v0.2.0? v1.0.0?)

Questions for Discussion

  1. Should we maintain backwards compatibility for API routes in one transitional release?
  2. What version number should the merged work become? (v0.2.0, v0.3.0, v1.0.0?)
  3. Should any of these PRs be combined or split differently?
  4. Priority order - does Phase 1 → Phase 4 make sense, or should we reorder?
  5. Should we create a v0.1.4 patch release before merging breaking changes?

Related Issues

Branch: infra/refactor
Base: origin/v0.1.3
Target: main

This issue serves as the master tracking issue for the infra/refactor merge effort. Individual PRs will reference this issue.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions