Skip to content

Phase 2: Cloud provider abstraction and snapshot_utils refactor#29

Open
wdvr wants to merge 1 commit into
devfrom
feat/phase2-storage-abstraction
Open

Phase 2: Cloud provider abstraction and snapshot_utils refactor#29
wdvr wants to merge 1 commit into
devfrom
feat/phase2-storage-abstraction

Conversation

@wdvr
Copy link
Copy Markdown
Owner

@wdvr wdvr commented Feb 4, 2026

Summary

This PR implements Phase 2 of the cloud-agnostic migration plan:

  • New providers/ module with abstract CloudProvider and AuthProvider interfaces
  • AWSProvider implementation wrapping existing boto3 code
  • GCPProvider and CustomProvider** stubs with extensive documentation
  • Refactored snapshot_utils.py to use provider interface instead of direct boto3 calls

Provider Interface

from providers import get_cloud_provider

provider = get_cloud_provider()  # Returns AWS/GCP/Custom based on CLOUD_PROVIDER env

# Storage operations
volume = provider.create_volume(size_gb=100, availability_zone='us-east-2a')
snapshot = provider.create_snapshot(volume.volume_id, tags={'user': 'john'})
snapshots = provider.list_snapshots(filters={'gpu-dev-user': 'john'}, status=['completed'])

# Object storage
uri = provider.upload_to_object_storage('bucket', 'key', content, content_type='text/plain')

Changes to snapshot_utils.py

Before After
import boto3; ec2_client = boto3.client("ec2") from providers import get_cloud_provider; provider = _get_provider()
ec2_client.describe_snapshots(Filters=[...]) provider.list_snapshots(filters={...}, status=[...])
ec2_client.create_snapshot(VolumeId=...) provider.create_snapshot(volume_id, tags={...})
s3_client.put_object(Bucket=..., Key=...) provider.upload_to_object_storage(bucket, key, content)

Next Steps

  • disk_reconciler.py needs refactoring (requires adding create_tags() / delete_tags() to interface)
  • Phase 3: K8s-native VolumeSnapshot support
  • Phase 4: GCP implementation (separate PR)
  • Phase 5: OIDC integration (separate PR)

Test plan

  • Unit tests for provider interface
  • Verify snapshot operations work with AWS backend
  • Test pagination for users with many snapshots

🤖 Generated with Claude Code

Phase 2 of cloud-agnostic migration:

- Add providers/ module with CloudProvider and AuthProvider interfaces
- Implement AWSProvider wrapping existing boto3 code
- Add GCP and Custom provider stubs with documentation
- Refactor snapshot_utils.py to use provider interface:
  - Replace direct boto3 ec2_client with _get_provider()
  - Replace s3_client with provider.upload/download_from_object_storage()
  - Update all snapshot operations to use provider methods
  - Support pagination in list_snapshots for large result sets
  - Add volume_id and status filtering to snapshot queries

Provider interface supports:
- Block storage (create/delete/attach/detach volumes)
- Snapshots (create/delete/list/wait)
- Object storage (upload/download)
- Compute node queries

Next steps: Refactor disk_reconciler.py (requires adding tag operations
to provider interface)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant