fix: queue reservations when no single node has enough GPUs + bin-pack pods

[wdvr]

Summary

• Fix GPU fragmentation scheduling bug: The availability check was summing GPUs across all nodes, so requesting 8 GPUs would see "12 available" (spread across 3 nodes as 4+3+5) and immediately try to create the pod. Since k8s requires all GPUs on a single node, the pod would get stuck Pending for 600s then fail. Now check_gpu_availability returns (total, max_per_node) and scheduling decisions use max_per_node. When no single node can fulfill the request, the reservation queues properly instead of creating an unschedulable pod.

• Add pod affinity for bin-packing: Adds a soft pod affinity (weight 50) that prefers nodes already running gpu-dev pods. This packs smaller reservations onto the same nodes, keeping whole nodes free for large (8-GPU) requests. The profiling node preference (weight 100) still takes priority.

• Better queue messages: When queued due to fragmentation, the message now says "Need 8 B200 GPUs on one node, max 5 available on any single node" instead of the misleading "only 12 available".

What was happening

User requests 8x B200 GPUs
→ Lambda checks: 4+3+5 = 12 B200 GPUs available ✓
→ Creates pod requesting 8 GPUs
→ k8s can't schedule: no single node has 8 free
→ Pod stuck Pending for 600s
→ Lambda times out → reservation marked "failed"

What happens now

User requests 8x B200 GPUs
→ Lambda checks: max on single node = 5, need 8
→ Queues reservation (position #1)
→ Scheduled Lambda retries when a node frees up

Test plan

• Verify 8-GPU B200 reservation queues when no single node has 8 free (instead of failing after 600s timeout)
• Verify small reservations (1-4 GPUs) still schedule immediately when capacity exists
• Verify bin-packing: new 1-GPU reservation lands on a node that already has pods (not an empty node)
• Verify queued reservations are picked up when a node fully frees up

🤖 Generated with Claude Code