feat: add OpenAI-compatible embeddings backend support

[aeneasr]

Summary

This PR adds support for OpenAI-compatible embedding APIs (OpenAI, Voyage AI, Azure OpenAI, Together AI, etc.) as a new embedder backend alongside the existing Ollama and LM Studio implementations.

Key Changes

• New OpenAI embedder implementation (internal/embedder/openai.go):

  • Implements the Embedder interface for OpenAI-compatible /v1/embeddings endpoints
  • Supports configurable base URL and API key for flexibility across different providers
  • Batches requests in groups of 32 with automatic retry logic for transient failures (5xx, 429 rate limits)
  • Handles out-of-order responses by sorting embeddings by index as per OpenAI spec
  • Respects context cancellation between retry attempts

• Comprehensive test coverage (internal/embedder/openai_test.go):

  • Tests basic embedding functionality with mock HTTP server
  • Validates batching behavior (50 texts split into 32+18 batches)
  • Verifies response ordering when items arrive out-of-order
  • Tests rate limit retry logic with exponential backoff
  • Validates error handling and context cancellation
  • Ensures API key validation at initialization

• Configuration updates (internal/config/config.go):

  • Added BackendOpenAI constant and configuration fields (OpenAIBaseURL, OpenAIAPIKey)
  • Updated backend validation to accept "openai" as a valid backend
  • Added required OPENAI_API_KEY environment variable validation
  • Defaults to text-embedding-3-small model for OpenAI backend

• Model registry expansion (internal/embedder/models.go, internal/embedder/models_test.go):

  • Added 5 OpenAI-compatible models: text-embedding-3-small, text-embedding-3-large, text-embedding-ada-002, voyage-code-3, voyage-3-large
  • Each model includes proper dimensions, context length, and minimum similarity score configuration

• Embedder factory (cmd/embedder.go):

  • Updated newEmbedder() to instantiate OpenAI embedder when backend is "openai"

• Health check improvements (cmd/stdio.go):

  • Extended health check handler to support OpenAI backend with proper probe URL and authorization header
  • Added specific error reporting for invalid API keys (401 responses)

• Documentation (CLAUDE.md):

  • Updated architecture table and environment variable documentation to reflect OpenAI support

Implementation Details

• Uses sethvargo/go-retry for exponential backoff retry logic with configurable max retries
• HTTP client configured with 10-minute timeout to handle large embedding requests
• Request batching respects OpenAI's batch size limits (32 items per request)
• Proper error wrapping and context propagation throughout the embedding pipeline

https://claude.ai/code/session_012e2TZ36P7dcbmBd2HjQrQk

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}