danny-avila
diff --git a/‎.gitignore‎
Lines changed: 2 additions & 0 deletions b/‎.gitignore‎
Lines changed: 2 additions & 0 deletions
diff --git a/‎README.md‎
Lines changed: 71 additions & 1 deletion b/‎README.md‎
Lines changed: 71 additions & 1 deletion
diff --git a/‎app/config.py‎
Lines changed: 176 additions & 6 deletions b/‎app/config.py‎
Lines changed: 176 additions & 6 deletions
@@ -1,10 +1,12 @@
 .idea
 .venv
 .env
+.env.beta
 __pycache__
 uploads/
 myenv/
 venv/
 *.pyc
 dev.yml
 SHOPIFY.md
+CLAUDE.md
@@ -64,7 +64,7 @@ The following environment variables are required to run the application:
 - `DEBUG_RAG_API`: (Optional) Set to "True" to show more verbose logging output in the server console, and to enable postgresql database routes
 - `DEBUG_PGVECTOR_QUERIES`: (Optional) Set to "True" to enable detailed PostgreSQL query logging for pgvector operations. Useful for debugging performance issues with vector database queries.
 - `CONSOLE_JSON`: (Optional) Set to "True" to log as json for Cloud Logging aggregations
-- `EMBEDDINGS_PROVIDER`: (Optional) either "openai", "bedrock", "azure", "huggingface", "huggingfacetei", "vertexai", or "ollama", where "huggingface" uses sentence_transformers; defaults to "openai"
+- `EMBEDDINGS_PROVIDER`: (Optional) either "openai", "bedrock", "azure", "huggingface", "huggingfacetei", "vertexai", "ollama", or "nvidia", where "huggingface" uses sentence_transformers; defaults to "openai"
 - `EMBEDDINGS_MODEL`: (Optional) Set a valid embeddings model to use from the configured provider.
     - **Defaults**
     - openai: "text-embedding-3-small"
@@ -74,6 +74,37 @@ The following environment variables are required to run the application:
     - vertexai: "text-embedding-004"
     - ollama: "nomic-embed-text"
     - bedrock: "amazon.titan-embed-text-v1"
+    - nvidia: "nvidia/llama-3.2-nemoretriever-300m-embed-v1"
+- `EMBEDDINGS_PROVIDER_BACKUP`: (Optional) Backup provider for automatic failover ("openai", "bedrock", "azure", "huggingface", "huggingfacetei", "vertexai", "ollama", "nvidia")
+- `EMBEDDINGS_MODEL_BACKUP`: (Optional) Backup model to use when primary provider fails
+- `PRIMARY_FAILOVER_COOLDOWN_MINUTES`: (Optional) Minutes to wait before retrying failed primary provider (default: 1)
+- `EMBED_CONCURRENCY_LIMIT`: (Optional) Maximum concurrent embedding requests to prevent overload (default: 3)
+
+#### Backup Embedding Provider Configuration
+The RAG API supports intelligent backup embedding providers for high availability:
+- **Automatic failover**: When primary provider fails, requests automatically switch to backup
+- **Smart cooldown**: Failed primary providers are avoided for configurable time period
+- **Transparent operation**: LibreChat receives success responses when backup succeeds
+- **Automatic recovery**: Primary provider is retried when cooldown expires
+
+#### NVIDIA Embedding Provider Configuration
+- `NVIDIA_BASE_URL`: (Optional) NVIDIA API endpoint URL (default: "http://localhost:8003/v1")
+- `NVIDIA_API_KEY`: (Optional) API key for NVIDIA embedding service
+- `NVIDIA_MODEL`: (Optional) NVIDIA model to use (default: "nvidia/llama-3.2-nemoretriever-300m-embed-v1")
+- `NVIDIA_INPUT_TYPE`: (Optional) Input type for embeddings ("query", "passage", default: "passage")
+- `NVIDIA_ENCODING_FORMAT`: (Optional) Encoding format ("float", "base64", default: "float")
+- `NVIDIA_TRUNCATE`: (Optional) Truncate input if too long ("NONE", "START", "END", default: "NONE")
+- `NVIDIA_MAX_RETRIES`: (Optional) Maximum retry attempts (default: 3)
+- `NVIDIA_TIMEOUT`: (Optional) Read timeout in seconds (default: 3, connection timeout: 2s)
+- `NVIDIA_MAX_BATCH_SIZE`: (Optional) Maximum texts per batch (default: 32)
+
+#### AWS Bedrock Enhanced Configuration
+- `BEDROCK_EMBEDDING_DIMENSIONS`: (Optional) For Titan V2 models - embedding dimensions (256, 512, or 1024, default: 1024)
+- `BEDROCK_EMBEDDING_NORMALIZE`: (Optional) For Titan V2 models - normalize embeddings ("true"/"false", default: "true")
+- `BEDROCK_MAX_BATCH_SIZE`: (Optional) Maximum texts per Bedrock batch (default: 15)
+- `BEDROCK_INITIAL_RETRY_DELAY`: (Optional) Initial retry delay in seconds for rate limiting (default: 1.0)
+- `BEDROCK_MAX_RETRY_DELAY`: (Optional) Maximum retry delay in seconds (default: 60.0)
+- `BEDROCK_BACKOFF_FACTOR`: (Optional) Exponential backoff multiplier (default: 2.0)
 - `RAG_AZURE_OPENAI_API_VERSION`: (Optional) Default is `2023-05-15`. The version of the Azure OpenAI API.
 - `RAG_AZURE_OPENAI_API_KEY`: (Optional) The API key for Azure OpenAI service.
     - Note: `AZURE_OPENAI_API_KEY` will work but `RAG_AZURE_OPENAI_API_KEY` will override it in order to not conflict with LibreChat setting.
@@ -125,6 +156,45 @@ The `ATLAS_MONGO_DB_URI` could be the same or different from what is used by Lib
 
 Follow one of the [four documented methods](https://www.mongodb.com/docs/atlas/atlas-vector-search/create-index/#procedure) to create the vector index.
 
+### High Availability Configuration Example
+
+For production environments requiring maximum uptime, you can configure redundant embedding providers with automatic failover. This example uses NVIDIA as the primary provider with AWS Bedrock as backup:
+
+```env
+# Primary Provider - NVIDIA Embeddings (Local/On-Premises)
+EMBEDDINGS_PROVIDER=nvidia
+EMBEDDINGS_MODEL=nvidia/llama-3.2-nemoretriever-300m-embed-v1
+NVIDIA_BASE_URL=http://your-nvidia-server:8003/v1
+NVIDIA_API_KEY=your-nvidia-api-key
+NVIDIA_MAX_BATCH_SIZE=32
+NVIDIA_TIMEOUT=3
+
+# Backup Provider - AWS Bedrock Titan V2 Embeddings  
+EMBEDDINGS_PROVIDER_BACKUP=bedrock
+EMBEDDINGS_MODEL_BACKUP=amazon.titan-embed-text-v2:0
+AWS_ACCESS_KEY_ID=your-aws-access-key
+AWS_SECRET_ACCESS_KEY=your-aws-secret-key
+AWS_DEFAULT_REGION=us-west-2
+BEDROCK_EMBEDDING_DIMENSIONS=512
+BEDROCK_EMBEDDING_NORMALIZE=true
+
+# Failover Configuration
+PRIMARY_FAILOVER_COOLDOWN_MINUTES=2
+EMBED_CONCURRENCY_LIMIT=3
+
+# Performance Tuning
+CHUNK_SIZE=1500
+CHUNK_OVERLAP=100
+```
+
+**How this works:**
+- **Primary**: NVIDIA embeddings serve all requests when available
+- **Failover**: If NVIDIA fails, requests automatically switch to Bedrock
+- **Cooldown**: After failure, NVIDIA is not retried for 2 minutes (prevents cascading failures)
+- **Recovery**: NVIDIA is automatically retried when cooldown expires
+- **Transparency**: LibreChat receives successful responses when backup succeeds
+
+This configuration ensures high availability with seamless failover while maintaining optimal performance and cost efficiency.
 
 ### Proxy Configuration
 
 
@@ -27,6 +27,7 @@ class EmbeddingsProvider(Enum):
     OLLAMA = "ollama"
     BEDROCK = "bedrock"
     GOOGLE_VERTEXAI = "vertexai"
+    NVIDIA = "nvidia"
 
 
 def get_env_variable(
@@ -37,6 +38,9 @@ def get_env_variable(
         if default_value is None and required:
             raise ValueError(f"Environment variable '{var_name}' not found.")
         return default_value
+    # Strip comments and whitespace from environment variables
+    if isinstance(value, str) and '#' in value:
+        value = value.split('#')[0].strip()
     return value
 
 
@@ -236,7 +240,7 @@ def init_embeddings(provider, model):
 
         return VertexAIEmbeddings(model=model)
     elif provider == EmbeddingsProvider.BEDROCK:
-        from langchain_aws import BedrockEmbeddings
+        from app.services.embeddings.bedrock_rate_limited import RateLimitedBedrockEmbeddings
 
         session_kwargs = {
             "aws_access_key_id": AWS_ACCESS_KEY_ID,
@@ -248,10 +252,53 @@ def init_embeddings(provider, model):
             session_kwargs["aws_session_token"] = AWS_SESSION_TOKEN
 
         session = boto3.Session(**session_kwargs)
-        return BedrockEmbeddings(
-            client=session.client("bedrock-runtime"),
+        
+        # Get reactive rate limiting configuration from environment
+        max_batch = int(get_env_variable("BEDROCK_MAX_BATCH_SIZE", "15"))
+        max_retries = int(get_env_variable("BEDROCK_MAX_RETRIES", "5"))
+        initial_delay = float(get_env_variable("BEDROCK_INITIAL_RETRY_DELAY", "0.1"))
+        max_delay = float(get_env_variable("BEDROCK_MAX_RETRY_DELAY", "30.0"))
+        backoff_factor = float(get_env_variable("BEDROCK_BACKOFF_FACTOR", "2.0"))
+        recovery_factor = float(get_env_variable("BEDROCK_RECOVERY_FACTOR", "0.9"))
+        
+        # Get Titan V2 specific parameters
+        dimensions = get_env_variable("BEDROCK_EMBEDDING_DIMENSIONS", None)
+        if dimensions is not None:
+            dimensions = int(dimensions)
+        normalize = get_env_variable("BEDROCK_EMBEDDING_NORMALIZE", "true").lower() == "true"
+        
+        # Create client with connection pooling for maximum performance
+        config = boto3.session.Config(
+            max_pool_connections=50,  # Increased for better concurrency
+            retries={'max_attempts': 0}  # We handle retries in our wrapper
+        )
+        
+        return RateLimitedBedrockEmbeddings(
+            client=session.client("bedrock-runtime", config=config),
             model_id=model,
             region_name=AWS_DEFAULT_REGION,
+            max_batch_size=max_batch,
+            max_retries=max_retries,
+            initial_retry_delay=initial_delay,
+            max_retry_delay=max_delay,
+            backoff_factor=backoff_factor,
+            recovery_factor=recovery_factor,
+            dimensions=dimensions,
+            normalize=normalize,
+        )
+    elif provider == EmbeddingsProvider.NVIDIA:
+        from app.services.embeddings.nvidia_embeddings import NVIDIAEmbeddings
+        
+        return NVIDIAEmbeddings(
+            base_url=RAG_OPENAI_BASEURL,
+            model=model,
+            api_key=RAG_OPENAI_API_KEY,
+            max_batch_size=int(get_env_variable("NVIDIA_MAX_BATCH_SIZE", "20")),
+            max_retries=int(get_env_variable("NVIDIA_MAX_RETRIES", "3")),
+            timeout=float(get_env_variable("NVIDIA_TIMEOUT", "3.0")),  # Fast failover - 3 seconds
+            input_type=get_env_variable("NVIDIA_INPUT_TYPE", "query"),
+            encoding_format=get_env_variable("NVIDIA_ENCODING_FORMAT", "float"),
+            truncate=get_env_variable("NVIDIA_TRUNCATE", "NONE"),
         )
     else:
         raise ValueError(f"Unsupported embeddings provider: {provider}")
@@ -285,13 +332,136 @@ def init_embeddings(provider, model):
     EMBEDDINGS_MODEL = get_env_variable(
         "EMBEDDINGS_MODEL", "amazon.titan-embed-text-v1"
     )
-    AWS_DEFAULT_REGION = get_env_variable("AWS_DEFAULT_REGION", "us-east-1")
+elif EMBEDDINGS_PROVIDER == EmbeddingsProvider.NVIDIA:
+    EMBEDDINGS_MODEL = get_env_variable(
+        "EMBEDDINGS_MODEL", "nvidia/llama-3.2-nemoretriever-300m-embed-v1"
+    )
 else:
     raise ValueError(f"Unsupported embeddings provider: {EMBEDDINGS_PROVIDER}")
 
-embeddings = init_embeddings(EMBEDDINGS_PROVIDER, EMBEDDINGS_MODEL)
+# Load AWS credentials ONLY if Bedrock is used as primary or backup
+backup_provider_str = get_env_variable("EMBEDDINGS_PROVIDER_BACKUP", None)
+bedrock_needed = (
+    EMBEDDINGS_PROVIDER == EmbeddingsProvider.BEDROCK or 
+    (backup_provider_str and backup_provider_str.lower() == "bedrock")
+)
 
-logger.info(f"Initialized embeddings of type: {type(embeddings)}")
+if bedrock_needed:
+    AWS_DEFAULT_REGION = get_env_variable("AWS_DEFAULT_REGION", "us-east-1")
+    AWS_ACCESS_KEY_ID = get_env_variable("AWS_ACCESS_KEY_ID", None)
+    AWS_SECRET_ACCESS_KEY = get_env_variable("AWS_SECRET_ACCESS_KEY", None)  
+    AWS_SESSION_TOKEN = get_env_variable("AWS_SESSION_TOKEN", None)
+    logger.debug("AWS credentials loaded for Bedrock provider")
+else:
+    # Set to None when not needed
+    AWS_DEFAULT_REGION = None
+    AWS_ACCESS_KEY_ID = None  
+    AWS_SECRET_ACCESS_KEY = None
+    AWS_SESSION_TOKEN = None
+    logger.debug("AWS credentials not required - no Bedrock provider configured")
+
+# Initialize embeddings with backup support
+def init_embeddings_with_backup():
+    """Initialize embeddings with automatic backup failover."""
+    # Use already loaded backup provider string
+    backup_model = get_env_variable("EMBEDDINGS_MODEL_BACKUP", None)
+    
+    if backup_provider_str and backup_model:
+        # Backup is configured, create backup embeddings with failover
+        backup_provider = EmbeddingsProvider(backup_provider_str.lower())
+        
+        logger.info(f"Backup provider configured: {backup_provider.value} / {backup_model}")
+        
+        try:
+            # Initialize primary provider
+            primary_embeddings = init_embeddings(EMBEDDINGS_PROVIDER, EMBEDDINGS_MODEL)
+            logger.info(f"✅ Primary provider initialized: {EMBEDDINGS_PROVIDER.value}")
+            
+            try:
+                # Initialize backup provider
+                backup_embeddings = init_embeddings(backup_provider, backup_model)
+                logger.info(f"✅ Backup provider initialized: {backup_provider.value}")
+                
+                # Create backup wrapper
+                from app.services.embeddings.backup_embeddings import BackupEmbeddingsProvider
+                
+                # Get cooldown configuration
+                primary_cooldown_minutes = int(get_env_variable("PRIMARY_FAILOVER_COOLDOWN_MINUTES", "1"))
+                
+                # For fast failover, reduce retries on primary provider if it's NVIDIA
+                if EMBEDDINGS_PROVIDER == EmbeddingsProvider.NVIDIA and hasattr(primary_embeddings, 'max_retries'):
+                    logger.info(f"Reducing NVIDIA max_retries from {primary_embeddings.max_retries} to 1 for faster backup failover")
+                    primary_embeddings.max_retries = 1
+                
+                return BackupEmbeddingsProvider(
+                    primary_provider=primary_embeddings,
+                    backup_provider=backup_embeddings,
+                    primary_name=f"{EMBEDDINGS_PROVIDER.value}:{EMBEDDINGS_MODEL}",
+                    backup_name=f"{backup_provider.value}:{backup_model}",
+                    primary_cooldown_minutes=primary_cooldown_minutes
+                )
+                
+            except Exception as backup_error:
+                logger.warning(f"⚠️ Backup provider failed to initialize: {str(backup_error)}")
+                logger.info(f"Continuing with primary provider only: {EMBEDDINGS_PROVIDER.value}")
+                return primary_embeddings
+                
+        except Exception as primary_error:
+            logger.error(f"❌ Primary provider failed to initialize: {str(primary_error)}")
+            
+            # Try to initialize backup as primary
+            try:
+                backup_embeddings = init_embeddings(backup_provider, backup_model)
+                logger.warning(f"🔄 Using backup provider as primary: {backup_provider.value}")
+                return backup_embeddings
+            except Exception as backup_error:
+                logger.error(f"❌ Both providers failed to initialize!")
+                raise RuntimeError(
+                    f"Failed to initialize any embedding provider. "
+                    f"Primary ({EMBEDDINGS_PROVIDER.value}): {str(primary_error)}, "
+                    f"Backup ({backup_provider.value}): {str(backup_error)}"
+                ) from primary_error
+    else:
+        # No backup configured, use single provider
+        return init_embeddings(EMBEDDINGS_PROVIDER, EMBEDDINGS_MODEL)
+
+try:
+    embeddings = init_embeddings_with_backup()
+    logger.info(f"Initialized embeddings of type: {type(embeddings)}")
+except Exception as e:
+    error_message = str(e)
+    
+    # Provide helpful configuration error messages
+    if EMBEDDINGS_PROVIDER == EmbeddingsProvider.BEDROCK:
+        if "model identifier is invalid" in error_message:
+            logger.error(
+                f"❌ BEDROCK CONFIGURATION ERROR ❌\n\n"
+                f"The Bedrock model '{EMBEDDINGS_MODEL}' is not available in region '{AWS_DEFAULT_REGION}'.\n\n"
+                f"💡 Quick Fix:\n"
+                f"   Set EMBEDDINGS_MODEL=amazon.titan-embed-text-v1 in your .env file\n\n"
+                f"🔍 Available models in most regions:\n"
+                f"   • amazon.titan-embed-text-v1\n"
+                f"   • cohere.embed-english-v3\n"
+                f"   • cohere.embed-multilingual-v3\n\n"
+                f"🌍 To check available models in {AWS_DEFAULT_REGION}:\n"
+                f"   AWS Console → Bedrock → Foundation models → Embedding"
+            )
+        elif "AccessDeniedException" in error_message:
+            logger.error(
+                f"❌ BEDROCK ACCESS ERROR ❌\n\n"
+                f"Your AWS account doesn't have access to Bedrock in '{AWS_DEFAULT_REGION}'.\n\n"
+                f"💡 Solutions:\n"
+                f"   1. AWS Console → Bedrock → Model access → Request model access\n"
+                f"   2. Enable foundation models you want to use\n"
+                f"   3. Verify IAM permissions include 'bedrock:InvokeModel'\n\n"
+                f"⚠️  Note: Bedrock may not be available in all regions"
+            )
+        else:
+            logger.error(f"❌ BEDROCK ERROR: {error_message}")
+    else:
+        logger.error(f"❌ EMBEDDINGS ERROR ({EMBEDDINGS_PROVIDER}): {error_message}")
+    
+    raise RuntimeError(f"Failed to initialize embeddings: {error_message}") from e
 
 # Vector store
 if VECTOR_DB_TYPE == VectorDBType.PGVECTOR: