ViDSPy 🎬

DSPy-style framework for optimizing text-to-video prompts via VBench metric feedback.

ViDSPy brings the power of DSPy's declarative programming paradigm to text-to-video generation. Optimize your prompts for any text-to-video API (Runway, Pika, Replicate, etc.) using VBench quality metrics and VLM-based feedback.

Note: ViDSPy optimizes how you prompt video generation models. It does not generate videos itself - you bring your own text-to-video API.

🎯 Key Features

DSPy-style Optimization: Tune instructions (prompt templates) and demonstrations (few-shot examples) using canonical DSPy optimizers
VBench Integration: Full support for all 10 CORE_METRICS from VBench evaluation
Multiple Optimizers: BootstrapFewShot, LabeledFewShot, MIPROv2, COPRO, and GEPA
Flexible VLM Backends: OpenRouter (cloud) and HuggingFace (local) support
Composite Metrics: Weighted combination of video quality (60%) and text-video alignment (40%)

📦 Installation

pip install vidspy

For full functionality with VBench evaluation:

pip install vidspy[vbench]

For development:

pip install vidspy[all]

⚙️ Configuration

ViDSPy can be configured in two ways:

Option 1: Pass Arguments Directly in Code

from vidspy import ViDSPy

vidspy = ViDSPy(
    vlm_backend="openrouter",
    vlm_model="google/gemini-2.5-flash",
    api_key="your-api-key",
    device="auto"
)

Option 2: Use Configuration File

Create a vidspy_config.yaml file from the template:

cp vidspy_config.yaml.example vidspy_config.yaml

Edit the configuration file:

# vidspy_config.yaml
# VLM Provider Settings
vlm:
  backend: openrouter  # or "huggingface"
  model: google/gemini-2.5-flash
  # api_key: your-api-key-here  # Or set OPENROUTER_API_KEY env var

# Optimizer LLM Settings
# This LLM is used by DSPy optimizers (MIPROv2, COPRO, GEPA) to generate instruction variations
optimizer:
  lm: openai/gpt-4o-mini  # Any model from OpenRouter
  # api_key: your-api-key-here  # Or set OPENROUTER_OPTIMIZER_API_KEY env var

# Optimization Settings
optimization:
  default_optimizer: mipro_v2
  max_bootstrapped_demos: 4
  max_labeled_demos: 4

# Metric Settings
metrics:
  quality_weight: 0.6
  alignment_weight: 0.4
  quality_metrics:
    - subject_consistency
    - motion_smoothness
    - temporal_flickering
    - human_anatomy
    - aesthetic_quality
    - imaging_quality
  alignment_metrics:
    - object_class
    - human_action
    - spatial_relationship
    - overall_consistency

# Target Thresholds
targets:
  human_anatomy: 0.85
  alignment: 0.80

# Cache Settings
cache:
  dir: ~/.cache/vidspy
  vbench_models: ~/.cache/vbench

# Hardware Settings
hardware:
  device: auto  # "cuda", "cpu", or "auto"
  dtype: float16

Then use ViDSPy without any arguments - it automatically loads the config:

from vidspy import ViDSPy, VideoChainOfThought, Example

# ViDSPy automatically finds and loads vidspy_config.yaml
vidspy = ViDSPy()

# All settings from the config file are now applied!
trainset = [Example(prompt="a cat jumping", video_path="cat.mp4")]
optimized = vidspy.optimize(VideoChainOfThought("prompt -> video"), trainset)

Config file search order:

ViDSPy automatically searches for vidspy_config.yaml in:

Current working directory: ./vidspy_config.yaml
User config directory: ~/.vidspy/config.yaml
User home directory: ~/vidspy_config.yaml

Custom config path:

You can also specify a custom config file location:

vidspy = ViDSPy(config_path="/path/to/custom_config.yaml")

Important:

Arguments passed directly to ViDSPy() always override config file values
API keys should be in environment variables or .env file (not in the config file):

# .env
OPENROUTER_API_KEY=your-api-key-here
OPENROUTER_OPTIMIZER_API_KEY=your-api-key-here  # For optimizer LLM

🤖 Optimizer LLM Configuration

ViDSPy uses two different LLMs:

VLM (Vision Language Model): Analyzes videos and enhances prompts during inference
Optimizer LLM: Used by MIPROv2, COPRO, and GEPA optimizers to generate instruction variations during optimization

Note: BootstrapFewShot and LabeledFewShot optimizers do NOT require an optimizer LLM - they work without this configuration.

Configure the Optimizer LLM:

from vidspy import ViDSPy

vidspy = ViDSPy(
    vlm_backend="openrouter",
    vlm_model="google/gemini-2.5-flash",          # For video analysis
    optimizer_lm="openai/gpt-4o-mini",            # For optimization
    optimizer_api_key="your-openrouter-api-key"   # Or set OPENROUTER_OPTIMIZER_API_KEY env var
)

Or in vidspy_config.yaml:

vlm:
  backend: openrouter
  model: google/gemini-2.5-flash

optimizer:
  lm: openai/gpt-4o-mini  # Any OpenRouter model

Choosing an Optimizer LLM:

For most users: openai/gpt-4o-mini (fast and cost-effective)
For better quality: openai/gpt-4o or anthropic/claude-3-5-sonnet
Any model from OpenRouter is supported

Note: If optimizer_api_key is not specified, it will use the VLM API key as fallback.

🎥 Connecting Video Generation Models

Important: ViDSPy is a prompt optimization framework that sits on top of existing text-to-video models. It does NOT generate videos itself. Instead, it optimizes how you prompt external video generation services.

How ViDSPy Works

User Prompt → ViDSPy (optimize prompt) → Text-to-Video API → Generated Video
                ↓
            VBench + VLM (evaluate quality)
                ↓
        Learn better prompting strategies

Setting Up Your Video Generator

You need to provide a video_generator function that connects to your preferred text-to-video service:

Example 1: Runway Gen-3

from vidspy import VideoChainOfThought

def runway_generator(prompt: str, **kwargs) -> str:
    """Generate video using Runway Gen-3 API."""
    import requests

    response = requests.post(
        "https://api.runwayml.com/v1/generate",
        headers={"Authorization": f"Bearer {RUNWAY_API_KEY}"},
        json={
            "prompt": prompt,
            "model": "gen3",
            "duration": kwargs.get("duration", 5)
        }
    )

    video_url = response.json()["output"]["url"]
    # Download and save video locally
    video_path = f"outputs/{response.json()['id']}.mp4"
    # ... download logic ...
    return video_path

# Create module with your generator
module = VideoChainOfThought(
    "prompt -> video",
    video_generator=runway_generator
)

Example 2: Replicate (Stable Video Diffusion, CogVideo, etc.)

import replicate

def replicate_generator(prompt: str, **kwargs) -> str:
    """Generate video using Replicate API."""
    output = replicate.run(
        "stability-ai/stable-video-diffusion",
        input={"prompt": prompt}
    )

    # Save video from output URL
    video_path = f"outputs/{uuid.uuid4()}.mp4"
    # ... download logic ...
    return video_path

module = VideoChainOfThought(
    "prompt -> video",
    video_generator=replicate_generator
)

Example 3: Pika Labs

from pika import PikaClient

def pika_generator(prompt: str, **kwargs) -> str:
    """Generate video using Pika Labs API."""
    client = PikaClient(api_key=PIKA_API_KEY)

    video = client.generate_video(
        prompt=prompt,
        aspect_ratio="16:9",
        duration=3
    )

    return video.download_path

module = VideoChainOfThought(
    "prompt -> video",
    video_generator=pika_generator
)

Supported Text-to-Video Services

ViDSPy works with any text-to-video API. Popular options include:

Service	API Available	Notes
Runway Gen-3	✅ Yes	High quality, good motion
Pika Labs	✅ Yes	Creative effects, good for social media
Stability AI Video	✅ Yes	Open weights available
Replicate	✅ Yes	Multiple models (CogVideo, SVD, etc.)
LumaAI Dream Machine	✅ Yes	Cinematic quality
HaiperAI	✅ Yes	Fast generation
Morph Studio	✅ Yes	Style control

Custom Video Generator Template

def my_video_generator(prompt: str, **kwargs) -> str:
    """
    Your custom video generation function.

    Args:
        prompt: Enhanced prompt from ViDSPy
        **kwargs: Additional parameters (duration, aspect_ratio, etc.)

    Returns:
        Local path to the generated video file
    """
    # 1. Call your text-to-video API
    # 2. Download the generated video
    # 3. Save it locally
    # 4. Return the file path

    video_path = "path/to/generated/video.mp4"
    return video_path

🚀 Quick Start

from vidspy import ViDSPy, VideoChainOfThought, Example

# Step 1: Define your video generator (connect to your text-to-video API)
def my_video_generator(prompt: str, **kwargs) -> str:
    """Your text-to-video API integration."""
    # Example: Runway, Pika, Replicate, etc.
    import my_video_api
    video = my_video_api.generate(prompt=prompt)
    return video.save_path()

# Step 2: Initialize ViDSPy with OpenRouter VLM backend
# Note: Optimizer LLM defaults to "openai/gpt-4o-mini" via OpenRouter
# You can customize: ViDSPy(vlm_backend="openrouter", optimizer_lm="openai/gpt-4o")
vidspy = ViDSPy(vlm_backend="openrouter")

# Step 3: Create training examples (use videos you've already generated)
trainset = [
    Example(prompt="a cat jumping over a fence", video_path="cat_jump.mp4"),
    Example(prompt="a dog running in a park", video_path="dog_run.mp4"),
    Example(prompt="a bird flying through clouds", video_path="bird_fly.mp4"),
]

# Step 4: Create module with your video generator
module = VideoChainOfThought(
    "prompt -> video",
    video_generator=my_video_generator  # Connect your generator!
)

# Step 5: Optimize prompting strategy
optimized = vidspy.optimize(
    module,
    trainset,
    optimizer="mipro_v2"  # Multi-stage instruction + demo optimization
)

# Step 6: Generate videos with optimized prompts
result = optimized("a dolphin swimming in the ocean")
print(f"Generated video: {result.video_path}")
print(f"Optimized prompt used: {result.enhanced_prompt}")

📊 VBench Metrics

ViDSPy uses VBench's 10 CORE_METRICS split into two categories:

Video Quality Metrics (60% weight, video-only)

Metric	Description
`subject_consistency`	Temporal stability of subjects
`motion_smoothness`	Natural motion quality
`temporal_flickering`	Absence of temporal jitter
`human_anatomy`	Correct hands/faces/torso rendering
`aesthetic_quality`	Artistic/visual beauty
`imaging_quality`	Technical clarity and sharpness

Text-Video Alignment Metrics (40% weight, prompt-conditioned)

Metric	Description
`object_class`	Prompt objects appear correctly
`human_action`	Prompt actions performed correctly
`spatial_relationship`	Correct spatial layout
`overall_consistency`	Holistic text-video alignment

Using Metrics

from vidspy.metrics import composite_reward, quality_score, alignment_score

# Default composite metric (60% quality + 40% alignment)
score = composite_reward(example, prediction)

# Quality-only score
q_score = quality_score(example, prediction)

# Alignment-only score
a_score = alignment_score(example, prediction)

# Custom metric configuration
from vidspy.metrics import VBenchMetric

custom_metric = VBenchMetric(
    quality_weight=0.5,
    alignment_weight=0.5,
    quality_metrics=["motion_smoothness", "aesthetic_quality"],
    alignment_metrics=["object_class", "overall_consistency"]
)

🔧 Optimizers

ViDSPy provides 5 DSPy-compatible optimizers:

Optimizer	Description	Key Parameters
`VidBootstrapFewShot`	Auto-generate/select few-shots	`max_bootstrapped_demos=4`
`VidLabeledFewShot`	Static few-shot assignment	`k=3`
`VidMIPROv2`	Multi-stage instruction + demo optimization	`num_candidates=10, auto="light"`
`VidCOPRO`	Cooperative multi-LM instruction optimization	`breadth=5, depth=3`
`VidGEPA`	Generate + Evaluate + Propose + Accept	`auto="light"`

Example: Using Different Optimizers

# Bootstrap few-shot
optimized = vidspy.optimize(
    module, trainset,
    optimizer="bootstrap",
    max_bootstrapped_demos=4
)

# MIPROv2 with more candidates
optimized = vidspy.optimize(
    module, trainset,
    optimizer="mipro_v2",
    num_candidates=15,
    auto="medium"
)

# COPRO with custom search
optimized = vidspy.optimize(
    module, trainset,
    optimizer="copro",
    breadth=10,
    depth=5
)

🤖 VLM Providers

Vision Language Models (VLMs) in ViDSPy are used for:

📝 Prompt enhancement - Improving user prompts before generation
🔍 Video analysis - Understanding generated video content
🎯 Quality assessment - Analyzing text-video alignment
🧠 Chain-of-thought reasoning - Planning video generation strategies

Note: VLMs do NOT generate videos. They help optimize the prompts you send to your text-to-video API.

OpenRouter (Default)

Cloud-based multimodal VLMs via unified API:

vidspy = ViDSPy(
    vlm_backend="openrouter",
    vlm_model="google/gemini-2.5-flash",
    api_key="your-api-key"  # Or set OPENROUTER_API_KEY env var
)

Example models:

google/gemini-2.5-flash
google/gemini-1.5-pro
anthropic/claude-opus-4.5
openai/gpt-4o

HuggingFace (Local)

Local video VLMs for offline usage:

vidspy = ViDSPy(
    vlm_backend="huggingface",
    vlm_model="llava-hf/llava-v1.6-mistral-7b-hf",
    device="cuda"
)

📁 Video Modules

ViDSPy provides several module types for different prompting strategies.

Module Signatures

When creating video modules, you'll typically use simple signature strings like "prompt -> video" or "prompt -> video_path". The library internally handles all the complex reasoning steps (scene analysis, motion planning, etc.) based on the module type you choose. You don't need to worry about the internal signature details—just use these standard formats and ViDSPy takes care of the rest.

from vidspy import VideoPredict, VideoChainOfThought, VideoReAct, VideoEnsemble

# Define your video generator once
def my_video_gen(prompt, **kwargs):
    # Your text-to-video API call
    return video_path

# Simple prediction with prompt enhancement
# Just use "prompt -> video_path" - ViDSPy handles the internal complexity
predictor = VideoPredict(
    "prompt -> video_path",
    video_generator=my_video_gen
)

# Chain-of-thought reasoning (analyzes scene, motion, style before generating)
# The simple "prompt -> video" signature is all you need
cot = VideoChainOfThought(
    "prompt -> video",
    video_generator=my_video_gen
)

# ReAct-style iterative refinement (generates, evaluates, refines)
# Same simple signature - internal reasoning is handled automatically
react = VideoReAct(
    "prompt -> video",
    video_generator=my_video_gen,
    max_iterations=3
)

# Ensemble multiple approaches (tries different strategies, picks best)
ensemble = VideoEnsemble([
    VideoPredict(video_generator=my_video_gen),
    VideoChainOfThought(video_generator=my_video_gen),
], selection_metric=composite_reward)

🛠️ Setup VBench Models

# Via CLI
vidspy setup

# Via Python
from vidspy import setup_vbench_models
setup_vbench_models()  # Downloads to ~/.cache/vbench

📝 Full Example

import os
import replicate
from vidspy import (
    ViDSPy,
    VideoChainOfThought,
    Example,
    composite_reward,
    VBenchMetric,
)

# Set API keys (or pass directly to ViDSPy)
os.environ["OPENROUTER_API_KEY"] = "your-vlm-api-key"
os.environ["OPENROUTER_OPTIMIZER_API_KEY"] = "your-optimizer-api-key"

# Define video generator (using Replicate as example)
def replicate_video_generator(prompt: str, **kwargs) -> str:
    """Generate video using Replicate API."""
    output = replicate.run(
        "stability-ai/stable-video-diffusion",
        input={"prompt": prompt}
    )

    # Download and save video
    video_path = f"outputs/{hash(prompt)}.mp4"
    # ... download logic ...
    return video_path

# Initialize ViDSPy with all available parameters
vidspy = ViDSPy(
    # VLM Provider Settings (corresponds to vlm: section in config)
    vlm_backend="openrouter",  # or "huggingface"
    vlm_model="google/gemini-2.5-flash",  # VLM for video analysis
    api_key="your-vlm-api-key",  # Or set OPENROUTER_API_KEY env var

    # Optimizer LLM Settings (corresponds to optimizer: section in config)
    optimizer_lm="openai/gpt-4o-mini",  # LLM for optimizers (MIPROv2, COPRO, GEPA)
    optimizer_api_key="your-optimizer-api-key",  # Or set OPENROUTER_OPTIMIZER_API_KEY env var

    # Cache Settings (corresponds to cache.dir in config)
    cache_dir="~/.cache/vidspy",  # Cache directory for models
    # Note: VBench models cache (cache.vbench_models) is set via setup_vbench_models()

    # Hardware Settings (corresponds to hardware: section in config)
    device="auto",  # "cuda", "cpu", or "auto"
    # Note: hardware.dtype from config is used internally by HuggingFace VLM

    # Config File (optional)
    # config_path="vidspy_config.yaml",  # Load all settings from config file
)

# Prepare training data (videos you've already generated)
trainset = [
    Example(
        prompt="a person walking through a forest",
        video_path="data/walk_forest.mp4"
    ),
    Example(
        prompt="a car driving on a highway",
        video_path="data/car_highway.mp4"
    ),
    # ... more examples
]

# Split for validation
valset = trainset[-2:]
trainset = trainset[:-2]

# Create module with your video generator
module = VideoChainOfThought(
    "prompt -> video",
    video_generator=replicate_video_generator
)

# Optional: Create custom metric with specific settings
# (Corresponds to metrics: section in config file)
custom_metric = VBenchMetric(
    quality_weight=0.6,  # Corresponds to metrics.quality_weight in config
    alignment_weight=0.4,  # Corresponds to metrics.alignment_weight in config
    quality_metrics=[  # Corresponds to metrics.quality_metrics in config
        "subject_consistency",
        "motion_smoothness",
        "temporal_flickering",
        "human_anatomy",
        "aesthetic_quality",
        "imaging_quality"
    ],
    alignment_metrics=[  # Corresponds to metrics.alignment_metrics in config
        "object_class",
        "human_action",
        "spatial_relationship",
        "overall_consistency"
    ]
)
# Note: Target thresholds (targets: section in config) are for reference/documentation

# Optimize prompting strategy with optimization settings
# (These correspond to the optimization: section in config file)
optimized = vidspy.optimize(
    module,
    trainset,
    valset=valset,
    metric=custom_metric,  # Or use composite_reward for defaults (60% quality, 40% alignment)
    optimizer="mipro_v2",  # Corresponds to optimization.default_optimizer in config
    num_candidates=10,
    max_bootstrapped_demos=4,  # Corresponds to optimization.max_bootstrapped_demos in config
    max_labeled_demos=4,  # Corresponds to optimization.max_labeled_demos in config
)

# Evaluate on test set
testset = [Example(prompt="a boat on a lake", video_path="data/boat.mp4")]
results = vidspy.evaluate(optimized, testset)

print(f"Mean Score: {results['mean_score']:.4f}")
print(f"Quality: {results['details'][0].get('quality_score', 'N/A')}")
print(f"Alignment: {results['details'][0].get('alignment_score', 'N/A')}")

# Generate new videos with optimized prompts
result = optimized("a butterfly landing on a flower")
print(f"Generated: {result.video_path}")
print(f"Enhanced prompt: {result.enhanced_prompt}")

📖 CLI Reference

# Show help
vidspy --help

# Setup VBench models
vidspy setup
vidspy setup --cache-dir /path/to/cache

# Check dependencies
vidspy setup --check-only

# Optimize a module
vidspy optimize trainset.json --optimizer mipro_v2 --output optimized_model

# Evaluate a module
vidspy evaluate testset.json --module optimized_model --output results.json

# Show information
vidspy info

🏗️ Project Structure

vidspy/
├── pyproject.toml              # Package configuration
├── README.md                   # This file
├── .env.example                # Environment variables template
├── vidspy_config.yaml.example  # Configuration file template
├── vidspy/
│   ├── __init__.py             # Main exports
│   ├── core.py                 # ViDSPy main class, Example
│   ├── signatures.py           # VideoSignature, etc.
│   ├── modules.py              # VideoPredict, VideoChainOfThought
│   ├── optimizers.py           # VidBootstrapFewShot, VidMIPROv2, etc.
│   ├── metrics.py              # VBench wrappers, composite_reward
│   ├── providers.py            # OpenRouterVLM, HuggingFaceVLM
│   ├── setup.py                # Setup utilities
│   └── cli.py                  # Command-line interface
├── examples/
│   └── basic_usage.py
└── tests/
    └── test_basic.py

🔬 Target Thresholds

For production-quality videos, aim for:

Human Anatomy: ≥ 0.85
Text-Video Alignment: ≥ 0.80

📚 References

DSPy - Declarative Self-improving Python
VBench - Video generation benchmark
OpenRouter - Unified AI API

📄 License

MIT License - see LICENSE for details.

🤝 Contributing

Contributions welcome! Please read our contributing guide first.

⭐ Star History

If you find ViDSPy useful, please consider giving it a star!

Name		Name	Last commit message	Last commit date
Latest commit History 9 Commits
.github/workflows		.github/workflows
examples		examples
scripts		scripts
tests		tests
vidspy		vidspy
.env.example		.env.example
.gitignore		.gitignore
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
README.md		README.md
pyproject.toml		pyproject.toml
vidspy_config.yaml.example		vidspy_config.yaml.example

Folders and files

Latest commit

History

Repository files navigation