Skip to content

[Feature] Add reference-only preprocessor modes (reference_adain, reference_adain+attn) #1295

@SolicTous

Description

@SolicTous

Feature Summary

Add support for reference-only preprocessor modes to control image generation using a reference image

Detailed Description

The sd-webui-controlnet project has implemented reference-only preprocessing modes (reference_adain, reference_adain+attn) that allow controlling the output generation using a reference image.

Key features:

  • Transfers style and visual characteristics from a reference image to the generated output
  • Works without requiring additional ControlNet model files (uses the base SD model's attention mechanisms)
  • Can be combined with traditional ControlNet models (canny, depth, pose) for more complex workflows

How it works:
These modes modify the self-attention layers during generation by injecting features from the reference image:

  • reference_only: Direct substitution of K/V matrices from reference image
  • reference_adain: Applies Adaptive Instance Normalization before attention injection
  • reference_adain+attn: Hybrid approach combining both methods (recommended for style transfer)

References:

This feature would enable powerful style transfer and reference-based generation in stable-diffusion.cpp without additional model loading.

Alternatives you considered

No response

Additional context

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions