Skip to content

Commit 964160d

Browse files
committed
feat(MTMDChatHandler): support audio inputs and fix interleaved media ordering
- Refactored `CHAT_FORMAT` to use a single loop for `message.content`, preserving the exact chronological order of interleaved text, images, and audio. - Added template routing for `audio_url`. - Added template routing for OpenAI's `input_audio` format, properly formatting it as a Data URI.
1 parent 5bd0870 commit 964160d

1 file changed

Lines changed: 9 additions & 4 deletions

File tree

llama_cpp/llama_chat_format.py

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -2803,10 +2803,15 @@ class MTMDChatHandler:
28032803
"{% for content in message.content %}"
28042804
"{% if content.type == 'image_url' %}"
28052805
"{{ content.image_url if content.image_url is string else content.image_url.url }}"
2806-
"{% endif %}"
2807-
"{% endfor %}"
2808-
"{% for content in message.content %}"
2809-
"{% if content.type == 'text' %}"
2806+
"{% elif content.type == 'audio_url' %}"
2807+
"{{ content.audio_url if content.audio_url is string else content.audio_url.url }}"
2808+
"{% elif content.type == 'input_audio' %}"
2809+
"{% if content.input_audio is string %}"
2810+
"{{ content.input_audio }}"
2811+
"{% else %}"
2812+
"data:audio/{{ content.input_audio.format }};base64,{{ content.input_audio.data }}"
2813+
"{% endif %}"
2814+
"{% elif content.type == 'text' %}"
28102815
"{{ content.text }}"
28112816
"{% endif %}"
28122817
"{% endfor %}"

0 commit comments

Comments
 (0)