How to add attachments ?

[jide]

How can we add attachments to the chat ?

I tried usings parts in append, but the chat does not take files into account :

                    append({
                      role: "user",
                      content: "What is in the attached file ?", // needed ?
                      parts: [
                        {
                          type: "text",
                          text: "What is in the attached file ?",
                        },
                        {
                          type: "file",
                          mimeType: "image/png",
                          data: "https://www.ffkarate.fr/wp-content/uploads/2024/12/FFK_OPENPARIS2025_KV_40X60_HD-1377x2048.png",
                        },
                      ],
                    });

[threepointone]

I haven't added support for attachments yet, will do so this week

[jide]

Ah ok thanks for the info !

[jide]

Hey ! Just wanted to know if you think you will look at this soon ? (sorry to be that guy)

[threepointone]

Not today, but I'm open to either PRs or specifics (low level details) of how and what to implement.

[jide]

Not sure I 'm comfortable enough with workers/DOs to make a PR, but I can take time to think about this. Roughly I think it should be equivalent to how Vercel AI SDK handles this.

[halfstack-dev]

@jide I was able to to a temporary work-around using experimental_attachments:

frontend react:

const {
  messages: agentMessages,
  input: agentInput,
  handleInputChange: handleAgentInputChange,
  handleSubmit: handleAgentSubmit,
  addToolResult,
  clearHistory,
} = useAgentChat({
  agent,
  maxSteps: 5,
});

const fileInputRef = useRef<HTMLInputElement>(null);

....

<form
  onSubmit={(e) =>
    handleAgentSubmit(e, {
      experimental_attachments: fileInputRef.current?.files ?? [],
     })
  }
  className="p-3 bg-input-background absolute bottom-0 left-0 right-0 z-10 border-t border-neutral-300 dark:border-neutral-800"
>
  <input
    ref={fileInputRef}
    type={"file"}
    accept="image/*"
    className="rounded-full h-10 w-10 flex-shrink-0"
  />
</form>

This will cause agent sdk, and vercel AI sdk, to pass the file to the LLM. (tested using openai provider).
You can also use the experimental_attachments to render the image, because it will still be present in the agent messages on the frontend

[riseandignite]

@jide you can attach files using experimental_attachments that @JonathanVelkeneers mentioned above.

Created a PR that enables it in agents-starter project.

[jide]

Wow so cool ! Looks so simple

[whoiskatrin]

@jide you can just do it by default using ai chat agent class