feat: add multi-modal attachment propagation to all worker agents thr…

[bittoby]

Enable Image Analysis for All Worker Agents

Problem

Only Multi-Modal Agent could analyze images. Worker agents (Developer, Browser, Document) couldn't see or process image attachments, causing tasks to fail.

Solution

• Added ImageAnalysisToolkit to Developer, Browser, and Document agents
• Registered toolkits with toolkits_to_register_agent parameter
• Updated system prompts with explicit image analysis instructions

Changes

Agent Factories:

• backend/app/agent/factory/developer.py
• backend/app/agent/factory/browser.py
• backend/app/agent/factory/document.py

System Prompts:

• backend/app/agent/prompt.py

Impact

All worker agents can now:

• Analyze screenshots and images
• Extract text from images
• Answer questions about visual content
• Process images alongside text in complex tasks

Testing

✅ Developer Agent: Extracts code from screenshots
✅ Browser Agent: Analyzes webpage screenshots
✅ Document Agent: Ready for document image analysis

Type

• Feature
• Bug Fix

Closes #956

[bittoby]

@Wendong-Fan Could you please review this PR? thanks

[Wendong-Fan]

@Wendong-Fan Could you please review this PR? thanks

thanks @bittoby 's contribution! could @nitpicker55555 and @Zephyroam help checking this?

[bittoby]

@Zephyroam @nitpicker55555 I would appreciate your feedback. please review the PR! thanks

[nitpicker55555]

Obviously, you didn’t test your code. You didn’t register the ImageAnalysisToolkit for the agent—so how is the agent supposed to gain image-reading capability?

Additionally, this design is overly complicated. Why not simply modify the prompt of the decompose agent so that, when breaking down the task, it passes the image location to the corresponding agent? That way, we would only need to adjust the prompt and register the ImageAnalysisToolkit for the agent.

[bittoby]

@nitpicker55555 Obviously, after I tested it correctly I pushed the PR. When I attach a test image and input “As a developer, analyze this screenshot and write code” as prompt, then the developer agent generates HTML that matches the screenshot.

[bittoby]

Obviously, you didn’t test your code. You didn’t register the ImageAnalysisToolkit for the agent—so how is the agent supposed to gain image-reading capability?

Additionally, this design is overly complicated. Why not simply modify the prompt of the decompose agent so that, when breaking down the task, it passes the image location to the corresponding agent? That way, we would only need to adjust the prompt and register the ImageAnalysisToolkit for the agent.

you're right - I haven't added ImageAnalysisToolkit to the worker agents yet. The current implementation works because it relies on the LLM's native vision capability (like gpt-5), but I agree we should add the toolkit for explicit tool calls and non-vision model support. Would you prefer I complete the current approach by adding the toolkit, or switch to your suggestion of modifying the decompose prompt to selectively pass images?

[nitpicker55555]

Obviously, you didn’t test your code. You didn’t register the ImageAnalysisToolkit for the agent—so how is the agent supposed to gain image-reading capability?
Additionally, this design is overly complicated. Why not simply modify the prompt of the decompose agent so that, when breaking down the task, it passes the image location to the corresponding agent? That way, we would only need to adjust the prompt and register the ImageAnalysisToolkit for the agent.

you're right - I haven't added ImageAnalysisToolkit to the worker agents yet. The current implementation works because it relies on the LLM's native vision capability (like gpt-5), but I agree we should add the toolkit for explicit tool calls and non-vision model support. Would you prefer I complete the current approach by adding the toolkit, or switch to your suggestion of modifying the decompose prompt to selectively pass images?

@bittoby Thank you for the explanation, but I still don’t understand: without providing a dedicated tool, how can the model read an image from a passed-in image URL instead of a base64-encoded input? Could you point out which part of your PR implements this functionality?

[bittoby]

Right now it only passes the image file paths along as additional_info. The current setup depends on the model’s built-in vision (which needs base64 data or a URL in the actual message). My PR doesn’t do that conversion or attach the images to the message - it just sends the paths as metadata. I will update PR. I misunderstood. I am clear now.

For this to really work, we need to either:

1. add ImageAnalysisToolkit to the worker agents, or
2. change how we build the LLM request so it includes the images as base64.

Which option do you recommend?

[nitpicker55555]

My idea is to tweak ImageAnalysisToolkit so it supports taking an image path as input and returning the actual image data back to the agent (right now ImageAnalysisToolkit only returns an image description).

The benefits are: the change is relatively small, we’d only need to update the decompose agent’s prompt to pass along the image path. It would also allow the agent to read other image files, not just images uploaded by the user in the prompt, and it could support users providing an image path instead of the image itself.

I also looked into how Claude Code does image reading, and it works in a similar way, via a Read tool.

What do you think @Wendong-Fan @bittoby

[bittoby]

@nitpicker55555 Thanks for your explanation!
Your idea (extending ImageAnalysisToolkit) is a solid quick win, but I see some limitations:

Concerns:

• Mixed responsibility: one toolkit ends up doing both “read” and “analyze”
• Less reusable: e.g., Document Agent might only need to read images for embeddings, but it would still have to pull in the whole analysis toolkit
• Heavier deps everywhere: any agent that needs to read an image would also load the analysis dependencies

My approach (like Claude Code): make a small, separate tool:

• read_image(path) -> base64
• Keep ImageAnalysisToolkit focused on analysis only
• More flexible: agents can read images for embeddings, uploads, etc.

But it requires more code, a bit more wiring

I agree with your approach for now since it's faster to ship. We can refactor to a separate ReadImageTool later if we see reusability issues.

@Wendong-Fan what's your preference?

[nitpicker55555]

@nitpicker55555 Thanks for your explanation!

Your idea (extending ImageAnalysisToolkit) is a solid quick win, but I see some limitations:

Concerns:

• Mixed responsibility: one toolkit ends up doing both “read” and “analyze”

• Less reusable: e.g., Document Agent might only need to read images for embeddings, but it would still have to pull in the whole analysis toolkit

• Heavier deps everywhere: any agent that needs to read an image would also load the analysis dependencies

My approach (like Claude Code): make a small, separate tool:

• read_image(path) -> base64

• Keep ImageAnalysisToolkit focused on analysis only

• More flexible: agents can read images for embeddings, uploads, etc.

But it requires more code, a bit more wiring

I agree with your approach for now since it's faster to ship. We can refactor to a separate ReadImageTool later if we see reusability issues.

@Wendong-Fan what's your preference?

@bittoby what do you mean document agent read image for embedding?

[bittoby]

I mean, here "embedding" = putting the image into a document not analyzing it.
For example:
User uploads screenshot.png and asks: "Create a PDF report with this screenshot",
Document Agent needs to read the png from disk , convert it to base64 data then insert it to pdf. It doesn't need analyze the png, describe or extract text etc

[nitpicker55555]

@bittoby It seems you haven’t reviewed this tool at all. I strongly recommend checking the codebase before continuing the discussion.
ImageAnalysisToolkit has only one dependency—Pillow—which is also required for simple image reading. The difference between this tool and the “read-only image” functionality we need is merely whether we leverage the built-in agent to generate a description. From the agent’s perspective, the only difference lies in the arguments passed in.
We could, of course, implement a separate tool as you suggested, but the supposed benefit of being “more flexible: agents can read images for embeddings, uploads, etc.” does not really exist.

[bittoby]

Okay. @nitpicker55555 I’ll update the PR to follow your approach.

[bittoby]

@nitpicker55555 I updated pr to follow your feedback. please review again

[bittoby]

@Wendong-Fan @nitpicker55555 I would appreciate your feedback.

[bittoby]

test.webm

@nitpicker55555 pls check this test video. My changes works well

[nitpicker55555]

I can see the problem now, develop agent alreday has Screenshot toolkit for image reading, we can use this tool directly, check this bittoby#1, and test browser agent/document agent to see if they can read image. You can achieve this by modifing the generated task plan to: "use browser agent/document agent to check the xxx image file content"

[bittoby]

browser.webm

The browser agent works, too.

[bittoby]

@nitpicker55555 I merged your changes. Now all agents work well for reading function. i think it's okay to merge this pr. @Wendong-Fan

[nitpicker55555]

You do not need to modify this file anymore since it is not used.

Others LGTM

[bittoby]

@nitpicker55555 I reverted image_analysis_toolkit since it is not used. it's ready to merge.
Thank you

[nitpicker55555]

Thanks @bittoby ! @Wendong-Fan can you take a look for this?

[bittoby]

@Wendong-Fan would you review this pr and merge that if it has no problem?
Thank you

[Zephyroam]

Do we need to update the agent skills here?

[Zephyroam]

I tested several times. All of them failed. Could you upload a successful demo?

Logs here
eigent-0.0.83-darwin-arm64-1771326253031.log

[bittoby]

browser.mp4

developer.mp4

@Zephyroam It works well without any errors. please check this demo.

[bittoby]

@Wendong-Fan please review this PR and would be appreciate to merge if it has no problem

[bittoby]

@Wendong-Fan @bytecii would appreciate feedback.
thank you

[Wendong-Fan]

thanks @bittoby 's PR and sorry for the late review, one issue to address is that ScreenshotToolkit is now used in multiple agents, this can misattribute toolkit activation/deactivation events in workflow logs/UI, so we need to add agent_name when init ScreenshotToolkit for those agent, i will fix this in another commit

cc @nitpicker55555 @Zephyroam