[API Proposal]: Abstract ToolResult Class.

[rogerbarreto]

Background and motivation

Currently there's no abstract way to represent different outputs from tools that aren't fully recognized and have specific abstractions, like Anthropic WebSearch and WebFetch although we have different tools already that share a same abstraction representation.

• ImageGenerationToolResultContent.Outputs
• CodeInterpreterToolResultContent.Outputs
• McpServerToolResultContent.Output
• FunctionResultContent(Result + Exception)

The proposal is to have a AIToolResultContent abstraction that we can represent for tool results that may have multiple outputs without having to resort to the AIContent.RawRepresentation for an unrecognized tool result.

In other words.

WebFetchToolResult even not existing as a concrete class in Abstractions, could be better represented as a AIToolResultContent with multiple Outputs as AIContents before any specialization like WebSearchResultContent : AItoolResultContent is added to the library.

Benefits

This middle term abstraction has the benefit of being able to represent unrecognized tool results via the IList<AIContent>? Outputs property. Allowing callers to be able to detect failure/results from tools without necessarily needing to break glass.

API Proposal

/// <summary>
/// Represents an abstract tool call result by a hosted service.
/// </summary>
/// <remarks>
/// This content type represents any hosted AI service tool invocation result.
/// </remarks>
[Experimental("MEAI001")]
public class HostedToolResultContent : AIContent
{
    /// <summary>
    /// Initializes a new instance of the <see cref="HostedToolResultContent"/> class.
    /// </summary>
    public HostedToolResultContent ()
    {
    }

    /// <summary>
    /// Gets or sets the unique identifier of the image generation item.
    /// </summary>
    public virtual string? ToolId { get; set; }

    /// <summary>
    /// Gets or sets the resulting contents from the tool.
    /// </summary>
    /// <remarks>
    /// Content is typically <see cref="ErrorContent"/> for any tool error, <see cref="DataContent"/> for data contents, or <see cref="UriContent"/> for remotely hosted contents, or <see cref="TextContent"/> for text based contents.
    /// </remarks>
    public virtual IList<AIContent>? Outputs { get; set; }
}

Existing tools that has the IList<AIContent>? Outputs property can be updated to be specializations of this class.

API Usage

// Fancy the value
var message = await client.GetResponseAsync(...);

// Fetching WebSearchResult tools (there's no such tool result representation in abstractions).

for (var toolResultContent in message.Contents.OfType<HostedToolResultContent>()) 
{
    switch (toolResultContent) 
    { 
        McpServerToolResultContent: { break; // MCP }
        ImageGenerationToolResultContent: { break; // Image Result }
        CodeInterpreterToolResultContent: { break; // Code interpreter result }
        default: {
            // A singular logic can be used across all tools or just for non-identified tools
            
            // Identifying if any tool error happened.
            var error = toolResultContent.Outputs.FirstOrDefault(c => c is ErrorContent);

            // Generated binary data contents
            var generatedDataContents = toolResultContent.Outputs.OfType<DataContent>();

            // Generated text based contents
            var textContents = toolResultContent.Outputs.OfType<TextContent>();
        }
    }
}

Alternative Designs

No response

Risks

No response

[stephentoub]

How would web search results be represented for example? It's going to be a list of results, with title, excerpt, etc. How would you represent that without introducing a strongly-typed class for that data? How do you interpret the resulting data? I'm not against a base type here, but I'm missing how it actually helps.

[rogerbarreto]

How would web search results be represented for example? How would you represent that without introducing a strongly-typed class for that data?

If a tool have multiple results, non concrete abstractions can be represented as multiple different AIContent in Outputs results.

I'm not against a base type here, but I'm missing how it actually helps.

The benefit I see is being able to identify multiple permutations/results for a tool result in an abstract form, like detecting if a tool failed checking if any of them are a ErrorContent. Or scenarios where there was any tool result (regardless of the tool) in the message at all.

[stephentoub]

If a tool have multiple results, non concrete abstractions can be represented as multiple different AIContent in Outputs results.

Yes, but for web search for example, what would those be?

[rogerbarreto]

Yes, but for web search for example, what would those be?

// i.e: Anthorpic WebFetch

 case BetaContentBlock ct when ct.TryPickWebFetchToolResult(out var webFetchResult):
 {
     var webToolResult = new AIToolResult() { Outputs = [] };
     if (webFetchResult.Content.TryPickBetaWebFetchToolResultErrorBlock(out var webFetchErr))
     {
         webToolResult.Outputs.Add(new ErrorContent(null) { ErrorCode = webFetchErr.ErrorCode.Value().ToString() });
     }
     if (webFetchResult.Content.TryPickBetaWebFetchBlock(out var webFetchResultBlock))
     {
         webToolResult.Outputs.Add(new UriContent(webFetchResultBlock.URL) { RawRepresentation = webFetchResultBlock });
         if (webFetchResultBlock.Content.Source.TryPickBetaBase64PDF(out var pdfContent))
         {
             webToolResult.Outputs.Add(new DataContent($"data:;base64,{pdfContent.Data}", "application/pdf") { RawRepresentation = pdfContent });
         }

         if (webFetchResultBlock.Content.Source.TryPickBetaPlainText(out var plainText))
         {
             webToolResult.Outputs.Add(new DataContent($"data:;base64,{plainText.Data}", "text/plain") { RawRepresentation = plainText });
         }
     }

[kzu]

XAI represents hosted tool executions as just the {callid, function {name, args}} in the responses. So the actual output isn't present, but you can inspect the invoked hosted tool for display, for example.

See https://docs.x.ai/docs/grpc-reference#chat: xai_api.GetChatCompletionResponse.choices[].message.tool_calls[]

Perhaps a separate HostedFunctionCallContent that pretty much mirrors FunctionCallContent but is entirely ignored by the FunctionInvokingChatClient would be enough in that case?

[kzu]

Just came across the inconsistency between McpServerToolResultContent.Output and CodeInterpreterToolResultContent.Outputs. +1 for this unification.

[stephentoub]

Yes, but for web search for example, what would those be?

// i.e: Anthorpic WebFetch

case BetaContentBlock ct when ct.TryPickWebFetchToolResult(out var webFetchResult):
{
var webToolResult = new AIToolResult() { Outputs = [] };
if (webFetchResult.Content.TryPickBetaWebFetchToolResultErrorBlock(out var webFetchErr))
{
webToolResult.Outputs.Add(new ErrorContent(null) { ErrorCode = webFetchErr.ErrorCode.Value().ToString() });
}
if (webFetchResult.Content.TryPickBetaWebFetchBlock(out var webFetchResultBlock))
{
webToolResult.Outputs.Add(new UriContent(webFetchResultBlock.URL) { RawRepresentation = webFetchResultBlock });
if (webFetchResultBlock.Content.Source.TryPickBetaBase64PDF(out var pdfContent))
{
webToolResult.Outputs.Add(new DataContent($"data:;base64,{pdfContent.Data}", "application/pdf") { RawRepresentation = pdfContent });
}

     if (webFetchResultBlock.Content.Source.TryPickBetaPlainText(out var plainText))
     {
         webToolResult.Outputs.Add(new DataContent($"data:;base64,{plainText.Data}", "text/plain") { RawRepresentation = plainText });
     }
 }

Each web search result can contain various pieces of information, e.g. anthropic's API gives you back url, title, cited text, etc. for each result. I don't understand from your example where that data would be stored. Presumably it would need a dedicated type to store the relevant information in a user-friendly manner?

[stephentoub]

Just came across the inconsistency between McpServerToolResultContent.Output and CodeInterpreterToolResultContent.Outputs. +1 for this unification.

That is temporary; we added CodeInterpreterToolResultContent.Outputs after McpServerToolResultContent.Output and decided the right answer is "Outputs", but didn't want to cause unnecessary churn on the MCP types, since they'll be getting additional changes.
cc: @jozkee

[stephentoub]

cc: @jozkee

[jozkee]

McpServerToolResultContent.Output

This is no longer a concern with #7245 since it breaks the original alignment with CodeInterpreterToolResultContent in favor of FunctionResultContent.

[jozkee]

One concern I have with Outputs and Results is that there isn’t a clear way to indicate that they represent an error, particularly for FunctionResultContent.

[stephentoub]

One concern I have with Outputs and Results is that there isn’t a clear way to indicate that they represent an error, particularly for FunctionResultContent.

ErrorContent is what we currently use for that.