`ClientTools` does not invoke registered client functions

[prahaladd]

Environment Details

SDK Version: 2.16.0.
OS: MacOS Sequoia 15.6.1

Background

I am developing an application that performs voice based web navigation. This application is structured as follows:

• A simple agent that is responsible for bootstrapping the program. The start up of the program involves creating an stdio MCP server as shown in the MCP SDK Example. It also creates the ClientTools as shown in the ElevenLabs Documentation here.
• A Chrome browser is automated using the Browser MCP server. Hence once the server starts up there would be a Chrome browser extension that would connect to the Browser MCP server (possibly using CDP, but that detail is not relevant).
• The conversational agent starts up and I would instruct it to perform some navigation task. For example, I have registered the tool navigate_to_url within the agent that I have configured in the Eleven Labs Dashboard

Expected results

• The browser navigation is executed when requested by the user during the conversation with the agent.

Actual result

• The Client Tools are not invoked as expected. I get the error cannot schedule new futures after interpreter shutdown when I look at the transcript in the Agent Console.

Debugging done

Here is a simplistic test implementation being done for the integration. To eliminate the errors from the Browser MCP server, I have simply got rid of the actual browser interaction and only performed some elementary logging. Even this is not working as expected

#!/usr/bin/env python3
"""
Absolute minimal test - just see if ClientTools calls ANY function
"""

import os
import sys
import asyncio
import logging
from typing import Dict, Any

# Set up logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

from elevenlabs.client import ElevenLabs
from elevenlabs.conversational_ai.conversation import Conversation, ClientTools
from elevenlabs.conversational_ai.default_audio_interface import DefaultAudioInterface

def navigate_to_url(parameters: Dict[str, Any]):
    """Dummy navigate_to_url tool - just prints a message"""
    print("🎯 HELLO URL - navigate_to_url WAS CALLED!")
    logger.info("🎯 HELLO URL - navigate_to_url WAS CALLED!")
    print(f"Parameters received: {parameters}")
    url = parameters.get('url', 'unknown') if isinstance(parameters, dict) else str(parameters)
    print(f"🌐 Would navigate to: {url}")
    return f"Successfully navigated to {url}"

def main():
    print("🧪 Minimal ClientTools Test")
    
    # Environment 
    api_key = os.environ.get('ELEVENLABS_API_KEY', 'sk-test-123')
    agent_id = os.environ.get('ELEVENLABS_AGENT_ID', 'test-agent')
    
    try:
        # Create ElevenLabs client
        client = ElevenLabs(api_key=api_key)
        
        # Create ClientTools 
        client_tools = ClientTools()
        
        # Register navigate_to_url tool (this should match what ElevenLabs agent expects)
        client_tools.register("navigate_to_url", navigate_to_url)
        logger.info("✅ navigate_to_url tool registered")
        
        # NOTE: Official docs don't call client_tools.start() - removing it
        # client_tools.start()
        # logger.info("✅ ClientTools started")
        
        # Test if we can call the tool directly
        print("🧪 Testing direct tool call...")
        try:
            # Try to call the tool directly to see if it's registered properly
            test_result = navigate_to_url({"url": "https://test.com"})
            print(f"✅ Direct call result: {test_result}")
        except Exception as e:
            print(f"❌ Direct call failed: {e}")
        
        # Check if ClientTools has any way to list or call tools
        print(f"🔍 ClientTools type: {type(client_tools)}")
        print(f"🔍 ClientTools attributes: {[attr for attr in dir(client_tools) if not attr.startswith('_')]}")
        
        # Create conversation
        conversation = Conversation(
            client=client,
            agent_id=agent_id,
            requires_auth=False,
            audio_interface=DefaultAudioInterface(),
            callback_agent_response=lambda response: print(f"Agent: {response}"),
            callback_user_transcript=lambda transcript: print(f"User: {transcript}"),
            client_tools=client_tools,  # Try passing ClientTools explicitly
        )
        
        print("🎙️ Say 'navigate to google.com' to test if ClientTools calls our function")
        print("⚠️  NOTE: Tools might need to be configured in ElevenLabs dashboard first!")
        print("Press Ctrl+C to stop")
        
        # Start conversation
        conversation.start_session()
        
    except KeyboardInterrupt:
        print("\n🛑 Stopped")
    except Exception as e:
        print(f"❌ Error: {e}")
        import traceback
        traceback.print_exc()
    finally:
        try:
            # Official docs don't show client_tools.stop() either
            # if 'client_tools' in locals():
            #     client_tools.stop()
            pass
        except:
            pass

if __name__ == "__main__":
    main()

Here is the conversation detail

INFO:__main__:✅ navigate_to_url tool registered
🧪 Testing direct tool call...
🎯 HELLO URL - navigate_to_url WAS CALLED!
INFO:__main__:🎯 HELLO URL - navigate_to_url WAS CALLED!
Parameters received: {'url': 'https://test.com'}
🌐 Would navigate to: https://test.com
✅ Direct call result: Successfully navigated to https://test.com
🔍 ClientTools type: <class 'elevenlabs.conversational_ai.conversation.ClientTools'>
🔍 ClientTools attributes: ['execute_tool', 'handle', 'lock', 'register', 'start', 'stop', 'thread_pool', 'tools']
🎙️ Say 'navigate to google.com' to test if ClientTools calls our function
⚠️  NOTE: Tools might need to be configured in ElevenLabs dashboard first!
Press Ctrl+C to stop
Agent: Hello and greetings of the day!!
User: Hello and greetings of the day.
Agent: Hello! How can I assist you today?
User: Hello, how can I assist you today?
Agent: Hello! I'm here to help you. What would you like to know or do today?
^CTraceback (most recent call last):
  File "/Users/prahaladd/.pyenv/versions/3.13.3/lib/python3.13/threading.py", line 1540, in _shutdown
    _thread_shutdown()
KeyboardInterrupt: 
(elevenlabs-voice-toolkit-py3.13) prahaladd@Prahalads-MacBook-Pro elevenlabs-voice-toolkit % source .venv/bin/activate && python -m src.elevenlabs_voice_toolkit.minimal_test
🧪 Minimal ClientTools Test
INFO:__main__:✅ navigate_to_url tool registered
🧪 Testing direct tool call...
🎯 HELLO URL - navigate_to_url WAS CALLED!
INFO:__main__:🎯 HELLO URL - navigate_to_url WAS CALLED!
Parameters received: {'url': 'https://test.com'}
🌐 Would navigate to: https://test.com
✅ Direct call result: Successfully navigated to https://test.com
🔍 ClientTools type: <class 'elevenlabs.conversational_ai.conversation.ClientTools'>
🔍 ClientTools attributes: ['execute_tool', 'handle', 'lock', 'register', 'start', 'stop', 'thread_pool', 'tools']
🎙️ Say 'navigate to google.com' to test if ClientTools calls our function
⚠️  NOTE: Tools might need to be configured in ElevenLabs dashboard first!
Press Ctrl+C to stop
Agent: Hello and greetings of the day!!
User: Greetings of the day.
Agent: Thank you! How can I assist you today?
User: Thank you. How can I assist you today? Navigate to Microsoft.com.
Agent: Navigating to Microsoft's website now. Please hold on for a moment.
User: Navigating to Microsoft's website now. Please hold on for a moment.
Agent: It seems there was an issue with navigating to the website. Could you please try again or let me know if there's anything else I can assist you with?
User: It seems there was an issue with navigating to the website, could you please try again or let me know if there's anything else I can assist you with?
Agent: Certainly, let me...
User: Certainly, let me try navigating to Microsoft's website again for you.
Agent: Attempting to navigate to Microsoft's website once more. Please hold on.
User: Attempting to navigate to Microsoft's website once more, please hold on.
Agent: It seems there is a persistent issue with navigating to the website. Is there anything else I can assist you with?
User: It seems there is a persistent issue with navigating to the website.
Agent: I apologize for the inconvenience. If there's anything else you'd like to explore or need assistance with, please let me know!
^CTraceback (most recent call last):
  File "/Users/prahaladd/.pyenv/versions/3.13.3/lib/python3.13/threading.py", line 1540, in _shutdown
    _thread_shutdown()
KeyboardInterrupt: 
zsh: trace trap  python -m src.elevenlabs_voice_toolkit.minimal_test

Tool Configuration

{
  "type": "client",
  "name": "navigate_to_url",
  "description": "Navigate to the URL specified in the console or dictated by the user. The user can provide a full URL or can provide just the name of the website in the conversation. for e.g. if the user simply states the domain or site microsoft.com in the conversation, the intent is actually to navigate to `https://www.microsoft.com`.  \nThe output of the tool call contains the page structure. Use the page structure for  further tasks such as page content analysis or for deciding the parameters for invoking the next tool in sequence if any",
  "disable_interruptions": false,
  "force_pre_tool_speech": "auto",
  "assignments": [],
  "expects_response": true,
  "response_timeout_secs": 50,
  "parameters": [
    {
      "id": "parameters",
      "type": "object",
      "description": "The parameters contains the parameters required for the tool. This would typically contain the URL to which the navigation should be done. The URL is present as a value of the property named `url` which is a child of the property named `parameters` which is a JSON object.",
      "properties": [
        {
          "id": "url",
          "type": "string",
          "value_type": "llm_prompt",
          "description": "The url to which the navigation should be performed as specified by the user within the console or as a dialog input",
          "dynamic_variable": "",
          "constant_value": "",
          "enum": null,
          "required": true
        }
      ],
      "required": true,
      "value_type": "llm_prompt"
    }
  ],
  "dynamic_variables": {
    "dynamic_variable_placeholders": {}
  }
}

[kraenhansen]

Thanks for the thorough report, including the code, tool config, and conversation logs! The SDK has changed significantly since v2.16.0 (now v2.45.0). A related issue (#647) tracking ClientTools behavior is still open. If you're still seeing this with the latest version, please comment here or there or reopen this one.