Program Atoms
-The Agentive Operating System for Generalist Robotics
++
The Agentive Operating System for Generalist Robotics
-[](https://discord.gg/8m6HMArf) +[](https://discord.gg/dimos) [](https://github.com/dimensionalOS/dimos/stargazers) [](https://github.com/dimensionalOS/dimos/fork) [](https://github.com/dimensionalOS/dimos/graphs/contributors) -  - + [](https://www.docker.com/) -
- Features • - Installation • - Documentation • - Development • - Contributing -
- -
+ 
+ 
-Option B: Editable install with uv
-```sh
-uv venv && . .venv/bin/activate
-uv pip install -e '.[base,dev]'
-```
+# Development
-For system deps, Nix setups, and testing, see `/docs/development/README.md`.
+## Develop on DimOS
-### Monitoring & Debugging
+```sh
+export GIT_LFS_SKIP_SMUDGE=1
+git clone -b dev https://github.com/dimensionalOS/dimos.git
+cd dimos
-DimOS comes with a number of monitoring tools:
-- Run `lcmspy` to see how fast messages are being published on streams.
-- Run `skillspy` to see how skills are being called, how long they are running, which are active, etc.
-- Run `agentspy` to see the agent's status over time.
-- If you suspect there is a bug within DimOS itself, you can enable extreme logging by prefixing the dimos command with `DIMOS_LOG_LEVEL=DEBUG RERUN_SAVE=1 `. Ex: `DIMOS_LOG_LEVEL=DEBUG RERUN_SAVE=1 dimos --replay run unitree-go2`
+uv sync --all-extras --no-extra dds
+
+# Run fast test suite
+uv run pytest dimos
+```
+## Multi Language Support
-# Documentation
+Python is our glue and prototyping language, but we support many languages via LCM interop.
-Concepts:
-- [Modules](/docs/concepts/modules.md): The building blocks of DimOS, modules run in parallel and are singleton python classes.
-- [Streams](/docs/api/sensor_streams/index.md): How modules communicate, a Pub / Sub system.
-- [Blueprints](/dimos/core/README_BLUEPRINTS.md): a way to group modules together and define their connections to each other.
-- [RPC](/dimos/core/README_BLUEPRINTS.md#calling-the-methods-of-other-modules): how one module can call a method on another module (arguments get serialized to JSON-like binary data).
-- [Skills](/dimos/core/README_BLUEPRINTS.md#defining-skills): An RPC function, except it can be called by an AI agent (a tool for an AI).
+Check our language interop examples:
+- [C++](examples/language-interop/cpp/)
+- [Lua](examples/language-interop/lua/)
+- [TypeScript](examples/language-interop/ts/)
-## Contributing
+## ROS interop
-We welcome contributions! See our [Bounty List](https://docs.google.com/spreadsheets/d/1tzYTPvhO7Lou21cU6avSWTQOhACl5H8trSvhtYtsk8U/edit?usp=sharing) for open requests for contributions. If you would like to suggest a feature or sponsor a bounty, open an issue.
+For researchers, we can talk to ROS directly via [ROS Transports](docs/usage/transports/index.md), or host dockerized ROS deployments as first-class DimOS modules, allowing you easy installation and portability
diff --git a/README_installation.md b/README_installation.md
deleted file mode 100644
index dc9117798f..0000000000
--- a/README_installation.md
+++ /dev/null
@@ -1,136 +0,0 @@
-# DimOS
-
-## Installation
-
-Clone the repo:
-
-```bash
-git clone -b main --single-branch https://github.com/dimensionalOS/dimos.git
-cd dimos
-```
-
-### System dependencies
-
-Tested on Ubuntu 22.04/24.04.
-
-```bash
-sudo apt update
-sudo apt install git-lfs python3-venv python3-pyaudio portaudio19-dev libturbojpeg0-dev
-```
-
-### Python dependencies
-
-Install `uv` by [following their instructions](https://docs.astral.sh/uv/getting-started/installation/) or just run:
-
-```bash
-curl -LsSf https://astral.sh/uv/install.sh | sh
-```
-
-Install Python dependencies:
-
-```bash
-uv sync
-```
-
-Depending on what you want to test you might want to install more optional dependencies as well (recommended):
-
-```bash
-uv sync --extra dev --extra cpu --extra sim --extra drone
-```
-
-### Install Foxglove Studio (robot visualization and control)
-
-> **Note:** This will be obsolete once we finish our migration to open source [Rerun](https://rerun.io/).
-
-Download and install [Foxglove Studio](https://foxglove.dev/download):
-
-```bash
-wget https://get.foxglove.dev/desktop/latest/foxglove-studio-latest-linux-amd64.deb
-sudo apt install ./foxglove-studio-*.deb
-```
-
-[Register an account](https://app.foxglove.dev/signup) to use it.
-
-Open Foxglove Studio:
-
-```bash
-foxglove-studio
-```
-
-To connect and load our dashboard:
-
-1. Click on "Open connection"
-2. In the popup window, leave the WebSocket URL as `ws://localhost:8765` and click "Open"
-3. In the top right, click on the "Default" dropdown, then "Import from file..."
-4. Navigate to the `dimos` repo and select `assets/foxglove_dashboards/unitree.json`
-
-### Test the install
-
-Run the Python tests:
-
-```bash
-uv run pytest dimos
-```
-
-They should all pass in about 3 minutes.
-
-### Test a robot replay
-
-Run the system by playing back recorded data from a robot (the replay data is automatically downloaded via Git LFS):
-
-```bash
-uv run dimos --replay run unitree-go2-basic
-```
-
-You can visualize the robot data in Foxglove Studio.
-
-### Run a simulation
-
-```bash
-uv run dimos --simulation run unitree-go2-basic
-```
-
-This will open a MuJoCo simulation window. You can also visualize data in Foxglove.
-
-If you want to also teleoperate the simulated robot run:
-
-```bash
-uv run dimos --simulation run unitree-go2-basic --extra-module keyboard_teleop
-```
-
-This will also open a Keyboard Teleop window. Focus on the window and use WASD to control the robot.
-
-### Command center
-
-You can also control the robot from the `command-center` extension to Foxglove.
-
-First, pull the LFS file:
-
-```bash
-git lfs pull --include="assets/dimensional.command-center-extension-0.0.1.foxe"
-```
-
-To install it, drag that file over the Foxglove Studio window. The extension will be installed automatically. Then, click on the "Add panel" icon on the top right and add "command-center".
-
-You can now click on the map to give it a travel goal, or click on "Start Keyboard Control" to teleoperate it.
-
-### Using `dimos` in your code
-
-If you want to use dimos in your own project (not the cloned repo), you can install it as a dependency:
-
-```bash
-uv add dimos
-```
-
-Note, a few dependencies do not have PyPI packages and need to be installed from their Git repositories. These are only required for specific features:
-
-- **CLIP** and **detectron2**: Required for the Detic open-vocabulary object detector
-- **contact_graspnet_pytorch**: Required for robotic grasp prediction
-
-You can install them with:
-
-```bash
-uv add git+https://github.com/openai/CLIP.git
-uv add git+https://github.com/dimensionalOS/contact_graspnet_pytorch.git
-uv add git+https://github.com/facebookresearch/detectron2.git
-```
diff --git a/assets/dimensional-dark.svg b/assets/dimensional-dark.svg
new file mode 100644
index 0000000000..95edaadc0e
--- /dev/null
+++ b/assets/dimensional-dark.svg
@@ -0,0 +1,23 @@
+
diff --git a/assets/dimensional-light.svg b/assets/dimensional-light.svg
new file mode 100644
index 0000000000..f0a107bd10
--- /dev/null
+++ b/assets/dimensional-light.svg
@@ -0,0 +1,23 @@
+
diff --git a/assets/dimensional-text.svg b/assets/dimensional-text.svg
new file mode 100644
index 0000000000..0cf32b3d19
--- /dev/null
+++ b/assets/dimensional-text.svg
@@ -0,0 +1,20 @@
+
diff --git a/assets/dimensional.svg b/assets/dimensional.svg
new file mode 100644
index 0000000000..f36a0dea40
--- /dev/null
+++ b/assets/dimensional.svg
@@ -0,0 +1,23 @@
+
diff --git a/assets/readme/agentic_control.gif b/assets/readme/agentic_control.gif
new file mode 100644
index 0000000000..f9f5970441
--- /dev/null
+++ b/assets/readme/agentic_control.gif
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:eb0411de5e5967be8773d5d95e692a6a5859f75bb400164451a3b383b1025fb4
+size 2416274
diff --git a/assets/readme/agents.png b/assets/readme/agents.png
new file mode 100644
index 0000000000..b05bee0b03
--- /dev/null
+++ b/assets/readme/agents.png
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a255d32f9a0ecff12d99dda9b8a51e0958ac282d7a0f814f93fd39261afaf84d
+size 477123
diff --git a/assets/readme/dimos_demo.gif b/assets/readme/dimos_demo.gif
new file mode 100644
index 0000000000..5a68bd72ac
--- /dev/null
+++ b/assets/readme/dimos_demo.gif
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fda7f7a859ce98002e0faef88fb2942f395e19995b36b585c48447ec5a9435ee
+size 24011189
diff --git a/assets/readme/lidar.gif b/assets/readme/lidar.gif
new file mode 100644
index 0000000000..8302c2957d
--- /dev/null
+++ b/assets/readme/lidar.gif
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d47badc970572aa7badf98c908490c8b86ea9f1cafbb18507cfdb5d08655cdfb
+size 5900150
diff --git a/assets/readme/lidar.png b/assets/readme/lidar.png
new file mode 100644
index 0000000000..1b499de10f
--- /dev/null
+++ b/assets/readme/lidar.png
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:65b1797fd9ac8edae5dce0691397b6aca2e975badfd58462ed8e20a4dace655e
+size 927067
diff --git a/assets/readme/navigation.gif b/assets/readme/navigation.gif
new file mode 100644
index 0000000000..1402b1e85a
--- /dev/null
+++ b/assets/readme/navigation.gif
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:64e7965f421916cdb71667a9ed99eab96c14c64bd195bd483628d1b9b9a4e95c
+size 4395592
diff --git a/assets/readme/navigation.png b/assets/readme/navigation.png
new file mode 100644
index 0000000000..16819a5007
--- /dev/null
+++ b/assets/readme/navigation.png
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:472cabca4b0d661658bf9ffbde78e636668e9ef6499dc38ea0f552557d735bd9
+size 617989
diff --git a/assets/readme/perception.png b/assets/readme/perception.png
new file mode 100644
index 0000000000..7ec15aabbf
--- /dev/null
+++ b/assets/readme/perception.png
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:48e4c61c1ec588d56d61a74fd9f0d9251eadc042e7a514fb1896826d52a32988
+size 797817
diff --git a/assets/readme/spacer.png b/assets/readme/spacer.png
new file mode 100644
index 0000000000..8745fc9687
--- /dev/null
+++ b/assets/readme/spacer.png
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4a16ec40112698cf02b9abd3d18c8db65ce40f48f2c61076b45de58695f16532
+size 66
diff --git a/assets/readme/spatial_memory.gif b/assets/readme/spatial_memory.gif
new file mode 100644
index 0000000000..070c65270b
--- /dev/null
+++ b/assets/readme/spatial_memory.gif
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:50b9cb7898ae8d238a088252fd96d2278a1be96a0dbb761839bc58c99c17f7a7
+size 4655580
diff --git a/base-requirements.txt b/base-requirements.txt
deleted file mode 100644
index 68b485fb9a..0000000000
--- a/base-requirements.txt
+++ /dev/null
@@ -1,2 +0,0 @@
-torch==2.0.1
-torchvision==0.15.2
diff --git a/bin/dev b/bin/dev
index 2ccebcb071..459696c018 100755
--- a/bin/dev
+++ b/bin/dev
@@ -1,26 +1,26 @@
#!/usr/bin/env bash
REPO_ROOT=$(git rev-parse --show-toplevel)
REL_PATH=$(realpath --relative-to="$REPO_ROOT" "$(pwd)")
-IMAGE="ghcr.io/dimensionalos/dev"
-IMAGE_NAME="$IMAGE:dev"
+IMAGE="ghcr.io/dimensionalos/ros-dev"
+IMAGE_TAG="$IMAGE:dev"
image_exists() {
- docker image inspect "${IMAGE_NAME}" &>/dev/null
+ docker image inspect "${IMAGE_TAG}" &>/dev/null
}
image_pull() {
- docker pull "$IMAGE_NAME"
+ docker pull "$IMAGE_TAG"
}
ensure_image_downloaded() {
if ! image_exists "$1"; then
- echo "Image ${IMAGE_NAME} not found. Pulling..."
+ echo "Image ${IMAGE_TAG} not found. Pulling..."
image_pull "$1"
fi
}
check_image_running() {
- if docker ps -q --filter "ancestor=${IMAGE_NAME}" | grep -q .; then
+ if docker ps -q --filter "ancestor=${IMAGE_TAG}" | grep -q .; then
return 0
else
return 1
@@ -28,16 +28,16 @@ check_image_running() {
}
stop_image() {
- if check_image_running ${IMAGE_NAME}; then
- echo "Stopping containers from image ${IMAGE_NAME}..."
- docker stop $(docker ps -q --filter "ancestor=${IMAGE_NAME}")
+ if check_image_running ${IMAGE_TAG}; then
+ echo "Stopping containers from image ${IMAGE_TAG}..."
+ docker stop $(docker ps -q --filter "ancestor=${IMAGE_TAG}")
else
- echo "No containers from image ${IMAGE_NAME} are running."
+ echo "No containers from image ${IMAGE_TAG} are running."
fi
}
-get_tag() {
+get_branch_tag() {
local branch_name
branch_name=$(git rev-parse --abbrev-ref HEAD)
@@ -58,16 +58,16 @@ get_tag() {
build_image() {
local image_tag
- image_tag=$(get_tag)
+ image_tag=$(get_branch_tag)
docker build \
--build-arg GIT_COMMIT=$(git rev-parse --short HEAD) \
--build-arg GIT_BRANCH=$(git rev-parse --abbrev-ref HEAD) \
- -t "ghcr.io/dimensionalos/dev:${image_tag}" -f docker/dev/Dockerfile .
+ -t "ghcr.io/dimensionalos/ros-dev:${image_tag}" -f docker/dev/Dockerfile .
}
remove_image() {
- local tag=$(get_tag)
+ local tag=$(get_branch_tag)
docker rm -f "dimos-dev-${tag}" 2>/dev/null || true
}
@@ -141,7 +141,7 @@ else
shift
;;
pull)
- docker pull ghcr.io/dimensionalos/dev:dev
+ docker pull "${IMAGE_TAG}"
shift
;;
*)
diff --git a/bin/doclinks b/bin/doclinks
new file mode 100755
index 0000000000..5dee1c69b0
--- /dev/null
+++ b/bin/doclinks
@@ -0,0 +1,3 @@
+#!/usr/bin/env bash
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+python "$REPO_ROOT/dimos/utils/docs/doclinks.py" "$@"
diff --git a/data/.lfs/models_graspgen.tar.gz b/data/.lfs/models_graspgen.tar.gz
new file mode 100644
index 0000000000..8321530922
--- /dev/null
+++ b/data/.lfs/models_graspgen.tar.gz
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:058ff764c043dccc516c1519a1e23207500c20a10c432c15eb5e30104477c0a4
+size 2117602984
diff --git a/data/.lfs/piper_description.tar.gz b/data/.lfs/piper_description.tar.gz
new file mode 100644
index 0000000000..3ab8ab227b
--- /dev/null
+++ b/data/.lfs/piper_description.tar.gz
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d4ce51d4ea15f29d80e69b0fff4a4d667f086e010329bb5c66980a881f1ee539
+size 3091511
diff --git a/data/.lfs/xarm7.tar.gz b/data/.lfs/xarm7.tar.gz
new file mode 100644
index 0000000000..8e2cfa368a
--- /dev/null
+++ b/data/.lfs/xarm7.tar.gz
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:47dd79f13845ae6a35368345b7443a9190c7584d548caddd9c3eae224442c6fc
+size 3280557
diff --git a/data/.lfs/xarm_description.tar.gz b/data/.lfs/xarm_description.tar.gz
new file mode 100644
index 0000000000..4cccd9ab25
--- /dev/null
+++ b/data/.lfs/xarm_description.tar.gz
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6e25f1ede8e4022f5053a61717191a2c338ea5af5b81e26bd2c880343aff1316
+size 12709222
diff --git a/dimos/agents/__init__.py b/dimos/agents/__init__.py
deleted file mode 100644
index 2bac584249..0000000000
--- a/dimos/agents/__init__.py
+++ /dev/null
@@ -1,27 +0,0 @@
-from langchain_core.messages import (
- AIMessage,
- HumanMessage,
- MessageLikeRepresentation,
- SystemMessage,
- ToolCall,
- ToolMessage,
-)
-
-from dimos.agents.agent import Agent, deploy
-from dimos.agents.spec import AgentSpec
-from dimos.agents.vlm_agent import VLMAgent
-from dimos.agents.vlm_stream_tester import VlmStreamTester
-from dimos.protocol.skill.skill import skill
-from dimos.protocol.skill.type import Output, Reducer, Stream
-
-__all__ = [
- "Agent",
- "AgentSpec",
- "Output",
- "Reducer",
- "Stream",
- "VLMAgent",
- "VlmStreamTester",
- "deploy",
- "skill",
-]
diff --git a/dimos/agents/agent.py b/dimos/agents/agent.py
index e9c7c5d7b9..76195ccea0 100644
--- a/dimos/agents/agent.py
+++ b/dimos/agents/agent.py
@@ -1,4 +1,4 @@
-# Copyright 2025-2026 Dimensional Inc.
+# Copyright 2026 Dimensional Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
@@ -11,395 +11,195 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
-import asyncio
-import datetime
+
+from dataclasses import dataclass
import json
-from operator import itemgetter
-import os
-from typing import Any, TypedDict
+from queue import Empty, Queue
+from threading import Event, RLock, Thread
+from typing import TYPE_CHECKING, Any, Protocol
import uuid
-from langchain_core.messages import (
- AIMessage,
- HumanMessage,
- SystemMessage,
- ToolCall,
- ToolMessage,
-)
-
-from dimos.agents.llm_init import build_llm, build_system_message
-from dimos.agents.spec import AgentSpec, Model, Provider
-from dimos.core import DimosCluster, rpc
-from dimos.protocol.skill.coordinator import SkillCoordinator, SkillState, SkillStateDict
-from dimos.protocol.skill.skill import SkillContainer
-from dimos.protocol.skill.type import Output
-from dimos.utils.logging_config import setup_logger
-
-logger = setup_logger()
-
-
-SYSTEM_MSG_APPEND = "\nYour message history will always be appended with a System Overview message that provides situational awareness."
-
-
-def toolmsg_from_state(state: SkillState) -> ToolMessage:
- if state.skill_config.output != Output.standard:
- content = "output attached in separate messages"
- else:
- content = state.content() # type: ignore[assignment]
-
- return ToolMessage(
- # if agent call has been triggered by another skill,
- # and this specific skill didn't finish yet but we need a tool call response
- # we return a message explaining that execution is still ongoing
- content=content
- or "Running, you will be called with an update, no need for subsequent tool calls",
- name=state.name,
- tool_call_id=state.call_id,
- )
-
-
-class SkillStateSummary(TypedDict):
- name: str
- call_id: str
- state: str
- data: Any
-
-
-def summary_from_state(state: SkillState, special_data: bool = False) -> SkillStateSummary:
- content = state.content()
- if isinstance(content, dict):
- content = json.dumps(content)
-
- if not isinstance(content, str):
- content = str(content)
-
- return {
- "name": state.name,
- "call_id": state.call_id,
- "state": state.state.name,
- "data": state.content() if not special_data else "data will be in a separate message",
- }
-
-
-def _custom_json_serializers(obj): # type: ignore[no-untyped-def]
- if isinstance(obj, datetime.date | datetime.datetime):
- return obj.isoformat()
- raise TypeError(f"Type {type(obj)} not serializable")
-
-
-# takes an overview of running skills from the coorindator
-# and builds messages to be sent to an agent
-def snapshot_to_messages(
- state: SkillStateDict,
- tool_calls: list[ToolCall],
-) -> tuple[list[ToolMessage], AIMessage | None]:
- # builds a set of tool call ids from a previous agent request
- tool_call_ids = set(
- map(itemgetter("id"), tool_calls),
- )
-
- # build a tool msg responses
- tool_msgs: list[ToolMessage] = []
-
- # build a general skill state overview (for longer running skills)
- state_overview: list[dict[str, SkillStateSummary]] = []
-
- # for special skills that want to return a separate message
- # (images for example, requires to be a HumanMessage)
- special_msgs: list[HumanMessage] = []
-
- # for special skills that want to return a separate message that should
- # stay in history, like actual human messages, critical events
- history_msgs: list[HumanMessage] = []
-
- # Initialize state_msg
- state_msg = None
-
- for skill_state in sorted(
- state.values(),
- key=lambda skill_state: skill_state.duration(),
- ):
- if skill_state.call_id in tool_call_ids:
- tool_msgs.append(toolmsg_from_state(skill_state))
-
- if skill_state.skill_config.output == Output.human:
- content = skill_state.content()
- if not content:
- continue
- history_msgs.append(HumanMessage(content=content)) # type: ignore[arg-type]
- continue
-
- special_data = skill_state.skill_config.output == Output.image
- if special_data:
- content = skill_state.content()
- if not content:
- continue
- special_msgs.append(HumanMessage(content=content)) # type: ignore[arg-type]
-
- if skill_state.call_id in tool_call_ids:
- continue
-
- state_overview.append(summary_from_state(skill_state, special_data)) # type: ignore[arg-type]
-
- if state_overview:
- state_overview_str = "\n".join(
- json.dumps(s, default=_custom_json_serializers) for s in state_overview
+from langchain.agents import create_agent
+from langchain_core.messages import HumanMessage
+from langchain_core.messages.base import BaseMessage
+from langchain_core.tools import StructuredTool
+from langgraph.graph.state import CompiledStateGraph
+from reactivex.disposable import Disposable
+
+from dimos.agents.system_prompt import SYSTEM_PROMPT
+from dimos.agents.utils import pretty_print_langchain_message
+from dimos.core.core import rpc
+from dimos.core.module import Module, ModuleConfig, SkillInfo
+from dimos.core.rpc_client import RpcCall, RPCClient
+from dimos.core.stream import In, Out
+from dimos.protocol.rpc import RPCSpec
+from dimos.spec.utils import Spec
+
+if TYPE_CHECKING:
+ from langchain_core.language_models import BaseChatModel
+
+
+@dataclass
+class AgentConfig(ModuleConfig):
+ system_prompt: str | None = SYSTEM_PROMPT
+ model: str = "gpt-4o"
+ model_fixture: str | None = None
+
+
+class Agent(Module):
+ default_config: type[AgentConfig] = AgentConfig
+ config: AgentConfig
+ agent: Out[BaseMessage]
+ human_input: In[str]
+ agent_idle: Out[bool]
+
+ _lock: RLock
+ _state_graph: CompiledStateGraph[Any, Any, Any, Any] | None
+ _message_queue: Queue[BaseMessage]
+ _history: list[BaseMessage]
+ _thread: Thread
+ _stop_event: Event
+
+ def __init__(self, *args: Any, **kwargs: Any) -> None:
+ super().__init__(*args, **kwargs)
+ self._lock = RLock()
+ self._state_graph = None
+ self._message_queue = Queue()
+ self._history = []
+ self._thread = Thread(
+ target=self._thread_loop,
+ name=f"{self.__class__.__name__}-thread",
+ daemon=True,
)
- state_msg = AIMessage("State Overview:\n" + state_overview_str)
-
- return { # type: ignore[return-value]
- "tool_msgs": tool_msgs,
- "history_msgs": history_msgs,
- "state_msgs": ([state_msg] if state_msg else []) + special_msgs,
- }
-
-
-# Agent class job is to glue skill coordinator state to an agent, builds langchain messages
-class Agent(AgentSpec):
- system_message: SystemMessage
- state_messages: list[AIMessage | HumanMessage]
-
- def __init__( # type: ignore[no-untyped-def]
- self,
- *args,
- **kwargs,
- ) -> None:
- AgentSpec.__init__(self, *args, **kwargs)
-
- self.state_messages = []
- self.coordinator = SkillCoordinator()
- self._history = [] # type: ignore[var-annotated]
- self._agent_id = str(uuid.uuid4())
- self._agent_stopped = False
-
- self.system_message = build_system_message(self.config, append=SYSTEM_MSG_APPEND)
- self.publish(self.system_message)
- self._llm = build_llm(self.config)
-
- @rpc
- def get_agent_id(self) -> str:
- return self._agent_id
+ self._stop_event = Event()
@rpc
def start(self) -> None:
super().start()
- self.coordinator.start()
+
+ def _on_human_input(string: str) -> None:
+ self._message_queue.put(HumanMessage(content=string))
+
+ self._disposables.add(Disposable(self.human_input.subscribe(_on_human_input)))
@rpc
def stop(self) -> None:
- self.coordinator.stop()
- self._agent_stopped = True
+ self._stop_event.set()
+ if self._thread.is_alive():
+ self._thread.join(timeout=2.0)
super().stop()
- def clear_history(self) -> None:
- self._history.clear()
-
- def append_history(self, *msgs: list[AIMessage | HumanMessage]) -> None:
- for msg in msgs:
- self.publish(msg) # type: ignore[arg-type]
-
- self._history.extend(msgs)
-
- def history(self): # type: ignore[no-untyped-def]
- return [self.system_message, *self._history, *self.state_messages]
-
- # Used by agent to execute tool calls
- def execute_tool_calls(self, tool_calls: list[ToolCall]) -> None:
- """Execute a list of tool calls from the agent."""
- if self._agent_stopped:
- logger.warning("Agent is stopped, cannot execute tool calls.")
- return
- for tool_call in tool_calls:
- logger.info(f"executing skill call {tool_call}")
- self.coordinator.call_skill(
- tool_call.get("id"), # type: ignore[arg-type]
- tool_call.get("name"),
- tool_call.get("args"),
- )
-
- # used to inject skill calls into the agent loop without agent asking for it
- def run_implicit_skill(self, skill_name: str, **kwargs) -> None: # type: ignore[no-untyped-def]
- if self._agent_stopped:
- logger.warning("Agent is stopped, cannot execute implicit skill calls.")
- return
- self.coordinator.call_skill(False, skill_name, {"args": kwargs})
-
- async def agent_loop(self, first_query: str = ""): # type: ignore[no-untyped-def]
- # TODO: Should I add a lock here to prevent concurrent calls to agent_loop?
-
- if self._agent_stopped:
- logger.warning("Agent is stopped, cannot run agent loop.")
- # return "Agent is stopped."
- import traceback
-
- traceback.print_stack()
- return "Agent is stopped."
-
- self.state_messages = []
- if first_query:
- self.append_history(HumanMessage(first_query)) # type: ignore[arg-type]
-
- def _get_state() -> str:
- # TODO: FIX THIS EXTREME HACK
- update = self.coordinator.generate_snapshot(clear=False)
- snapshot_msgs = snapshot_to_messages(update, msg.tool_calls)
- return json.dumps(snapshot_msgs, sort_keys=True, default=lambda o: repr(o))
-
- try:
- while True:
- # we are getting tools from the coordinator on each turn
- # since this allows for skillcontainers to dynamically provide new skills
- tools = self.get_tools() # type: ignore[no-untyped-call]
- self._llm = self._llm.bind_tools(tools) # type: ignore[assignment]
-
- # publish to /agent topic for observability
- for state_msg in self.state_messages:
- self.publish(state_msg)
-
- # history() builds our message history dynamically
- # ensures we include latest system state, but not old ones.
- messages = self.history() # type: ignore[no-untyped-call]
-
- # Some LLMs don't work without any human messages. Add an initial one.
- if len(messages) == 1 and isinstance(messages[0], SystemMessage):
- messages.append(
- HumanMessage(
- "Everything is initialized. I'll let you know when you should act."
- )
- )
- self.append_history(messages[-1])
-
- msg = self._llm.invoke(messages)
-
- self.append_history(msg) # type: ignore[arg-type]
-
- logger.info(f"Agent response: {msg.content}")
-
- state = _get_state()
-
- if msg.tool_calls:
- self.execute_tool_calls(msg.tool_calls)
-
- # print(self)
- # print(self.coordinator)
-
- self._write_debug_history_file()
-
- if not self.coordinator.has_active_skills():
- logger.info("No active tasks, exiting agent loop.")
- return msg.content
-
- # coordinator will continue once a skill state has changed in
- # such a way that agent call needs to be executed
-
- if state == _get_state():
- await self.coordinator.wait_for_updates()
-
- # we request a full snapshot of currently running, finished or errored out skills
- # we ask for removal of finished skills from subsequent snapshots (clear=True)
- update = self.coordinator.generate_snapshot(clear=True)
+ @rpc
+ def on_system_modules(self, modules: list[RPCClient]) -> None:
+ assert self.rpc is not None
- # generate tool_msgs and general state update message,
- # depending on a skill having associated tool call from previous interaction
- # we will return a tool message, and not a general state message
- snapshot_msgs = snapshot_to_messages(update, msg.tool_calls)
+ if self.config.model.startswith("ollama:"):
+ from dimos.agents.ollama_agent import ensure_ollama_model
- self.state_messages = snapshot_msgs.get("state_msgs", []) # type: ignore[attr-defined]
- self.append_history(
- *snapshot_msgs.get("tool_msgs", []), # type: ignore[attr-defined]
- *snapshot_msgs.get("history_msgs", []), # type: ignore[attr-defined]
- )
+ ensure_ollama_model(self.config.model.removeprefix("ollama:"))
- except Exception as e:
- logger.error(f"Error in agent loop: {e}")
- import traceback
+ model: str | BaseChatModel = self.config.model
+ if self.config.model_fixture is not None:
+ from dimos.agents.testing import MockModel
- traceback.print_exc()
+ model = MockModel(json_path=self.config.model_fixture)
- @rpc
- def loop_thread(self) -> bool:
- asyncio.run_coroutine_threadsafe(self.agent_loop(), self._loop) # type: ignore[arg-type]
- return True
+ with self._lock:
+ self._state_graph = create_agent(
+ model=model,
+ tools=_get_tools_from_modules(self, modules, self.rpc),
+ system_prompt=self.config.system_prompt,
+ )
+ self._thread.start()
@rpc
- def query(self, query: str): # type: ignore[no-untyped-def]
- # TODO: could this be
- # from distributed.utils import sync
- # return sync(self._loop, self.agent_loop, query)
- return asyncio.run_coroutine_threadsafe(self.agent_loop(query), self._loop).result() # type: ignore[arg-type]
+ def add_message(self, message: BaseMessage) -> None:
+ self._message_queue.put(message)
+
+ def _thread_loop(self) -> None:
+ while not self._stop_event.is_set():
+ try:
+ message = self._message_queue.get(timeout=0.5)
+ except Empty:
+ continue
- async def query_async(self, query: str): # type: ignore[no-untyped-def]
- return await self.agent_loop(query)
+ with self._lock:
+ if not self._state_graph:
+ raise ValueError("No state graph initialized")
+ self._process_message(self._state_graph, message)
- @rpc
- def register_skills(self, container, run_implicit_name: str | None = None): # type: ignore[no-untyped-def]
- ret = self.coordinator.register_skills(container) # type: ignore[func-returns-value]
+ def _process_message(
+ self, state_graph: CompiledStateGraph[Any, Any, Any, Any], message: BaseMessage
+ ) -> None:
+ self.agent_idle.publish(False)
+ self._history.append(message)
+ pretty_print_langchain_message(message)
+ self.agent.publish(message)
- if run_implicit_name:
- self.run_implicit_skill(run_implicit_name)
+ for update in state_graph.stream({"messages": self._history}, stream_mode="updates"):
+ for node_output in update.values():
+ for msg in node_output.get("messages", []):
+ self._history.append(msg)
+ pretty_print_langchain_message(msg)
+ self.agent.publish(msg)
- return ret
+ if self._message_queue.empty():
+ self.agent_idle.publish(True)
- def get_tools(self): # type: ignore[no-untyped-def]
- return self.coordinator.get_tools()
- def _write_debug_history_file(self) -> None:
- file_path = os.getenv("DEBUG_AGENT_HISTORY_FILE")
- if not file_path:
- return
+class AgentSpec(Spec, Protocol):
+ def add_message(self, message: BaseMessage) -> None: ...
- history = [x.__dict__ for x in self.history()] # type: ignore[no-untyped-call]
- with open(file_path, "w") as f:
- json.dump(history, f, default=lambda x: repr(x), indent=2)
+def _get_tools_from_modules(
+ agent: Agent, modules: list[RPCClient], rpc: RPCSpec
+) -> list[StructuredTool]:
+ skills = [skill for module in modules for skill in (module.get_skills() or [])]
+ return [_skill_to_tool(agent, skill, rpc) for skill in skills]
-class LlmAgent(Agent):
- @rpc
- def start(self) -> None:
- super().start()
- self.loop_thread()
+def _skill_to_tool(agent: Agent, skill: SkillInfo, rpc: RPCSpec) -> StructuredTool:
+ rpc_call = RpcCall(None, rpc, skill.func_name, skill.class_name, [])
- @rpc
- def stop(self) -> None:
- super().stop()
+ def wrapped_func(*args: Any, **kwargs: Any) -> str | list[dict[str, Any]]:
+ result = None
+ try:
+ result = rpc_call(*args, **kwargs)
+ except Exception as e:
+ return f"Exception: Error: {e}"
-llm_agent = LlmAgent.blueprint
+ if result is None:
+ return "It has started. You will be updated later."
+ if hasattr(result, "agent_encode"):
+ uuid_ = str(uuid.uuid4())
+ _append_image_to_history(agent, skill, uuid_, result)
+ return f"Tool call started with UUID: {uuid_}"
-def deploy(
- dimos: DimosCluster,
- system_prompt: str = "You are a helpful assistant for controlling a Unitree Go2 robot.",
- model: Model = Model.GPT_4O,
- provider: Provider = Provider.OPENAI, # type: ignore[attr-defined]
- skill_containers: list[SkillContainer] | None = None,
-) -> Agent:
- from dimos.agents.cli.human import HumanInput
+ return str(result)
- if skill_containers is None:
- skill_containers = []
- agent = dimos.deploy( # type: ignore[attr-defined]
- Agent,
- system_prompt=system_prompt,
- model=model,
- provider=provider,
+ return StructuredTool(
+ name=skill.func_name,
+ func=wrapped_func,
+ args_schema=json.loads(skill.args_schema),
)
- human_input = dimos.deploy(HumanInput) # type: ignore[attr-defined]
- human_input.start()
- agent.register_skills(human_input)
-
- for skill_container in skill_containers:
- print("Registering skill container:", skill_container)
- agent.register_skills(skill_container)
-
- agent.run_implicit_skill("human")
- agent.start()
- agent.loop_thread()
+def _append_image_to_history(agent: Agent, skill: SkillInfo, uuid_: str, result: Any) -> None:
+ agent.add_message(
+ HumanMessage(
+ content=[
+ {
+ "type": "text",
+ "text": f"This is the artefact for the '{skill.func_name}' tool with UUID:={uuid_}.",
+ },
+ *result.agent_encode(),
+ ]
+ )
+ )
- return agent # type: ignore[no-any-return]
+agent = Agent.blueprint
-__all__ = ["Agent", "deploy", "llm_agent"]
+__all__ = ["Agent", "AgentSpec", "agent"]
diff --git a/dimos/agents/agent_test_runner.py b/dimos/agents/agent_test_runner.py
new file mode 100644
index 0000000000..7d7fbab03d
--- /dev/null
+++ b/dimos/agents/agent_test_runner.py
@@ -0,0 +1,80 @@
+# Copyright 2026 Dimensional Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from threading import Event, Thread
+
+from langchain_core.messages import AIMessage
+from langchain_core.messages.base import BaseMessage
+from reactivex.disposable import Disposable
+
+from dimos.agents.agent import AgentSpec
+from dimos.core.core import rpc
+from dimos.core.module import Module
+from dimos.core.rpc_client import RPCClient
+from dimos.core.stream import In, Out
+
+
+class AgentTestRunner(Module):
+ agent_spec: AgentSpec
+ agent: In[BaseMessage]
+ agent_idle: In[bool]
+ finished: Out[bool]
+ added: Out[bool]
+
+ def __init__(self, messages: list[BaseMessage]) -> None:
+ super().__init__()
+ self._messages = messages
+ self._idle_event = Event()
+ self._subscription_ready = Event()
+ self._thread = Thread(target=self._thread_loop, daemon=True)
+
+ @rpc
+ def start(self) -> None:
+ super().start()
+ self._disposables.add(Disposable(self.agent.subscribe(self._on_agent_message)))
+ self._disposables.add(Disposable(self.agent_idle.subscribe(self._on_agent_idle)))
+ # Signal that subscription is ready
+ self._subscription_ready.set()
+
+ @rpc
+ def stop(self) -> None:
+ super().stop()
+
+ @rpc
+ def on_system_modules(self, _modules: list[RPCClient]) -> None:
+ self._thread.start()
+
+ def _on_agent_idle(self, idle: bool) -> None:
+ if idle:
+ self._idle_event.set()
+
+ def _on_agent_message(self, message: BaseMessage) -> None:
+ # Check for final AIMessage (no tool calls) to signal completion
+ is_ai = isinstance(message, AIMessage)
+ has_tool_calls = hasattr(message, "tool_calls") and message.tool_calls
+ if is_ai and not has_tool_calls:
+ self.added.publish(True)
+
+ def _thread_loop(self) -> None:
+ # Ensure subscription is ready before sending messages
+ if not self._subscription_ready.wait(5):
+ raise TimeoutError("Timed out waiting for subscription to be ready.")
+
+ for message in self._messages:
+ self._idle_event.clear()
+ self.agent_spec.add_message(message)
+ if not self._idle_event.wait(60):
+ raise TimeoutError("Timed out waiting for message to be processed.")
+
+ self.finished.publish(True)
diff --git a/dimos/agents/annotation.py b/dimos/agents/annotation.py
new file mode 100644
index 0000000000..083a3cbc53
--- /dev/null
+++ b/dimos/agents/annotation.py
@@ -0,0 +1,24 @@
+# Copyright 2026 Dimensional Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from collections.abc import Callable
+from typing import Any, TypeVar
+
+F = TypeVar("F", bound=Callable[..., Any])
+
+
+def skill(func: F) -> F:
+ func.__rpc__ = True # type: ignore[attr-defined]
+ func.__skill__ = True # type: ignore[attr-defined]
+ return func
diff --git a/dimos/agents/cli/human.py b/dimos/agents/cli/human.py
deleted file mode 100644
index e842b3cc8a..0000000000
--- a/dimos/agents/cli/human.py
+++ /dev/null
@@ -1,57 +0,0 @@
-# Copyright 2025-2026 Dimensional Inc.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import queue
-
-from reactivex.disposable import Disposable
-
-from dimos.agents import Output, Reducer, Stream, skill # type: ignore[attr-defined]
-from dimos.core import pLCMTransport, rpc
-from dimos.core.module import Module
-from dimos.core.rpc_client import RpcCall
-
-
-class HumanInput(Module):
- running: bool = False
-
- @skill(stream=Stream.call_agent, reducer=Reducer.string, output=Output.human, hide_skill=True) # type: ignore[arg-type]
- def human(self): # type: ignore[no-untyped-def]
- """receives human input, no need to run this, it's running implicitly"""
- if self.running:
- return "already running"
- self.running = True
- transport = pLCMTransport("/human_input") # type: ignore[var-annotated]
-
- msg_queue = queue.Queue() # type: ignore[var-annotated]
- unsub = transport.subscribe(msg_queue.put) # type: ignore[func-returns-value]
- self._disposables.add(Disposable(unsub))
- yield from iter(msg_queue.get, None)
-
- @rpc
- def start(self) -> None:
- super().start()
-
- @rpc
- def stop(self) -> None:
- super().stop()
-
- @rpc
- def set_AgentSpec_register_skills(self, callable: RpcCall) -> None:
- callable.set_rpc(self.rpc) # type: ignore[arg-type]
- callable(self, run_implicit_name="human")
-
-
-human_input = HumanInput.blueprint
-
-__all__ = ["HumanInput", "human_input"]
diff --git a/dimos/agents/conftest.py b/dimos/agents/conftest.py
index 52d7d5a6bb..23d888b0fe 100644
--- a/dimos/agents/conftest.py
+++ b/dimos/agents/conftest.py
@@ -1,4 +1,4 @@
-# Copyright 2025-2026 Dimensional Inc.
+# Copyright 2026 Dimensional Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
@@ -12,74 +12,98 @@
# See the License for the specific language governing permissions and
# limitations under the License.
+import os
from pathlib import Path
+from threading import Event
+from dotenv import load_dotenv
+from langchain_core.messages.base import BaseMessage
import pytest
from dimos.agents.agent import Agent
-from dimos.agents.testing import MockModel
-from dimos.protocol.skill.test_coordinator import SkillContainerTest
+from dimos.agents.agent_test_runner import AgentTestRunner
+from dimos.core.blueprints import autoconnect
+from dimos.core.global_config import global_config
+from dimos.core.transport import pLCMTransport
+load_dotenv()
-@pytest.fixture
-def fixture_dir():
- return Path(__file__).parent / "fixtures"
+FIXTURE_DIR = Path(__file__).parent / "fixtures"
@pytest.fixture
-def potato_system_prompt() -> str:
- return "Your name is Mr. Potato, potatoes are bad at math. Use a tools if asked to calculate"
+def fixture_dir() -> Path:
+ return FIXTURE_DIR
@pytest.fixture
-def skill_container():
- container = SkillContainerTest()
- try:
- yield container
- finally:
- container.stop()
-
-
-@pytest.fixture
-def create_fake_agent(fixture_dir):
- agent = None
-
- def _agent_factory(*, system_prompt, skill_containers, fixture):
- mock_model = MockModel(json_path=fixture_dir / fixture)
-
- nonlocal agent
- agent = Agent(system_prompt=system_prompt, model_instance=mock_model)
-
- for skill_container in skill_containers:
- agent.register_skills(skill_container)
-
- agent.start()
-
- return agent
-
- try:
- yield _agent_factory
- finally:
- if agent:
- agent.stop()
-
-
-@pytest.fixture
-def create_potato_agent(potato_system_prompt, skill_container, fixture_dir):
- agent = None
-
- def _agent_factory(*, fixture):
- mock_model = MockModel(json_path=fixture_dir / fixture)
-
- nonlocal agent
- agent = Agent(system_prompt=potato_system_prompt, model_instance=mock_model)
- agent.register_skills(skill_container)
- agent.start()
-
- return agent
-
- try:
- yield _agent_factory
- finally:
- if agent:
- agent.stop()
+def agent_setup(request):
+ coordinator = None
+ transports: list[pLCMTransport] = []
+ unsubs: list = []
+ recording = bool(os.getenv("RECORD"))
+
+ def fn(
+ *,
+ blueprints,
+ messages: list[BaseMessage],
+ dask: bool = False,
+ system_prompt: str | None = None,
+ fixture: str | None = None,
+ ) -> list[BaseMessage]:
+ history: list[BaseMessage] = []
+ finished_event = Event()
+
+ agent_transport: pLCMTransport = pLCMTransport("/agent")
+ finished_transport: pLCMTransport = pLCMTransport("/finished")
+ transports.extend([agent_transport, finished_transport])
+
+ def on_message(msg: BaseMessage) -> None:
+ history.append(msg)
+
+ unsubs.append(agent_transport.subscribe(on_message))
+ unsubs.append(finished_transport.subscribe(lambda _: finished_event.set()))
+
+ # Derive fixture path from test name if not explicitly provided.
+ if fixture is not None:
+ fixture_path = FIXTURE_DIR / fixture
+ else:
+ fixture_path = FIXTURE_DIR / f"{request.node.name}.json"
+
+ agent_kwargs: dict = {"system_prompt": system_prompt}
+
+ if recording or fixture_path.exists():
+ # RECORD=1: use real LLM, save responses to fixture file.
+ # No RECORD but fixture exists: play back recorded responses.
+ # The MockModel checks RECORD internally to decide record vs playback.
+ agent_kwargs["model_fixture"] = str(fixture_path)
+
+ blueprint = autoconnect(
+ *blueprints,
+ Agent.blueprint(**agent_kwargs),
+ AgentTestRunner.blueprint(messages=messages),
+ )
+
+ global_config.update(
+ viewer_backend="none",
+ dask=dask,
+ )
+
+ nonlocal coordinator
+ coordinator = blueprint.build()
+
+ if not finished_event.wait(60):
+ raise TimeoutError("Timed out waiting for agent to finish processing messages.")
+
+ return history
+
+ yield fn
+
+ if coordinator is not None:
+ coordinator.stop()
+
+ for transport in transports:
+ transport.stop()
+
+ for unsub in unsubs:
+ unsub()
diff --git a/dimos/agents/demo_agent.py b/dimos/agents/demo_agent.py
new file mode 100644
index 0000000000..b3250fba5b
--- /dev/null
+++ b/dimos/agents/demo_agent.py
@@ -0,0 +1,32 @@
+# Copyright 2026 Dimensional Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from dimos.agents.agent import Agent
+from dimos.core.blueprints import autoconnect
+from dimos.hardware.sensors.camera import zed
+from dimos.hardware.sensors.camera.module import camera_module
+from dimos.hardware.sensors.camera.webcam import Webcam
+
+demo_agent = autoconnect(Agent.blueprint())
+
+demo_agent_camera = autoconnect(
+ Agent.blueprint(),
+ camera_module(
+ hardware=lambda: Webcam(
+ camera_index=0,
+ fps=15,
+ camera_info=zed.CameraInfo.SingleWebcam,
+ ),
+ ),
+)
diff --git a/dimos/agents/fixtures/test_can_call_again_on_error[False].json b/dimos/agents/fixtures/test_can_call_again_on_error[False].json
new file mode 100644
index 0000000000..762b452cff
--- /dev/null
+++ b/dimos/agents/fixtures/test_can_call_again_on_error[False].json
@@ -0,0 +1,34 @@
+{
+ "responses": [
+ {
+ "content": "",
+ "tool_calls": [
+ {
+ "name": "register_user",
+ "args": {
+ "name": "Paul"
+ },
+ "id": "call_gizJWFgoQiYOQMCDqjlshkvk",
+ "type": "tool_call"
+ }
+ ]
+ },
+ {
+ "content": "",
+ "tool_calls": [
+ {
+ "name": "register_user",
+ "args": {
+ "name": "paul"
+ },
+ "id": "call_O9p0ktNw0frMfNXjbul6Do1m",
+ "type": "tool_call"
+ }
+ ]
+ },
+ {
+ "content": "The user named \"paul\" has been registered successfully.",
+ "tool_calls": []
+ }
+ ]
+}
diff --git a/dimos/agents/fixtures/test_can_call_again_on_error[True].json b/dimos/agents/fixtures/test_can_call_again_on_error[True].json
new file mode 100644
index 0000000000..b67efe84e0
--- /dev/null
+++ b/dimos/agents/fixtures/test_can_call_again_on_error[True].json
@@ -0,0 +1,34 @@
+{
+ "responses": [
+ {
+ "content": "",
+ "tool_calls": [
+ {
+ "name": "register_user",
+ "args": {
+ "name": "Paul"
+ },
+ "id": "call_4l78eCMbfsbIC2qPL86jA4S0",
+ "type": "tool_call"
+ }
+ ]
+ },
+ {
+ "content": "",
+ "tool_calls": [
+ {
+ "name": "register_user",
+ "args": {
+ "name": "paul"
+ },
+ "id": "call_Jzi8RU0jRMeCcUDEcRkHzahs",
+ "type": "tool_call"
+ }
+ ]
+ },
+ {
+ "content": "The user named \"paul\" has been registered successfully.",
+ "tool_calls": []
+ }
+ ]
+}
diff --git a/dimos/agents/fixtures/test_can_call_tool[False].json b/dimos/agents/fixtures/test_can_call_tool[False].json
new file mode 100644
index 0000000000..32d7770899
--- /dev/null
+++ b/dimos/agents/fixtures/test_can_call_tool[False].json
@@ -0,0 +1,22 @@
+{
+ "responses": [
+ {
+ "content": "",
+ "tool_calls": [
+ {
+ "name": "add",
+ "args": {
+ "x": 33333,
+ "y": 100
+ },
+ "id": "call_vIdFXzNnojiCXtnEi9J2gXQN",
+ "type": "tool_call"
+ }
+ ]
+ },
+ {
+ "content": "33333 + 100 is 33433.",
+ "tool_calls": []
+ }
+ ]
+}
diff --git a/dimos/agents/fixtures/test_can_call_tool[True].json b/dimos/agents/fixtures/test_can_call_tool[True].json
new file mode 100644
index 0000000000..e9431bb8ea
--- /dev/null
+++ b/dimos/agents/fixtures/test_can_call_tool[True].json
@@ -0,0 +1,22 @@
+{
+ "responses": [
+ {
+ "content": "",
+ "tool_calls": [
+ {
+ "name": "add",
+ "args": {
+ "x": 33333,
+ "y": 100
+ },
+ "id": "call_ZlA2HzNAuHF0H52CKQIPX9Te",
+ "type": "tool_call"
+ }
+ ]
+ },
+ {
+ "content": "The result of 33333 + 100 is 33433.",
+ "tool_calls": []
+ }
+ ]
+}
diff --git a/dimos/agents/fixtures/test_get_gps_position_for_queries.json b/dimos/agents/fixtures/test_get_gps_position_for_queries.json
index 5d95b91bac..c2f163598d 100644
--- a/dimos/agents/fixtures/test_get_gps_position_for_queries.json
+++ b/dimos/agents/fixtures/test_get_gps_position_for_queries.json
@@ -6,19 +6,19 @@
{
"name": "get_gps_position_for_queries",
"args": {
- "args": [
+ "queries": [
"Hyde Park",
"Regent Park",
"Russell Park"
]
},
- "id": "call_xO0VDst53tzetEUq8mapKGS1",
+ "id": "call_KqAjbd5E9VE69YMWBINdWnkw",
"type": "tool_call"
}
]
},
{
- "content": "Here are the latitude and longitude coordinates for the parks:\n\n- Hyde Park: Latitude 37.782601, Longitude -122.413201\n- Regent Park: Latitude 37.782602, Longitude -122.413202\n- Russell Park: Latitude 37.782603, Longitude -122.413203",
+ "content": "Here are the latitude and longitude coordinates for the parks you asked about:\n\n- **Hyde Park**: Latitude 37.782601, Longitude -122.413201\n- **Regent Park**: Latitude 37.782602, Longitude -122.413202\n- **Russell Park**: Latitude 37.782603, Longitude -122.413203",
"tool_calls": []
}
]
diff --git a/dimos/agents/fixtures/test_go_to_object.json b/dimos/agents/fixtures/test_go_to_object.json
deleted file mode 100644
index 80f1e95379..0000000000
--- a/dimos/agents/fixtures/test_go_to_object.json
+++ /dev/null
@@ -1,27 +0,0 @@
-{
- "responses": [
- {
- "content": "I will navigate to the nearest chair.",
- "tool_calls": [
- {
- "name": "navigate_with_text",
- "args": {
- "args": [
- "chair"
- ]
- },
- "id": "call_LP4eewByfO9XaxMtnnWxDUz7",
- "type": "tool_call"
- }
- ]
- },
- {
- "content": "I'm on my way to the chair. Let me know if there's anything else you'd like me to do!",
- "tool_calls": []
- },
- {
- "content": "I have successfully navigated to the chair. Let me know if you need anything else!",
- "tool_calls": []
- }
- ]
-}
diff --git a/dimos/agents/fixtures/test_go_to_semantic_location.json b/dimos/agents/fixtures/test_go_to_semantic_location.json
index 1a10711543..8ea3006142 100644
--- a/dimos/agents/fixtures/test_go_to_semantic_location.json
+++ b/dimos/agents/fixtures/test_go_to_semantic_location.json
@@ -6,17 +6,15 @@
{
"name": "navigate_with_text",
"args": {
- "args": [
- "bookshelf"
- ]
+ "query": "bookshelf"
},
- "id": "call_yPoqcavMo05ogNNy5LMNQl2a",
+ "id": "call_PkS6DWAciWAAAdZfatiXoEdu",
"type": "tool_call"
}
]
},
{
- "content": "I have successfully arrived at the bookshelf. Is there anything specific you need here?",
+ "content": "I have successfully navigated to the bookshelf.",
"tool_calls": []
}
]
diff --git a/dimos/agents/fixtures/test_how_much_is_124181112_plus_124124.json b/dimos/agents/fixtures/test_how_much_is_124181112_plus_124124.json
deleted file mode 100644
index f4dbe0c3a5..0000000000
--- a/dimos/agents/fixtures/test_how_much_is_124181112_plus_124124.json
+++ /dev/null
@@ -1,52 +0,0 @@
-{
- "responses": [
- {
- "content": "",
- "tool_calls": [
- {
- "name": "add",
- "args": {
- "args": [
- 124181112,
- 124124
- ]
- },
- "id": "call_SSoVXz5yihrzR8TWIGnGKSpi",
- "type": "tool_call"
- }
- ]
- },
- {
- "content": "Let me do some potato math... Calculating this will take some time, hold on! \ud83e\udd54",
- "tool_calls": []
- },
- {
- "content": "The result of adding 124,181,112 and 124,124 is 124,305,236. Potatoes work well with tools! \ud83e\udd54\ud83c\udf89",
- "tool_calls": []
- },
- {
- "content": "",
- "tool_calls": [
- {
- "name": "add",
- "args": {
- "args": [
- 1000000000,
- -1000000
- ]
- },
- "id": "call_ge9pv6IRa3yo0vjVaORvrGby",
- "type": "tool_call"
- }
- ]
- },
- {
- "content": "Let's get those numbers crunched. Potatoes need a bit of time! \ud83e\udd54\ud83d\udcca",
- "tool_calls": []
- },
- {
- "content": "The result of one billion plus negative one million is 999,000,000. Potatoes are amazing with some help! \ud83e\udd54\ud83d\udca1",
- "tool_calls": []
- }
- ]
-}
diff --git a/dimos/agents/fixtures/test_image.json b/dimos/agents/fixtures/test_image.json
new file mode 100644
index 0000000000..196c2122ef
--- /dev/null
+++ b/dimos/agents/fixtures/test_image.json
@@ -0,0 +1,23 @@
+{
+ "responses": [
+ {
+ "content": "",
+ "tool_calls": [
+ {
+ "name": "take_a_picture",
+ "args": {},
+ "id": "call_11vyrbtZXlsEgoKuQs8jRjP8",
+ "type": "tool_call"
+ }
+ ]
+ },
+ {
+ "content": "I have taken a picture, let me analyze it to provide a description.\nBased on the image captured, the setting resembles a \"cafe\". It shows an indoor space with tables, chairs, and a warm, inviting atmosphere typical of a cafe environment, where people might gather for drinks and conversation.",
+ "tool_calls": []
+ },
+ {
+ "content": "The image depicts a \"cafe\" setting. It shows people sitting outside at tables, likely enjoying drinks or meals, with a cozy and inviting ambiance typical of a cafe.",
+ "tool_calls": []
+ }
+ ]
+}
diff --git a/dimos/agents/fixtures/test_multiple_tool_calls_with_multiple_messages.json b/dimos/agents/fixtures/test_multiple_tool_calls_with_multiple_messages.json
new file mode 100644
index 0000000000..7fd8172d35
--- /dev/null
+++ b/dimos/agents/fixtures/test_multiple_tool_calls_with_multiple_messages.json
@@ -0,0 +1,98 @@
+{
+ "responses": [
+ {
+ "content": "",
+ "tool_calls": [
+ {
+ "name": "locate_person",
+ "args": {
+ "name": "John"
+ },
+ "id": "call_w4cTUUpojE2zRMYda93ma3io",
+ "type": "tool_call"
+ }
+ ]
+ },
+ {
+ "content": "",
+ "tool_calls": [
+ {
+ "name": "register_person",
+ "args": {
+ "name": "John"
+ },
+ "id": "call_Rij2ZZHG1u2yEKALNqH4L0WH",
+ "type": "tool_call"
+ }
+ ]
+ },
+ {
+ "content": "",
+ "tool_calls": [
+ {
+ "name": "locate_person",
+ "args": {
+ "name": "John"
+ },
+ "id": "call_Pr1HUeq1j2L9bYU3CsTXVmnk",
+ "type": "tool_call"
+ }
+ ]
+ },
+ {
+ "content": "",
+ "tool_calls": [
+ {
+ "name": "go_to_location",
+ "args": {
+ "description": "kitchen"
+ },
+ "id": "call_F6Pw6Da2mixJ1mmhb0q4uI3F",
+ "type": "tool_call"
+ }
+ ]
+ },
+ {
+ "content": "I have moved to the kitchen where John is located.",
+ "tool_calls": []
+ },
+ {
+ "content": "",
+ "tool_calls": [
+ {
+ "name": "register_person",
+ "args": {
+ "name": "Jane"
+ },
+ "id": "call_52hL9l50BzPFPruLoPxigoVa",
+ "type": "tool_call"
+ },
+ {
+ "name": "locate_person",
+ "args": {
+ "name": "Jane"
+ },
+ "id": "call_0NbGZIn9BOCENTfrKvuacarZ",
+ "type": "tool_call"
+ }
+ ]
+ },
+ {
+ "content": "",
+ "tool_calls": [
+ {
+ "name": "go_to_location",
+ "args": {
+ "description": "living room"
+ },
+ "id": "call_XoErhAixvK31yHOeCKiwgrKj",
+ "type": "tool_call"
+ }
+ ]
+ },
+ {
+ "content": "I have moved to the living room where Jane is located.",
+ "tool_calls": []
+ }
+ ]
+}
diff --git a/dimos/agents/fixtures/test_pounce.json b/dimos/agents/fixtures/test_pounce.json
index 99e84d003a..4213c12c95 100644
--- a/dimos/agents/fixtures/test_pounce.json
+++ b/dimos/agents/fixtures/test_pounce.json
@@ -6,32 +6,15 @@
{
"name": "execute_sport_command",
"args": {
- "args": [
- "FrontPounce"
- ]
+ "command_name": "FrontPounce"
},
- "id": "call_Ukj6bCAnHQLj28RHRp697blZ",
+ "id": "call_Z7X0sJfUygGiJUUL67I64eRs",
"type": "tool_call"
}
]
},
{
- "content": "",
- "tool_calls": [
- {
- "name": "speak",
- "args": {
- "args": [
- "I have successfully performed a front pounce."
- ]
- },
- "id": "call_FR9DtqEvJ9zSY85qVD2UFrll",
- "type": "tool_call"
- }
- ]
- },
- {
- "content": "I have successfully performed a front pounce.",
+ "content": "The robot has successfully performed a \"FrontPounce\" action!",
"tool_calls": []
}
]
diff --git a/dimos/agents/fixtures/test_prompt.json b/dimos/agents/fixtures/test_prompt.json
new file mode 100644
index 0000000000..b924f66f52
--- /dev/null
+++ b/dimos/agents/fixtures/test_prompt.json
@@ -0,0 +1,8 @@
+{
+ "responses": [
+ {
+ "content": "My name is Johnny. How can I assist you today?",
+ "tool_calls": []
+ }
+ ]
+}
diff --git a/dimos/agents/fixtures/test_set_gps_travel_points.json b/dimos/agents/fixtures/test_set_gps_travel_points.json
index eb5b2a9195..e1392125a1 100644
--- a/dimos/agents/fixtures/test_set_gps_travel_points.json
+++ b/dimos/agents/fixtures/test_set_gps_travel_points.json
@@ -1,29 +1,36 @@
{
"responses": [
{
- "content": "I understand you want me to navigate to the specified location. I will set the GPS travel point accordingly.",
+ "content": "",
+ "tool_calls": [
+ {
+ "name": "set_gps_travel_points",
+ "args": {},
+ "id": "call_0Onzw4fDoT68BWNfZUiGlIbg",
+ "type": "tool_call"
+ }
+ ]
+ },
+ {
+ "content": "It seems there was an issue with the initial attempt. Let me try again by providing the correct parameter.",
"tool_calls": [
{
"name": "set_gps_travel_points",
"args": {
- "args": [
+ "points": [
{
"lat": 37.782654,
"lon": -122.413273
}
]
},
- "id": "call_q6JCCYFuyAjqUgUibJHqcIMD",
+ "id": "call_b0QRHVc09ZtY8jbQaUsny7Yx",
"type": "tool_call"
}
]
},
{
- "content": "I'm on my way to the specified location. Let me know if there is anything else I can assist you with!",
- "tool_calls": []
- },
- {
- "content": "I've reached the specified location. Do you need any further assistance?",
+ "content": "The GPS travel point has been successfully set to latitude: 37.782654 and longitude: -122.413273.",
"tool_calls": []
}
]
diff --git a/dimos/agents/fixtures/test_set_gps_travel_points_multiple.json b/dimos/agents/fixtures/test_set_gps_travel_points_multiple.json
index 9d8f7e9e00..2391e81fe1 100644
--- a/dimos/agents/fixtures/test_set_gps_travel_points_multiple.json
+++ b/dimos/agents/fixtures/test_set_gps_travel_points_multiple.json
@@ -1,12 +1,23 @@
{
"responses": [
+ {
+ "content": "",
+ "tool_calls": [
+ {
+ "name": "set_gps_travel_points",
+ "args": {},
+ "id": "call_JWgodQUZD16l2tjePmTXGX7V",
+ "type": "tool_call"
+ }
+ ]
+ },
{
"content": "",
"tool_calls": [
{
"name": "set_gps_travel_points",
"args": {
- "args": [
+ "points": [
{
"lat": 37.782654,
"lon": -122.413273
@@ -21,13 +32,13 @@
}
]
},
- "id": "call_Q09MRMEgRnJPBOGZpM0j8sL2",
+ "id": "call_e5szuuZrTdq6deN8mUM5kusY",
"type": "tool_call"
}
]
},
{
- "content": "I've successfully set the travel points and will navigate to them sequentially.",
+ "content": "The GPS travel points have been successfully set in the specified order.",
"tool_calls": []
}
]
diff --git a/dimos/agents/fixtures/test_show_your_love.json b/dimos/agents/fixtures/test_show_your_love.json
deleted file mode 100644
index 941906e781..0000000000
--- a/dimos/agents/fixtures/test_show_your_love.json
+++ /dev/null
@@ -1,38 +0,0 @@
-{
- "responses": [
- {
- "content": "",
- "tool_calls": [
- {
- "name": "execute_sport_command",
- "args": {
- "args": [
- "FingerHeart"
- ]
- },
- "id": "call_VFp6x9F00FBmiiUiemFWewop",
- "type": "tool_call"
- }
- ]
- },
- {
- "content": "",
- "tool_calls": [
- {
- "name": "speak",
- "args": {
- "args": [
- "Here's a gesture to show you some love!"
- ]
- },
- "id": "call_WUUmBJ95s9PtVx8YQsmlJ4EU",
- "type": "tool_call"
- }
- ]
- },
- {
- "content": "Just did a finger heart gesture to show my affection!",
- "tool_calls": []
- }
- ]
-}
diff --git a/dimos/agents/fixtures/test_take_a_look_around.json b/dimos/agents/fixtures/test_start_exploration.json
similarity index 51%
rename from dimos/agents/fixtures/test_take_a_look_around.json
rename to dimos/agents/fixtures/test_start_exploration.json
index c30fe71017..713e6e2dba 100644
--- a/dimos/agents/fixtures/test_take_a_look_around.json
+++ b/dimos/agents/fixtures/test_start_exploration.json
@@ -6,17 +6,15 @@
{
"name": "start_exploration",
"args": {
- "args": [
- 10
- ]
+ "timeout": 10
},
- "id": "call_AMNeD8zTkvyFHKG90DriDPuM",
+ "id": "call_o5O7xLaI4iayDhszXblOiWVS",
"type": "tool_call"
}
]
},
{
- "content": "I have completed a brief exploration of the surroundings. Let me know if there's anything specific you need!",
+ "content": "I have completed the exploration for 10 seconds. If there's anything specific you would like to do next, please let me know!",
"tool_calls": []
}
]
diff --git a/dimos/agents/fixtures/test_stop_movement.json b/dimos/agents/fixtures/test_stop_movement.json
index b80834213e..f9f10af0e0 100644
--- a/dimos/agents/fixtures/test_stop_movement.json
+++ b/dimos/agents/fixtures/test_stop_movement.json
@@ -5,16 +5,14 @@
"tool_calls": [
{
"name": "stop_movement",
- "args": {
- "args": null
- },
- "id": "call_oAKe9W8s3xRGioZhBJJDOZB1",
+ "args": {},
+ "id": "call_dHP1UE2Bw180bzxPT2wI2Dam",
"type": "tool_call"
}
]
},
{
- "content": "I have stopped moving. Let me know if you need anything else!",
+ "content": "The movement has been stopped.",
"tool_calls": []
}
]
diff --git a/dimos/agents/fixtures/test_what_do_you_see_in_this_picture.json b/dimos/agents/fixtures/test_what_do_you_see_in_this_picture.json
deleted file mode 100644
index 27ac3453bc..0000000000
--- a/dimos/agents/fixtures/test_what_do_you_see_in_this_picture.json
+++ /dev/null
@@ -1,25 +0,0 @@
-{
- "responses": [
- {
- "content": "",
- "tool_calls": [
- {
- "name": "take_photo",
- "args": {
- "args": []
- },
- "id": "call_o6ikJtK3vObuEFD6hDtLoyGQ",
- "type": "tool_call"
- }
- ]
- },
- {
- "content": "I took a photo, but as an AI, I can't see or interpret images. If there's anything specific you need to know, feel free to ask!",
- "tool_calls": []
- },
- {
- "content": "It looks like a cozy outdoor cafe where people are sitting and enjoying a meal. There are flowers and a nice, sunny ambiance. If you have any specific questions about the image, let me know!",
- "tool_calls": []
- }
- ]
-}
diff --git a/dimos/agents/fixtures/test_what_is_your_name.json b/dimos/agents/fixtures/test_what_is_your_name.json
deleted file mode 100644
index a74d793b1d..0000000000
--- a/dimos/agents/fixtures/test_what_is_your_name.json
+++ /dev/null
@@ -1,8 +0,0 @@
-{
- "responses": [
- {
- "content": "Hi! My name is Mr. Potato. How can I assist you today?",
- "tool_calls": []
- }
- ]
-}
diff --git a/dimos/agents/fixtures/test_where_am_i.json b/dimos/agents/fixtures/test_where_am_i.json
index 2d274f8fa6..a4a06a5c72 100644
--- a/dimos/agents/fixtures/test_where_am_i.json
+++ b/dimos/agents/fixtures/test_where_am_i.json
@@ -5,16 +5,14 @@
"tool_calls": [
{
"name": "where_am_i",
- "args": {
- "args": []
- },
- "id": "call_uRJLockZ5JWtGWbsSL1dpHm3",
+ "args": {},
+ "id": "call_4eiRtfr8mI7SgLfR0FTjH7Pp",
"type": "tool_call"
}
]
},
{
- "content": "You are on Bourbon Street.",
+ "content": "You are currently on Bourbon Street.",
"tool_calls": []
}
]
diff --git a/dimos/agents/llm_init.py b/dimos/agents/llm_init.py
deleted file mode 100644
index eb8c33c631..0000000000
--- a/dimos/agents/llm_init.py
+++ /dev/null
@@ -1,62 +0,0 @@
-# Copyright 2025-2026 Dimensional Inc.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from typing import cast
-
-from langchain.chat_models import init_chat_model
-from langchain_core.language_models.chat_models import BaseChatModel
-from langchain_core.messages import SystemMessage
-from langchain_huggingface import ChatHuggingFace, HuggingFacePipeline
-
-from dimos.agents.ollama_agent import ensure_ollama_model
-from dimos.agents.spec import AgentConfig
-from dimos.agents.system_prompt import SYSTEM_PROMPT
-
-
-def build_llm(config: AgentConfig) -> BaseChatModel:
- if config.model_instance:
- return config.model_instance
-
- if config.provider.value.lower() == "ollama":
- ensure_ollama_model(config.model)
-
- if config.provider.value.lower() == "huggingface":
- llm = HuggingFacePipeline.from_model_id(
- model_id=config.model,
- task="text-generation",
- pipeline_kwargs={
- "max_new_tokens": 512,
- "temperature": 0.7,
- },
- )
- return ChatHuggingFace(llm=llm, model_id=config.model)
-
- return cast(
- "BaseChatModel",
- init_chat_model( # type: ignore[call-overload]
- model_provider=config.provider,
- model=config.model,
- ),
- )
-
-
-def build_system_message(config: AgentConfig, *, append: str = "") -> SystemMessage:
- if config.system_prompt:
- if isinstance(config.system_prompt, str):
- return SystemMessage(config.system_prompt + append)
- if append:
- config.system_prompt.content += append # type: ignore[operator]
- return config.system_prompt
-
- return SystemMessage(SYSTEM_PROMPT + append)
diff --git a/dimos/agents/skills/conftest.py b/dimos/agents/skills/conftest.py
deleted file mode 100644
index 0e2e3e0636..0000000000
--- a/dimos/agents/skills/conftest.py
+++ /dev/null
@@ -1,113 +0,0 @@
-# Copyright 2025-2026 Dimensional Inc.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from functools import partial
-
-import pytest
-from reactivex.scheduler import ThreadPoolScheduler
-
-from dimos.agents.skills.google_maps_skill_container import GoogleMapsSkillContainer
-from dimos.agents.skills.gps_nav_skill import GpsNavSkillContainer
-from dimos.agents.skills.navigation import NavigationSkillContainer
-from dimos.agents.system_prompt import SYSTEM_PROMPT
-from dimos.robot.unitree_webrtc.unitree_skill_container import UnitreeSkillContainer
-
-
-@pytest.fixture(autouse=True)
-def cleanup_threadpool_scheduler(monkeypatch):
- # TODO: get rid of this global threadpool
- """Clean up and recreate the global ThreadPoolScheduler after each test."""
- # Disable ChromaDB telemetry to avoid leaking threads
- monkeypatch.setenv("CHROMA_ANONYMIZED_TELEMETRY", "False")
- yield
- from dimos.utils import threadpool
-
- # Shutdown the global scheduler's executor
- threadpool.scheduler.executor.shutdown(wait=True)
- # Recreate it for the next test
- threadpool.scheduler = ThreadPoolScheduler(max_workers=threadpool.get_max_workers())
-
-
-@pytest.fixture
-def navigation_skill_container(mocker):
- container = NavigationSkillContainer()
- container.color_image.connection = mocker.MagicMock()
- container.odom.connection = mocker.MagicMock()
- container.start()
- yield container
- container.stop()
-
-
-@pytest.fixture
-def gps_nav_skill_container(mocker):
- container = GpsNavSkillContainer()
- container.gps_location.connection = mocker.MagicMock()
- container.gps_goal = mocker.MagicMock()
- container.start()
- yield container
- container.stop()
-
-
-@pytest.fixture
-def google_maps_skill_container(mocker):
- container = GoogleMapsSkillContainer()
- container.gps_location.connection = mocker.MagicMock()
- container.start()
- container._client = mocker.MagicMock()
- yield container
- container.stop()
-
-
-@pytest.fixture
-def unitree_skills(mocker):
- container = UnitreeSkillContainer()
- container.start()
- yield container
- container.stop()
-
-
-@pytest.fixture
-def create_navigation_agent(navigation_skill_container, create_fake_agent):
- return partial(
- create_fake_agent,
- system_prompt=SYSTEM_PROMPT,
- skill_containers=[navigation_skill_container],
- )
-
-
-@pytest.fixture
-def create_gps_nav_agent(gps_nav_skill_container, create_fake_agent):
- return partial(
- create_fake_agent, system_prompt=SYSTEM_PROMPT, skill_containers=[gps_nav_skill_container]
- )
-
-
-@pytest.fixture
-def create_google_maps_agent(
- gps_nav_skill_container, google_maps_skill_container, create_fake_agent
-):
- return partial(
- create_fake_agent,
- system_prompt=SYSTEM_PROMPT,
- skill_containers=[gps_nav_skill_container, google_maps_skill_container],
- )
-
-
-@pytest.fixture
-def create_unitree_skills_agent(unitree_skills, create_fake_agent):
- return partial(
- create_fake_agent,
- system_prompt=SYSTEM_PROMPT,
- skill_containers=[unitree_skills],
- )
diff --git a/dimos/agents/skills/demo_calculator_skill.py b/dimos/agents/skills/demo_calculator_skill.py
index 2ed8050ca5..61d66e301a 100644
--- a/dimos/agents/skills/demo_calculator_skill.py
+++ b/dimos/agents/skills/demo_calculator_skill.py
@@ -12,18 +12,18 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-from dimos.core.skill_module import SkillModule
-from dimos.protocol.skill.skill import skill
+from dimos.agents.annotation import skill
+from dimos.core.module import Module
-class DemoCalculatorSkill(SkillModule):
+class DemoCalculatorSkill(Module):
def start(self) -> None:
super().start()
def stop(self) -> None:
super().stop()
- @skill()
+ @skill
def sum_numbers(self, n1: int, n2: int, *args: int, **kwargs: int) -> str:
"""This skill adds two numbers. Always use this tool. Never add up numbers yourself.
diff --git a/dimos/agents/skills/demo_google_maps_skill.py b/dimos/agents/skills/demo_google_maps_skill.py
index cd8cad9d6a..13f2ebc19b 100644
--- a/dimos/agents/skills/demo_google_maps_skill.py
+++ b/dimos/agents/skills/demo_google_maps_skill.py
@@ -13,20 +13,13 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-from dotenv import load_dotenv
-
-from dimos.agents.agent import llm_agent
-from dimos.agents.cli.human import human_input
+from dimos.agents.agent import agent
from dimos.agents.skills.demo_robot import demo_robot
from dimos.agents.skills.google_maps_skill_container import google_maps_skill
from dimos.core.blueprints import autoconnect
-load_dotenv()
-
-
demo_google_maps_skill = autoconnect(
demo_robot(),
google_maps_skill(),
- human_input(),
- llm_agent(),
+ agent(),
)
diff --git a/dimos/agents/skills/demo_gps_nav.py b/dimos/agents/skills/demo_gps_nav.py
index 4204b23dc7..7a6abd32dd 100644
--- a/dimos/agents/skills/demo_gps_nav.py
+++ b/dimos/agents/skills/demo_gps_nav.py
@@ -13,20 +13,13 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-from dotenv import load_dotenv
-
-from dimos.agents.agent import llm_agent
-from dimos.agents.cli.human import human_input
+from dimos.agents.agent import agent
from dimos.agents.skills.demo_robot import demo_robot
from dimos.agents.skills.gps_nav_skill import gps_nav_skill
from dimos.core.blueprints import autoconnect
-load_dotenv()
-
-
-demo_gps_nav_skill = autoconnect(
+demo_gps_nav = autoconnect(
demo_robot(),
gps_nav_skill(),
- human_input(),
- llm_agent(),
+ agent(),
)
diff --git a/dimos/agents/skills/demo_skill.py b/dimos/agents/skills/demo_skill.py
index 547d81c5b8..b067a3fbc2 100644
--- a/dimos/agents/skills/demo_skill.py
+++ b/dimos/agents/skills/demo_skill.py
@@ -13,18 +13,11 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-from dotenv import load_dotenv
-
-from dimos.agents.agent import llm_agent
-from dimos.agents.cli.human import human_input
+from dimos.agents.agent import agent
from dimos.agents.skills.demo_calculator_skill import demo_calculator_skill
from dimos.core.blueprints import autoconnect
-load_dotenv()
-
-
demo_skill = autoconnect(
demo_calculator_skill(),
- human_input(),
- llm_agent(),
+ agent(),
)
diff --git a/dimos/agents/skills/google_maps_skill_container.py b/dimos/agents/skills/google_maps_skill_container.py
index d5a30904ed..33b2ee9f10 100644
--- a/dimos/agents/skills/google_maps_skill_container.py
+++ b/dimos/agents/skills/google_maps_skill_container.py
@@ -15,18 +15,18 @@
import json
from typing import Any
+from dimos.agents.annotation import skill
from dimos.core.core import rpc
-from dimos.core.skill_module import SkillModule
+from dimos.core.module import Module
from dimos.core.stream import In
from dimos.mapping.google_maps.google_maps import GoogleMaps
from dimos.mapping.types import LatLon
-from dimos.protocol.skill.skill import skill
from dimos.utils.logging_config import setup_logger
logger = setup_logger()
-class GoogleMapsSkillContainer(SkillModule):
+class GoogleMapsSkillContainer(Module):
_latest_location: LatLon | None = None
_client: GoogleMaps
@@ -55,7 +55,7 @@ def _get_latest_location(self) -> LatLon:
raise ValueError("The position has not been set yet.")
return self._latest_location
- @skill()
+ @skill
def where_am_i(self, context_radius: int = 200) -> str:
"""This skill returns information about what street/locality/city/etc
you are in. It also gives you nearby landmarks.
@@ -81,7 +81,7 @@ def where_am_i(self, context_radius: int = 200) -> str:
return result.model_dump_json()
- @skill()
+ @skill
def get_gps_position_for_queries(self, queries: list[str]) -> str:
"""Get the GPS position (latitude/longitude) from Google Maps for know landmarks or searchable locations.
This includes anything that wouldn't be viewable on a physical OSM map including intersections (5th and Natoma)
diff --git a/dimos/agents/skills/gps_nav_skill.py b/dimos/agents/skills/gps_nav_skill.py
index c7325a5b64..721119f6e6 100644
--- a/dimos/agents/skills/gps_nav_skill.py
+++ b/dimos/agents/skills/gps_nav_skill.py
@@ -14,19 +14,19 @@
import json
+from dimos.agents.annotation import skill
from dimos.core.core import rpc
+from dimos.core.module import Module
from dimos.core.rpc_client import RpcCall
-from dimos.core.skill_module import SkillModule
from dimos.core.stream import In, Out
from dimos.mapping.types import LatLon
from dimos.mapping.utils.distance import distance_in_meters
-from dimos.protocol.skill.skill import skill
from dimos.utils.logging_config import setup_logger
logger = setup_logger()
-class GpsNavSkillContainer(SkillModule):
+class GpsNavSkillContainer(Module):
_latest_location: LatLon | None = None
_max_valid_distance: int = 50000
_set_gps_travel_goal_points: RpcCall | None = None
@@ -59,8 +59,8 @@ def _get_latest_location(self) -> LatLon:
raise ValueError("The position has not been set yet.")
return self._latest_location
- @skill()
- def set_gps_travel_points(self, *points: dict[str, float]) -> str:
+ @skill
+ def set_gps_travel_points(self, points: list[dict[str, float]]) -> str:
"""Define the movement path determined by GPS coordinates. Requires at least one. You can get the coordinates by using the `get_gps_position_for_queries` skill.
Example:
diff --git a/dimos/agents/skills/navigation.py b/dimos/agents/skills/navigation.py
index 054246d6ee..322a09c2bb 100644
--- a/dimos/agents/skills/navigation.py
+++ b/dimos/agents/skills/navigation.py
@@ -15,8 +15,11 @@
import time
from typing import Any
+from reactivex.disposable import Disposable
+
+from dimos.agents.annotation import skill
from dimos.core.core import rpc
-from dimos.core.skill_module import SkillModule
+from dimos.core.module import Module
from dimos.core.stream import In
from dimos.models.qwen.video_query import BBox
from dimos.models.vl.qwen import QwenVlModel
@@ -25,14 +28,13 @@
from dimos.msgs.sensor_msgs import Image
from dimos.navigation.base import NavigationState
from dimos.navigation.visual.query import get_object_bbox_from_image
-from dimos.protocol.skill.skill import skill
from dimos.types.robot_location import RobotLocation
from dimos.utils.logging_config import setup_logger
logger = setup_logger()
-class NavigationSkillContainer(SkillModule):
+class NavigationSkillContainer(Module):
_latest_image: Image | None = None
_latest_odom: PoseStamped | None = None
_skill_started: bool = False
@@ -49,9 +51,6 @@ class NavigationSkillContainer(SkillModule):
"ObjectTracking.track",
"ObjectTracking.stop_track",
"ObjectTracking.is_tracking",
- "WavefrontFrontierExplorer.stop_exploration",
- "WavefrontFrontierExplorer.explore",
- "WavefrontFrontierExplorer.is_exploration_active",
]
color_image: In[Image]
@@ -64,8 +63,9 @@ def __init__(self) -> None:
@rpc
def start(self) -> None:
- self._disposables.add(self.color_image.subscribe(self._on_color_image)) # type: ignore[arg-type]
- self._disposables.add(self.odom.subscribe(self._on_odom)) # type: ignore[arg-type]
+ super().start()
+ self._disposables.add(Disposable(self.color_image.subscribe(self._on_color_image)))
+ self._disposables.add(Disposable(self.odom.subscribe(self._on_odom)))
self._skill_started = True
@rpc
@@ -78,7 +78,7 @@ def _on_color_image(self, image: Image) -> None:
def _on_odom(self, odom: PoseStamped) -> None:
self._latest_odom = odom
- @skill()
+ @skill
def tag_location(self, location_name: str) -> str:
"""Tag this location in the spatial memory with a name.
@@ -93,12 +93,12 @@ def tag_location(self, location_name: str) -> str:
if not self._skill_started:
raise ValueError(f"{self} has not been started.")
- tf = self.tf.get("map", "base_link", time_tolerance=2.0)
- if not tf:
- return "Could not get the robot's current transform."
- position = tf.translation
- rotation = tf.rotation.to_euler()
+ if not self._latest_odom:
+ return "No odometry data received yet, cannot tag location."
+
+ position = self._latest_odom.position
+ rotation = self._latest_odom.orientation
location = RobotLocation(
name=location_name,
@@ -113,7 +113,7 @@ def tag_location(self, location_name: str) -> str:
logger.info(f"Tagged {location}")
return f"Tagged '{location_name}': ({position.x},{position.y})."
- @skill()
+ @skill
def navigate_with_text(self, query: str) -> str:
"""Navigate to a location by querying the existing semantic map using natural language.
@@ -158,48 +158,31 @@ def _navigate_by_tagged_location(self, query: str) -> str | None:
if not robot_location:
return None
- print("Found tagged location:", robot_location)
+ logger.info("Found tagged location", location=robot_location)
goal_pose = PoseStamped(
position=make_vector3(*robot_location.position),
orientation=Quaternion.from_euler(Vector3(*robot_location.rotation)),
frame_id="map",
)
- result = self._navigate_to(goal_pose)
- if not result:
- return "Error: Faild to reach the tagged location."
-
- return (
- f"Successfuly arrived at location tagged '{robot_location.name}' from query '{query}'."
- )
+ return self._navigate_to(goal_pose, f"Found a tagged location called '{query}'.")
- def _navigate_to(self, pose: PoseStamped) -> bool:
+ def _navigate_to(self, pose: PoseStamped, message: str) -> str:
try:
- set_goal_rpc, get_state_rpc, is_goal_reached_rpc = self.get_rpc_calls(
- "NavigationInterface.set_goal",
- "NavigationInterface.get_state",
- "NavigationInterface.is_goal_reached",
- )
+ set_goal_rpc = self.get_rpc_calls("NavigationInterface.set_goal")
except Exception:
logger.error("Navigation module not connected properly")
- return False
+ return "Error: Navigation module is not connected, cannot set goal."
logger.info(
f"Navigating to pose: ({pose.position.x:.2f}, {pose.position.y:.2f}, {pose.position.z:.2f})"
)
set_goal_rpc(pose)
- time.sleep(1.0)
-
- while get_state_rpc() == NavigationState.FOLLOWING_PATH:
- time.sleep(0.25)
- time.sleep(1.0)
- if not is_goal_reached_rpc():
- logger.info("Navigation was cancelled or failed")
- return False
- else:
- logger.info("Navigation goal reached")
- return True
+ return (
+ f"{message}. Started navigating to that position. "
+ f"To cancel movement call the 'stop_navigation' tool."
+ )
def _navigate_to_object(self, query: str) -> str | None:
try:
@@ -286,24 +269,15 @@ def _navigate_using_semantic_map(self, query: str) -> str:
goal_pose = self._get_goal_pose_from_result(best_match)
- print("Goal pose for semantic nav:", goal_pose)
+ logger.info("Goal pose for semantic nav", pose=goal_pose)
if not goal_pose:
return f"Found a result for '{query}' but it didn't have a valid position."
- result = self._navigate_to(goal_pose)
+ message = f"Found a location in the semantic map matching '{query}'."
+ return self._navigate_to(goal_pose, message)
- if not result:
- return f"Failed to navigate for '{query}'"
-
- return f"Successfuly arrived at '{query}'"
-
- @skill()
- def follow_human(self, person: str) -> str:
- """Follow a specific person"""
- return "Not implemented yet."
-
- @skill()
- def stop_movement(self) -> str:
+ @skill
+ def stop_navigation(self) -> str:
"""Immediatly stop moving."""
if not self._skill_started:
@@ -320,57 +294,7 @@ def _cancel_goal_and_stop(self) -> None:
logger.warning("Navigation module not connected, cannot cancel goal")
return
- try:
- stop_exploration_rpc = self.get_rpc_calls("WavefrontFrontierExplorer.stop_exploration")
- except Exception:
- logger.warning("FrontierExplorer module not connected, cannot stop exploration")
- return
-
cancel_goal_rpc()
- return stop_exploration_rpc() # type: ignore[no-any-return]
-
- @skill()
- def start_exploration(self, timeout: float = 240.0) -> str:
- """A skill that performs autonomous frontier exploration.
-
- This skill continuously finds and navigates to unknown frontiers in the environment
- until no more frontiers are found or the exploration is stopped.
-
- Don't call any other skills except stop_movement skill when needed.
-
- Args:
- timeout (float, optional): Maximum time (in seconds) allowed for exploration
- """
-
- if not self._skill_started:
- raise ValueError(f"{self} has not been started.")
-
- try:
- return self._start_exploration(timeout)
- finally:
- self._cancel_goal_and_stop()
-
- def _start_exploration(self, timeout: float) -> str:
- try:
- explore_rpc, is_exploration_active_rpc = self.get_rpc_calls(
- "WavefrontFrontierExplorer.explore",
- "WavefrontFrontierExplorer.is_exploration_active",
- )
- except Exception:
- return "Error: The WavefrontFrontierExplorer module is not connected."
-
- logger.info("Starting autonomous frontier exploration")
-
- start_time = time.time()
-
- has_started = explore_rpc()
- if not has_started:
- return "Error: Could not start exploration."
-
- while time.time() - start_time < timeout and is_exploration_active_rpc():
- time.sleep(0.5)
-
- return "Exploration completed successfuly"
def _get_goal_pose_from_result(self, result: dict[str, Any]) -> PoseStamped | None:
similarity = 1.0 - (result.get("distance") or 1)
@@ -383,9 +307,7 @@ def _get_goal_pose_from_result(self, result: dict[str, Any]) -> PoseStamped | No
metadata = result.get("metadata")
if not metadata:
return None
- print(metadata)
first = metadata[0]
- print(first)
pos_x = first.get("pos_x", 0)
pos_y = first.get("pos_y", 0)
theta = first.get("rot_z", 0)
diff --git a/dimos/agents/skills/osm.py b/dimos/agents/skills/osm.py
index 71f453069f..9bb731cf72 100644
--- a/dimos/agents/skills/osm.py
+++ b/dimos/agents/skills/osm.py
@@ -13,19 +13,19 @@
# limitations under the License.
-from dimos.core.skill_module import SkillModule
+from dimos.agents.annotation import skill
+from dimos.core.module import Module
from dimos.core.stream import In
from dimos.mapping.osm.current_location_map import CurrentLocationMap
from dimos.mapping.types import LatLon
from dimos.mapping.utils.distance import distance_in_meters
from dimos.models.vl.qwen import QwenVlModel
-from dimos.protocol.skill.skill import skill
from dimos.utils.logging_config import setup_logger
logger = setup_logger()
-class OsmSkill(SkillModule):
+class OsmSkill(Module):
_latest_location: LatLon | None
_current_location_map: CurrentLocationMap
@@ -46,7 +46,7 @@ def stop(self) -> None:
def _on_gps_location(self, location: LatLon) -> None:
self._latest_location = location
- @skill()
+ @skill
def map_query(self, query_sentence: str) -> str:
"""This skill uses a vision language model to find something on the map
based on the query sentence. You can query it with something like "Where
diff --git a/dimos/agents/skills/person_follow.py b/dimos/agents/skills/person_follow.py
index 0d4420632c..641055e6f6 100644
--- a/dimos/agents/skills/person_follow.py
+++ b/dimos/agents/skills/person_follow.py
@@ -12,16 +12,19 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-from threading import Event, RLock
+from threading import Event, RLock, Thread
import time
from typing import TYPE_CHECKING
+from langchain_core.messages import HumanMessage
import numpy as np
from reactivex.disposable import Disposable
+from dimos.agents.agent import AgentSpec
+from dimos.agents.annotation import skill
from dimos.core.core import rpc
from dimos.core.global_config import GlobalConfig
-from dimos.core.skill_module import SkillModule
+from dimos.core.module import Module
from dimos.core.stream import In, Out
from dimos.models.qwen.video_query import BBox
from dimos.models.segmentation.edge_tam import EdgeTAMProcessor
@@ -31,7 +34,6 @@
from dimos.navigation.visual.query import get_object_bbox_from_image
from dimos.navigation.visual_servoing.detection_navigation import DetectionNavigation
from dimos.navigation.visual_servoing.visual_servoing_2d import VisualServoing2D
-from dimos.protocol.skill.skill import skill
from dimos.utils.logging_config import setup_logger
if TYPE_CHECKING:
@@ -40,7 +42,7 @@
logger = setup_logger()
-class PersonFollowSkillContainer(SkillModule):
+class PersonFollowSkillContainer(Module):
"""Skill container for following a person.
This skill uses:
@@ -54,28 +56,30 @@ class PersonFollowSkillContainer(SkillModule):
global_map: In[PointCloud2]
cmd_vel: Out[Twist]
+ _agent_spec: AgentSpec
_frequency: float = 20.0 # Hz - control loop frequency
_max_lost_frames: int = 15 # number of frames to wait before declaring person lost
def __init__(
self,
camera_info: CameraInfo,
- global_config: GlobalConfig,
+ cfg: GlobalConfig,
use_3d_navigation: bool = False,
) -> None:
super().__init__()
- self._global_config: GlobalConfig = global_config
+ self._global_config: GlobalConfig = cfg
self._use_3d_navigation: bool = use_3d_navigation
self._latest_image: Image | None = None
self._latest_pointcloud: PointCloud2 | None = None
self._vl_model: VlModel = QwenVlModel()
self._tracker: EdgeTAMProcessor | None = None
+ self._thread: Thread | None = None
self._should_stop: Event = Event()
self._lock = RLock()
# Use MuJoCo camera intrinsics in simulation mode
if self._global_config.simulation:
- from dimos.robot.unitree_webrtc.mujoco_connection import MujocoConnection
+ from dimos.robot.unitree.mujoco_connection import MujocoConnection
camera_info = MujocoConnection.camera_info_static
@@ -102,7 +106,7 @@ def stop(self) -> None:
self._vl_model.stop()
super().stop()
- @skill()
+ @skill
def follow_person(self, query: str) -> str:
"""Follow a person matching the given description using visual servoing.
@@ -139,9 +143,9 @@ def follow_person(self, query: str) -> str:
if initial_bbox is None:
return f"Could not find '{query}' in the current view."
- return self._follow_loop(query, initial_bbox)
+ return self._follow_person(query, initial_bbox)
- @skill()
+ @skill
def stop_following(self) -> str:
"""Stop following the current person.
@@ -152,6 +156,10 @@ def stop_following(self) -> str:
self.cmd_vel.publish(Twist.zero())
+ if self._thread is not None:
+ self._thread.join(timeout=2)
+ self._thread = None
+
return "Stopped following."
def _on_color_image(self, image: Image) -> None:
@@ -162,7 +170,7 @@ def _on_pointcloud(self, pointcloud: PointCloud2) -> None:
with self._lock:
self._latest_pointcloud = pointcloud
- def _follow_loop(self, query: str, initial_bbox: BBox) -> str:
+ def _follow_person(self, query: str, initial_bbox: BBox) -> str:
x1, y1, x2, y2 = initial_bbox
box = np.array([x1, y1, x2, y2], dtype=np.float32)
@@ -186,6 +194,15 @@ def _follow_loop(self, query: str, initial_bbox: BBox) -> str:
logger.info(f"EdgeTAM initialized with {len(initial_detections)} detections")
+ self._thread = Thread(target=self._follow_loop, args=(tracker, query))
+ self._thread.start()
+
+ return (
+ "Found the person. Starting to follow. You can stop following by calling "
+ "the 'stop_following' tool."
+ )
+
+ def _follow_loop(self, tracker: EdgeTAMProcessor, query: str) -> None:
lost_count = 0
period = 1.0 / self._frequency
next_time = time.monotonic()
@@ -204,8 +221,8 @@ def _follow_loop(self, query: str, initial_bbox: BBox) -> str:
lost_count += 1
if lost_count > self._max_lost_frames:
- self.cmd_vel.publish(Twist.zero())
- return f"Lost track of '{query}'. Stopping."
+ self._send_stop_reason(query, "lost track of the person")
+ return
else:
lost_count = 0
best_detection = max(detections.detections, key=lambda d: d.bbox_2d_volume())
@@ -214,16 +231,16 @@ def _follow_loop(self, query: str, initial_bbox: BBox) -> str:
with self._lock:
pointcloud = self._latest_pointcloud
if pointcloud is None:
- self.cmd_vel.publish(Twist.zero())
- return "No pointcloud available for 3D navigation. Stopping."
+ self._send_stop_reason(query, "no pointcloud available for 3D navigation")
+ return
twist = self._detection_navigation.compute_twist_for_detection_3d(
pointcloud,
best_detection,
latest_image,
)
if twist is None:
- self.cmd_vel.publish(Twist.zero())
- return f"3D navigation failed for '{query}'. Stopping."
+ self._send_stop_reason(query, "3D navigation failed")
+ return
else:
twist = self._visual_servo.compute_twist(
best_detection.bbox,
@@ -236,12 +253,17 @@ def _follow_loop(self, query: str, initial_bbox: BBox) -> str:
if sleep_duration > 0:
time.sleep(sleep_duration)
- self.cmd_vel.publish(Twist.zero())
- return "Stopped following as requested."
+ self._send_stop_reason(query, "it was requested to stop following")
def _stop_following(self) -> None:
self._should_stop.set()
+ def _send_stop_reason(self, query: str, reason: str) -> None:
+ self.cmd_vel.publish(Twist.zero())
+ message = f"Person follow stopped for '{query}'. Reason: {reason}."
+ self._agent_spec.add_message(HumanMessage(message))
+ logger.info("Person follow stopped", query=query, reason=reason)
+
person_follow_skill = PersonFollowSkillContainer.blueprint
diff --git a/dimos/agents/skills/speak_skill.py b/dimos/agents/skills/speak_skill.py
index 073dda656a..aa06d30ba4 100644
--- a/dimos/agents/skills/speak_skill.py
+++ b/dimos/agents/skills/speak_skill.py
@@ -17,9 +17,9 @@
from reactivex import Subject
+from dimos.agents.annotation import skill
from dimos.core.core import rpc
-from dimos.core.skill_module import SkillModule
-from dimos.protocol.skill.skill import skill
+from dimos.core.module import Module
from dimos.stream.audio.node_output import SounddeviceAudioOutput
from dimos.stream.audio.tts.node_openai import OpenAITTSNode, Voice
from dimos.utils.logging_config import setup_logger
@@ -27,7 +27,7 @@
logger = setup_logger()
-class SpeakSkill(SkillModule):
+class SpeakSkill(Module):
_tts_node: OpenAITTSNode | None = None
_audio_output: SounddeviceAudioOutput | None = None
_audio_lock: threading.Lock = threading.Lock()
@@ -49,7 +49,7 @@ def stop(self) -> None:
self._audio_output = None
super().stop()
- @skill()
+ @skill
def speak(self, text: str) -> str:
"""Speak text out loud through the robot's speakers.
diff --git a/dimos/agents/skills/test_google_maps_skill_container.py b/dimos/agents/skills/test_google_maps_skill_container.py
index 0af206fbb1..84da91e886 100644
--- a/dimos/agents/skills/test_google_maps_skill_container.py
+++ b/dimos/agents/skills/test_google_maps_skill_container.py
@@ -14,34 +14,83 @@
import re
+from langchain_core.messages import HumanMessage
+import pytest
+
+from dimos.agents.skills.google_maps_skill_container import GoogleMapsSkillContainer
+from dimos.core.module import Module
+from dimos.core.stream import Out
from dimos.mapping.google_maps.types import Coordinates, LocationContext, Position
from dimos.mapping.types import LatLon
-def test_where_am_i(create_google_maps_agent, google_maps_skill_container) -> None:
- google_maps_skill_container._latest_location = LatLon(lat=37.782654, lon=-122.413273)
- google_maps_skill_container._client.get_location_context.return_value = LocationContext(
- street="Bourbon Street", coordinates=Coordinates(lat=37.782654, lon=-122.413273)
- )
- agent = create_google_maps_agent(fixture="test_where_am_i.json")
+class FakeGPS(Module):
+ """Provides a gps_location output so GoogleMapsSkillContainer's input port gets a transport."""
+
+ gps_location: Out[LatLon]
+
+
+class FakeLocationClient:
+ def get_location_context(self, location, radius=200):
+ return LocationContext(
+ street="Bourbon Street",
+ coordinates=Coordinates(lat=37.782654, lon=-122.413273),
+ )
+
+
+class MockedWhereAmISkill(GoogleMapsSkillContainer):
+ def __init__(self):
+ Module.__init__(self) # Skip GoogleMapsSkillContainer's __init__.
+ self._client = FakeLocationClient()
+ self._latest_location = LatLon(lat=37.782654, lon=-122.413273)
+ self._started = True
+ self._max_valid_distance = 20000
- response = agent.query("what street am I on")
- assert "bourbon" in response.lower()
+class FakePositionClient:
+ def __init__(self):
+ self._positions = iter(
+ [
+ Position(lat=37.782601, lon=-122.413201, description="address 1"),
+ Position(lat=37.782602, lon=-122.413202, description="address 2"),
+ Position(lat=37.782603, lon=-122.413203, description="address 3"),
+ ]
+ )
+ def get_position(self, query, location):
+ return next(self._positions)
-def test_get_gps_position_for_queries(
- create_google_maps_agent, google_maps_skill_container
-) -> None:
- google_maps_skill_container._latest_location = LatLon(lat=37.782654, lon=-122.413273)
- google_maps_skill_container._client.get_position.side_effect = [
- Position(lat=37.782601, lon=-122.413201, description="address 1"),
- Position(lat=37.782602, lon=-122.413202, description="address 2"),
- Position(lat=37.782603, lon=-122.413203, description="address 3"),
- ]
- agent = create_google_maps_agent(fixture="test_get_gps_position_for_queries.json")
- response = agent.query("what are the lat/lon for hyde park, regent park, russell park?")
+class MockedPositionSkill(GoogleMapsSkillContainer):
+ def __init__(self):
+ Module.__init__(self)
+ self._client = FakePositionClient()
+ self._latest_location = LatLon(lat=37.782654, lon=-122.413273)
+ self._started = True
+ self._max_valid_distance = 20000
+
+
+@pytest.mark.integration
+def test_where_am_i(agent_setup) -> None:
+ history = agent_setup(
+ blueprints=[FakeGPS.blueprint(), MockedWhereAmISkill.blueprint()],
+ messages=[HumanMessage("What street am I on? Use the where_am_i tool.")],
+ )
+
+ assert "bourbon" in history[-1].content.lower()
+
+
+@pytest.mark.integration
+def test_get_gps_position_for_queries(agent_setup) -> None:
+ history = agent_setup(
+ blueprints=[FakeGPS.blueprint(), MockedPositionSkill.blueprint()],
+ messages=[
+ HumanMessage(
+ "What are the lat/lon for hyde park, regent park, russell park? "
+ "Use the get_gps_position_for_queries tool."
+ )
+ ],
+ )
regex = r".*37\.782601.*122\.413201.*37\.782602.*122\.413202.*37\.782603.*122\.413203.*"
- assert re.match(regex, response, re.DOTALL)
+ assert re.match(regex, history[-1].content, re.DOTALL)
diff --git a/dimos/agents/skills/test_gps_nav_skills.py b/dimos/agents/skills/test_gps_nav_skills.py
index ab0d1ec318..afcb4d36d0 100644
--- a/dimos/agents/skills/test_gps_nav_skills.py
+++ b/dimos/agents/skills/test_gps_nav_skills.py
@@ -12,47 +12,57 @@
# See the License for the specific language governing permissions and
# limitations under the License.
+from langchain_core.messages import HumanMessage
+import pytest
+from dimos.agents.skills.gps_nav_skill import GpsNavSkillContainer
+from dimos.core.module import Module
+from dimos.core.stream import Out
from dimos.mapping.types import LatLon
-def test_set_gps_travel_points(create_gps_nav_agent, gps_nav_skill_container, mocker) -> None:
- gps_nav_skill_container._latest_location = LatLon(lat=37.782654, lon=-122.413273)
- gps_nav_skill_container._set_gps_travel_goal_points = mocker.Mock()
- agent = create_gps_nav_agent(fixture="test_set_gps_travel_points.json")
+class FakeGPS(Module):
+ """Provides a gps_location output so GpsNavSkillContainer's input port gets a transport."""
- agent.query("go to lat: 37.782654, lon: -122.413273")
+ gps_location: Out[LatLon]
- gps_nav_skill_container._set_gps_travel_goal_points.assert_called_once_with(
- [LatLon(lat=37.782654, lon=-122.413273)]
- )
- gps_nav_skill_container.gps_goal.publish.assert_called_once_with(
- [LatLon(lat=37.782654, lon=-122.413273)]
- )
+class MockedGpsNavSkill(GpsNavSkillContainer):
+ def __init__(self):
+ Module.__init__(self)
+ self._latest_location = LatLon(lat=37.782654, lon=-122.413273)
+ self._started = True
+ self._max_valid_distance = 50000
-def test_set_gps_travel_points_multiple(
- create_gps_nav_agent, gps_nav_skill_container, mocker
-) -> None:
- gps_nav_skill_container._latest_location = LatLon(lat=37.782654, lon=-122.413273)
- gps_nav_skill_container._set_gps_travel_goal_points = mocker.Mock()
- agent = create_gps_nav_agent(fixture="test_set_gps_travel_points_multiple.json")
- agent.query(
- "go to lat: 37.782654, lon: -122.413273, then 37.782660,-122.413260, and then 37.782670,-122.413270"
+@pytest.mark.integration
+def test_set_gps_travel_points(agent_setup) -> None:
+ history = agent_setup(
+ blueprints=[FakeGPS.blueprint(), MockedGpsNavSkill.blueprint()],
+ messages=[
+ HumanMessage(
+ 'Set GPS travel points to [{"lat": 37.782654, "lon": -122.413273}]. '
+ "Use the set_gps_travel_points tool."
+ )
+ ],
)
- gps_nav_skill_container._set_gps_travel_goal_points.assert_called_once_with(
- [
- LatLon(lat=37.782654, lon=-122.413273),
- LatLon(lat=37.782660, lon=-122.413260),
- LatLon(lat=37.782670, lon=-122.413270),
- ]
- )
- gps_nav_skill_container.gps_goal.publish.assert_called_once_with(
- [
- LatLon(lat=37.782654, lon=-122.413273),
- LatLon(lat=37.782660, lon=-122.413260),
- LatLon(lat=37.782670, lon=-122.413270),
- ]
+ assert "success" in history[-1].content.lower()
+
+
+@pytest.mark.integration
+def test_set_gps_travel_points_multiple(agent_setup) -> None:
+ history = agent_setup(
+ blueprints=[FakeGPS.blueprint(), MockedGpsNavSkill.blueprint()],
+ messages=[
+ HumanMessage(
+ "Set GPS travel points to these locations in order: "
+ '{"lat": 37.782654, "lon": -122.413273}, '
+ '{"lat": 37.782660, "lon": -122.413260}, '
+ '{"lat": 37.782670, "lon": -122.413270}. '
+ "Use the set_gps_travel_points tool."
+ )
+ ],
)
+
+ assert "success" in history[-1].content.lower()
diff --git a/dimos/agents/skills/test_navigation.py b/dimos/agents/skills/test_navigation.py
index 67e0429cb5..91737ada77 100644
--- a/dimos/agents/skills/test_navigation.py
+++ b/dimos/agents/skills/test_navigation.py
@@ -12,84 +12,105 @@
# See the License for the specific language governing permissions and
# limitations under the License.
+from langchain_core.messages import HumanMessage
import pytest
-from dimos.msgs.geometry_msgs import PoseStamped, Vector3
-from dimos.utils.transform_utils import euler_to_quaternion
+from dimos.agents.skills.navigation import NavigationSkillContainer
+from dimos.core.module import Module
+from dimos.core.stream import Out
+from dimos.msgs.geometry_msgs import PoseStamped
+from dimos.msgs.sensor_msgs import Image
-def test_stop_movement(create_navigation_agent, navigation_skill_container, mocker) -> None:
- cancel_goal_mock = mocker.Mock()
- stop_exploration_mock = mocker.Mock()
- navigation_skill_container._bound_rpc_calls["NavigationInterface.cancel_goal"] = (
- cancel_goal_mock
- )
- navigation_skill_container._bound_rpc_calls["WavefrontFrontierExplorer.stop_exploration"] = (
- stop_exploration_mock
- )
- agent = create_navigation_agent(fixture="test_stop_movement.json")
+class FakeCamera(Module):
+ color_image: Out[Image]
+
+
+class FakeOdom(Module):
+ odom: Out[PoseStamped]
+
+
+class MockedStopNavSkill(NavigationSkillContainer):
+ rpc_calls: list[str] = []
+
+ def __init__(self):
+ Module.__init__(self)
+ self._skill_started = True
+
+ def _cancel_goal_and_stop(self):
+ pass
+
+
+class MockedExploreNavSkill(NavigationSkillContainer):
+ rpc_calls: list[str] = []
+
+ def __init__(self):
+ Module.__init__(self)
+ self._skill_started = True
+
+ def _start_exploration(self, timeout):
+ return "Exploration completed successfuly"
+
+ def _cancel_goal_and_stop(self):
+ pass
+
- agent.query("stop")
+class MockedSemanticNavSkill(NavigationSkillContainer):
+ rpc_calls: list[str] = []
- cancel_goal_mock.assert_called_once_with()
- stop_exploration_mock.assert_called_once_with()
+ def __init__(self):
+ Module.__init__(self)
+ self._skill_started = True
+
+ def _navigate_by_tagged_location(self, query):
+ return None
+
+ def _navigate_to_object(self, query):
+ return None
+
+ def _navigate_using_semantic_map(self, query):
+ return f"Successfuly arrived at '{query}'"
@pytest.mark.integration
-def test_take_a_look_around(create_navigation_agent, navigation_skill_container, mocker) -> None:
- explore_mock = mocker.Mock()
- is_exploration_active_mock = mocker.Mock()
- navigation_skill_container._bound_rpc_calls["WavefrontFrontierExplorer.explore"] = explore_mock
- navigation_skill_container._bound_rpc_calls[
- "WavefrontFrontierExplorer.is_exploration_active"
- ] = is_exploration_active_mock
- mocker.patch("dimos.agents.skills.navigation.time.sleep")
- agent = create_navigation_agent(fixture="test_take_a_look_around.json")
-
- agent.query("take a look around for 10 seconds")
-
- explore_mock.assert_called_once_with()
-
-
-def test_go_to_semantic_location(
- create_navigation_agent, navigation_skill_container, mocker
-) -> None:
- mocker.patch(
- "dimos.agents.skills.navigation.NavigationSkillContainer._navigate_by_tagged_location",
- return_value=None,
- )
- mocker.patch(
- "dimos.agents.skills.navigation.NavigationSkillContainer._navigate_to_object",
- return_value=None,
+def test_stop_movement(agent_setup) -> None:
+ history = agent_setup(
+ blueprints=[
+ FakeCamera.blueprint(),
+ FakeOdom.blueprint(),
+ MockedStopNavSkill.blueprint(),
+ ],
+ messages=[HumanMessage("Stop moving. Use the stop_movement tool.")],
)
- navigate_to_mock = mocker.patch(
- "dimos.agents.skills.navigation.NavigationSkillContainer._navigate_to",
- return_value=True,
- )
- query_by_text_mock = mocker.Mock(
- return_value=[
- {
- "distance": 0.5,
- "metadata": [
- {
- "pos_x": 1,
- "pos_y": 2,
- "rot_z": 3,
- }
- ],
- }
- ]
+
+ assert "stopped" in history[-1].content.lower()
+
+
+@pytest.mark.integration
+def test_start_exploration(agent_setup) -> None:
+ history = agent_setup(
+ blueprints=[
+ FakeCamera.blueprint(),
+ FakeOdom.blueprint(),
+ MockedExploreNavSkill.blueprint(),
+ ],
+ messages=[
+ HumanMessage("Take a look around for 10 seconds. Use the start_exploration tool.")
+ ],
)
- navigation_skill_container._bound_rpc_calls["SpatialMemory.query_by_text"] = query_by_text_mock
- agent = create_navigation_agent(fixture="test_go_to_semantic_location.json")
-
- agent.query("go to the bookshelf")
-
- query_by_text_mock.assert_called_once_with("bookshelf")
- navigate_to_mock.assert_called_once_with(
- PoseStamped(
- position=Vector3(1, 2, 0),
- orientation=euler_to_quaternion(Vector3(0, 0, 3)),
- frame_id="world",
- ),
+
+ assert "explor" in history[-1].content.lower()
+
+
+@pytest.mark.integration
+def test_go_to_semantic_location(agent_setup) -> None:
+ history = agent_setup(
+ blueprints=[
+ FakeCamera.blueprint(),
+ FakeOdom.blueprint(),
+ MockedSemanticNavSkill.blueprint(),
+ ],
+ messages=[HumanMessage("Go to the bookshelf. Use the navigate_with_text tool.")],
)
+
+ assert "success" in history[-1].content.lower()
diff --git a/dimos/agents/skills/test_unitree_skill_container.py b/dimos/agents/skills/test_unitree_skill_container.py
index 29dfade979..ea1cfba5cf 100644
--- a/dimos/agents/skills/test_unitree_skill_container.py
+++ b/dimos/agents/skills/test_unitree_skill_container.py
@@ -12,21 +12,35 @@
# See the License for the specific language governing permissions and
# limitations under the License.
+import difflib
-def test_pounce(mocker, create_unitree_skills_agent, unitree_skills) -> None:
- agent = create_unitree_skills_agent(fixture="test_pounce.json")
- publish_request_mock = mocker.Mock()
- unitree_skills.get_rpc_calls = mocker.Mock(return_value=publish_request_mock)
+from langchain_core.messages import HumanMessage
+import pytest
- response = agent.query("pounce")
+from dimos.robot.unitree.unitree_skill_container import _UNITREE_COMMANDS, UnitreeSkillContainer
- assert "front pounce" in response.lower()
- publish_request_mock.assert_called_once_with("rt/api/sport/request", {"api_id": 1032})
+class MockedUnitreeSkill(UnitreeSkillContainer):
+ rpc_calls: list[str] = []
-def test_did_you_mean(mocker, unitree_skills) -> None:
- unitree_skills.get_rpc_calls = mocker.Mock()
- assert (
- unitree_skills.execute_sport_command("Pounce")
- == "There's no 'Pounce' command. Did you mean: ['FrontPounce', 'Pose']"
+ def __init__(self):
+ super().__init__()
+ # Provide a fake RPC so the real execute_sport_command runs end-to-end.
+ self._bound_rpc_calls["GO2Connection.publish_request"] = lambda *args, **kwargs: None
+
+
+@pytest.mark.integration
+def test_pounce(agent_setup) -> None:
+ history = agent_setup(
+ blueprints=[MockedUnitreeSkill.blueprint()],
+ messages=[HumanMessage("Pounce! Use the execute_sport_command tool.")],
)
+
+ response = history[-1].content.lower()
+ assert "pounce" in response
+
+
+def test_did_you_mean() -> None:
+ suggestions = difflib.get_close_matches("Pounce", _UNITREE_COMMANDS.keys(), n=3, cutoff=0.6)
+ assert "FrontPounce" in suggestions
+ assert "Pose" in suggestions
diff --git a/dimos/agents/spec.py b/dimos/agents/spec.py
deleted file mode 100644
index b0a0324e89..0000000000
--- a/dimos/agents/spec.py
+++ /dev/null
@@ -1,242 +0,0 @@
-# Copyright 2025-2026 Dimensional Inc.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-"""Base agent module that wraps BaseAgent for DimOS module usage."""
-
-from abc import ABC, abstractmethod
-from dataclasses import dataclass, field
-from enum import Enum
-from typing import TYPE_CHECKING, Any, Union
-
-if TYPE_CHECKING:
- from dimos.protocol.skill.skill import SkillContainer
-
-from langchain.chat_models.base import _SUPPORTED_PROVIDERS
-from langchain_core.language_models.chat_models import BaseChatModel
-from langchain_core.messages import (
- AIMessage,
- HumanMessage,
- SystemMessage,
- ToolMessage,
-)
-from rich.console import Console
-from rich.table import Table
-from rich.text import Text
-
-from dimos.core import Module, rpc
-from dimos.core.module import ModuleConfig
-from dimos.protocol.pubsub import PubSub, lcm # type: ignore[attr-defined]
-from dimos.protocol.service import Service # type: ignore[attr-defined]
-from dimos.protocol.skill.skill import SkillContainer
-from dimos.utils.generic import truncate_display_string
-from dimos.utils.logging_config import setup_logger
-
-logger = setup_logger()
-
-
-# Dynamically create ModelProvider enum from LangChain's supported providers
-_providers = {provider.upper(): provider for provider in _SUPPORTED_PROVIDERS}
-Provider = Enum("Provider", _providers, type=str) # type: ignore[misc]
-
-
-class Model(str, Enum):
- """Common model names across providers.
-
- Note: This is not exhaustive as model names change frequently.
- Based on langchain's _attempt_infer_model_provider patterns.
- """
-
- # OpenAI models (prefix: gpt-3, gpt-4, o1, o3)
- GPT_4O = "gpt-4o"
- GPT_4O_MINI = "gpt-4o-mini"
- GPT_4_TURBO = "gpt-4-turbo"
- GPT_4_TURBO_PREVIEW = "gpt-4-turbo-preview"
- GPT_4 = "gpt-4"
- GPT_35_TURBO = "gpt-3.5-turbo"
- GPT_35_TURBO_16K = "gpt-3.5-turbo-16k"
- O1_PREVIEW = "o1-preview"
- O1_MINI = "o1-mini"
- O3_MINI = "o3-mini"
-
- # Anthropic models (prefix: claude)
- CLAUDE_3_OPUS = "claude-3-opus-20240229"
- CLAUDE_3_SONNET = "claude-3-sonnet-20240229"
- CLAUDE_3_HAIKU = "claude-3-haiku-20240307"
- CLAUDE_35_SONNET = "claude-3-5-sonnet-20241022"
- CLAUDE_35_SONNET_LATEST = "claude-3-5-sonnet-latest"
- CLAUDE_3_7_SONNET = "claude-3-7-sonnet-20250219"
-
- # Google models (prefix: gemini)
- GEMINI_20_FLASH = "gemini-2.0-flash"
- GEMINI_15_PRO = "gemini-1.5-pro"
- GEMINI_15_FLASH = "gemini-1.5-flash"
- GEMINI_10_PRO = "gemini-1.0-pro"
-
- # Amazon Bedrock models (prefix: amazon)
- AMAZON_TITAN_EXPRESS = "amazon.titan-text-express-v1"
- AMAZON_TITAN_LITE = "amazon.titan-text-lite-v1"
-
- # Cohere models (prefix: command)
- COMMAND_R_PLUS = "command-r-plus"
- COMMAND_R = "command-r"
- COMMAND = "command"
- COMMAND_LIGHT = "command-light"
-
- # Fireworks models (prefix: accounts/fireworks)
- FIREWORKS_LLAMA_V3_70B = "accounts/fireworks/models/llama-v3-70b-instruct"
- FIREWORKS_MIXTRAL_8X7B = "accounts/fireworks/models/mixtral-8x7b-instruct"
-
- # Mistral models (prefix: mistral)
- MISTRAL_LARGE = "mistral-large"
- MISTRAL_MEDIUM = "mistral-medium"
- MISTRAL_SMALL = "mistral-small"
- MIXTRAL_8X7B = "mixtral-8x7b"
- MIXTRAL_8X22B = "mixtral-8x22b"
- MISTRAL_7B = "mistral-7b"
-
- # DeepSeek models (prefix: deepseek)
- DEEPSEEK_CHAT = "deepseek-chat"
- DEEPSEEK_CODER = "deepseek-coder"
- DEEPSEEK_R1_DISTILL_LLAMA_70B = "deepseek-r1-distill-llama-70b"
-
- # xAI models (prefix: grok)
- GROK_1 = "grok-1"
- GROK_2 = "grok-2"
-
- # Perplexity models (prefix: sonar)
- SONAR_SMALL_CHAT = "sonar-small-chat"
- SONAR_MEDIUM_CHAT = "sonar-medium-chat"
- SONAR_LARGE_CHAT = "sonar-large-chat"
-
- # Meta Llama models (various providers)
- LLAMA_3_70B = "llama-3-70b"
- LLAMA_3_8B = "llama-3-8b"
- LLAMA_31_70B = "llama-3.1-70b"
- LLAMA_31_8B = "llama-3.1-8b"
- LLAMA_33_70B = "llama-3.3-70b"
- LLAMA_2_70B = "llama-2-70b"
- LLAMA_2_13B = "llama-2-13b"
- LLAMA_2_7B = "llama-2-7b"
-
-
-@dataclass
-class AgentConfig(ModuleConfig):
- system_prompt: str | SystemMessage | None = None
- skills: SkillContainer | list[SkillContainer] | None = None
-
- # we can provide model/provvider enums or instantiated model_instance
- model: Model = Model.GPT_4O
- provider: Provider = Provider.OPENAI # type: ignore[attr-defined]
- model_instance: BaseChatModel | None = None
-
- agent_transport: type[PubSub] = lcm.PickleLCM # type: ignore[type-arg]
- agent_topic: Any = field(default_factory=lambda: lcm.Topic("/agent"))
-
-
-AnyMessage = Union[SystemMessage, ToolMessage, AIMessage, HumanMessage]
-
-
-class AgentSpec(Service[AgentConfig], Module, ABC):
- default_config: type[AgentConfig] = AgentConfig
-
- def __init__(self, *args, **kwargs) -> None: # type: ignore[no-untyped-def]
- Service.__init__(self, *args, **kwargs)
- Module.__init__(self, *args, **kwargs)
-
- if self.config.agent_transport:
- self.transport = self.config.agent_transport()
-
- def publish(self, msg: AnyMessage) -> None:
- if self.transport:
- self.transport.publish(self.config.agent_topic, msg)
-
- def start(self) -> None:
- super().start()
-
- def stop(self) -> None:
- if hasattr(self, "transport") and self.transport:
- self.transport.stop() # type: ignore[attr-defined]
- self.transport = None # type: ignore[assignment]
- super().stop()
-
- @rpc
- @abstractmethod
- def clear_history(self): ... # type: ignore[no-untyped-def]
-
- @abstractmethod
- def append_history(self, *msgs: list[AIMessage | HumanMessage]): ... # type: ignore[no-untyped-def]
-
- @abstractmethod
- def history(self) -> list[AnyMessage]: ...
-
- @rpc
- @abstractmethod
- def register_skills(
- self, container: "SkillContainer", run_implicit_name: str | None = None
- ) -> None: ...
-
- @rpc
- @abstractmethod
- def query(self, query: str): ... # type: ignore[no-untyped-def]
-
- def __str__(self) -> str:
- console = Console(force_terminal=True, legacy_windows=False)
- table = Table(show_header=True)
-
- table.add_column("Message Type", style="cyan", no_wrap=True)
- table.add_column("Content")
-
- for message in self.history():
- if isinstance(message, HumanMessage):
- content = message.content
- if not isinstance(content, str):
- content = "