SmartRobot

SmartRobot is a Python-based robotic control system designed to operate an SO-101 robotic arm using natural language instructions interpreted by Large Language Models (LLMs). It features a modern web interface for manual monitoring, semi-autonomous operation, and API configuration.

Features

• Natural Language Operation: Instruct the robot using plain English (e.g., "Pick up the red ball"). The LLM handles the step-by-step movement planning.
• Visual Feedback Loop: The system feeds live "Front" and "Overhead" camera frames to the LLM, enabling visual reasoning, depth triangulation, and progress tracking.
• Multi-Model Support: Swap between different LLMs on the fly. It supports ChatGPT Codex (OAuth), OpenRouter, and custom OpenAI-compatible endpoints.
• Safety Constraints:
  • Movements are clamped to a safe maximum distance per step.
  • Hardware stall detection automatically pauses if the arm is physically blocked.
• Safe Positioning: A one-click "Reset Pos" feature safely and gradually zeroes all joints to return the arm to its resting home state.

Architecture

• Backend: FastAPI (Python) powers the API, serving the frontend, communicating with the LLMs, and running the background Sense-Think-Act control loop.
• Hardware Abstraction: Integrates with the lerobot library to communicate securely with the SO-101 arm hardware in an asynchronous, thread-safe manner.
• Frontend: A sleek, dark-themed single-page application (SPA) built with HTML, CSS, and Vanilla JavaScript. Real-time logging is handled via Server-Sent Events (SSE).

Prerequisites

• Docker and Docker Compose: Ensure you have Docker installed to build and run the containerized application.

• Hardware Requirements:

  • An SO-101 Robotic Arm connected to your machine. Ensure you know the mount point (e.g., /dev/ttyACM0 or /dev/ttyACM1).
  • A front-facing camera mounted on the robot's claw (e.g., /dev/video0).
  • An overhead camera providing a bird's-eye view of the workspace (e.g., /dev/video2).

  Note: You must edit docker-compose.yml to match the exact USB / Video paths on your own machine before starting the container.

Getting Started

1. Clone the Repository:

   git clone <repository-url>
   cd smartrobot

2. Configure your Hardware Paths (Important): Before running anything, open docker-compose.yml and modify the environment variables to match your exact setup.

   • Ensure ROBOT_PORT reflects your arm's USB port (e.g., /dev/ttyACM0).
   • You can also add ROBOT_ID: "my_robot_name" under the environment sections if you want to use a name other than the default levi_follower.
   • Update CAMERA_FRONT_PATH and CAMERA_OVER_PATH to point to your respective webcams (e.g., /dev/video0).

3. Build and Run the Server: Docker handles all the dependencies, including the required lerobot routines and graphics libraries.

   docker compose up -d --build

4. Access the Web Interface: Once the container is running smoothly, open your browser and navigate to:

   http://localhost:8888
   

Usage

1. Configure LLM Settings: Click "LLM Settings" in the top right to configure your API keys for OpenRouter or OpenAI-compatible models, or click "Connect LLM" to use Codex OAuth.
2. Connect the Robot: Click "Connect Robot" to initialize the hardware link to the arm.
3. Start Cameras: Click "Start Cameras" in the UI to begin streaming the front and overhead views.
4. Issue Commands: Type your instruction into the task input box and click "Start Inference". The robot will begin thinking and moving autonomously.
5. Resetting: If you need to quickly home the robot arm back to its starting position safely, click the "Reset Pos" button next to the robot connection controls.

Troubleshooting

• Robot won't connect: Ensure the arm is plugged in and accessible at the USB port defined in docker-compose.yml (ROBOT_PORT).
• Cameras won't start: Verify your camera hardware indices in docker-compose.yml (CAMERA_FRONT_PATH, CAMERA_OVER_PATH). If the feeds are flipped or upside-down, toggle the respective FLIP and ROTATE boolean flags in the config.
• LLM Disconnection Error: If clicking "Disconnect LLM" causes an error, it is likely due to the robot actively attempting to reach a target. Ensure you have halted the stream or task before disconnecting.