AutoGesture

AutoGesture is a tool for recording and labeling hand gestures using Ultraleap hand tracking and smartwatch sensor data.

Getting Started

The main entry point for the project is autogesture.py, which provides a graphical interface to manage gesture poses and launch the recording tool.

Prerequisites

• Ultraleap Leap Motion Controller 2 (or compatible device).
• Ultraleap Gemini Tracking Software.
• Compatible Smartwatch (for accelerometer, gyroscope, and PPG data).
• Python 3.10+

Installation

1. Install the required Python packages:

   pip install -r requirements.txt

   Note: Ensure the Ultraleap Python bindings are correctly installed as per the instructions in requirements.txt.

Project Workflow

1. Pose Management

Before recording, you need to define the hand poses you want to track. Run the main application:

python3 autogesture.py

• Add Pose: Click "Add Pose", enter a name and value, then use the calibration window to capture your hand position. Press s or l to save the pose.
• Load/Save Poses: You can save your collection of poses to a .json file for future use.
• Show/Hide Poses: View the numerical "pose vector" for each loaded gesture.

2. Recording Data

Once your poses are loaded, click "Start Recorder" in the main GUI. This launches the recording tool (libs/finger_tracking.py).

Recording Controls:

• c: Connect to the smartwatch via Bluetooth.
• r: Start recording data (video, hand poses, and watch sensors).
• s: Stop recording and save data to the recordings/ directory.
• x: Exit the recorder.
• j, k, l, Space: Manually label gestures during recording (Fist, Pinch, IndexTap, Resting).

3. Output Data

Each recording session creates a timestamped folder in recordings/ containing:

• recording.mp4: Video feed with hand tracking overlay.
• acc.csv, gyro.csv, ppg.csv: Sensor data from the smartwatch.
• poses.csv: Detected hand poses based on your loaded library.
• manual_poses.csv: Manual labels applied during the session.

CLI Tools

You can also run the underlying tools directly from the terminal:

Pose Calibration

Capture and save a single pose to a JSON file.

python3 libs/pose_calibration.py

Finger Tracking (Recorder)

Launch the recorder directly with a specific set of poses.

python3 libs/finger_tracking.py --path path/to/poses.json

Project Structure

• autogesture.py: Main GUI application.
• libs/: Core logic for tracking, calibration, and hardware integration.
• recordings/: Default directory for saved recording sessions.
• config.json: Configuration settings.