Conversation
Add support for the Boreas (outdoor autonomous driving) and BoreasRT datasets from the University of Toronto, downloadable from the public S3 bucket s3://boreas via boto3. Implements lidar (.bin float32), IMU (Applanix CSV for Boreas, Silicon Sensing DMU41 for BoreasRT), and ground truth (Euler-angle poses converted to SE3) loaders.
Contributor
There was a problem hiding this comment.
Pull request overview
Adds FoMo dataset support (including synthetic orientation for GT visualization) and extends dataset/pointcloud loading utilities to handle datasets with non-trivial LiDAR row ordering.
Changes:
- Introduces
Fomodataset loader with S3 download logic and synthetic GT orientation via a newTrajectory.smooth_orientation()helper. - Adds new C++ helpers for Boreas/FoMo
.binformats and a genericfill_col_by_maprow-ordering mechanism (and updates BotanicGarden to use it). - Extends SE(3)/SO(3) utilities and test suite (new
SO3.from_rpy/from_ypr, new trajectory tests, updated test data generation).
Reviewed changes
Copilot reviewed 16 out of 27 changed files in this pull request and generated 14 comments.
Show a summary per file
| File | Description |
|---|---|
| uv.lock | Adds locked dependencies for awscli and its transitive deps. |
| pyproject.toml | Adds awscli as a runtime dependency. |
| python/evalio/types/base.py | Adds Trajectory.smooth_orientation() for synthetic pose orientations. |
| python/evalio/datasets/fomo.py | New FoMo dataset implementation + S3 download via AWS CLI. |
| python/evalio/datasets/botanic_garden.py | Switches row/column filling to new fill_col_by_map mapping approach. |
| python/evalio/datasets/boreas.py | Adds Boreas dataset loader and S3 download via AWS CLI. |
| python/evalio/datasets/boreas_rt.py | Adds BoreasRT dataset loader and S3 download via AWS CLI. |
| python/evalio/datasets/multi_campus.py | Adjusts IMU noise parameters and comments. |
| python/evalio/datasets/init.py | Registers Boreas, BoreasRT, and Fomo datasets. |
| python/evalio/cli/completions.py | Adds dataset name/* patterns to CLI completions. |
| cpp/evalio/types.h | Adds Duration unary negation and new SO3 constructors. |
| cpp/bindings/types.h | Exposes SO3.from_rpy / SO3.from_ypr to Python. |
| cpp/bindings/ros_pc2.h | Replaces specialized row-order logic with fill_col_by_map; adds Boreas/FoMo bin loaders. |
| tests/utils.py | Strengthens LiDAR equality check by validating point count. |
| tests/test_trajectory.py | Adds tests for Trajectory.smooth_orientation(). |
| tests/test_so3.py | Adds tests for SO3.from_rpy/from_ypr. |
| tests/make_test_data.py | Avoids regenerating cached test pickles if they already exist. |
| tests/data/imu_multi_campus.pkl | Adds/updates IMU fixture pickle. |
| tests/data/imu_fomo.pkl | Adds FoMo IMU fixture pickle. |
| tests/data/imu_cumulti.pkl | Adds/updates IMU fixture pickle. |
| tests/data/imu_boreas.pkl | Adds Boreas IMU fixture pickle. |
| tests/data/imu_boreas_rt.pkl | Adds BoreasRT IMU fixture pickle. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
Comment on lines
63
to
72
| Duration operator-(const Duration& other) const { | ||
| return Duration::from_nsec(nsec - other.nsec); | ||
| } | ||
|
|
||
| Duration operator-() const { | ||
| return Duration::from_nsec(-nsec); | ||
| } | ||
|
|
||
| Duration operator+(const Duration& other) const { | ||
| return Duration::from_nsec(nsec + other.nsec); |
Comment on lines
+237
to
+265
| def download(self): | ||
| from subprocess import Popen, PIPE, run | ||
| from tqdm import tqdm | ||
| import sys | ||
| # NOTE: This is experimental; not sure if should use aws cli or boto3 to download from S3 | ||
| # boto3 is slower for all those tiny lidar files | ||
|
|
||
| # Find the AWS cli installed in the same environment as evalio | ||
| aws = Path(sys.prefix) / "bin" / "aws" | ||
|
|
||
| seq = "boreas-" + self.seq_name.replace("_", "-").split("-", 1)[1] | ||
|
|
||
| # Figure out how many total files they are | ||
| output = run( | ||
| [ | ||
| str(aws), | ||
| "s3", | ||
| "ls", | ||
| "--no-sign-request", | ||
| f"s3://boreas/{seq}/lidar/", | ||
| "--summarize", | ||
| ], | ||
| capture_output=True, | ||
| text=True, | ||
| ) | ||
| # get the last few lines of aws CLI output to show total file count and size | ||
| count = int(output.stdout.splitlines()[-2].split(":")[1].strip()) | ||
| count += 2 | ||
|
|
Comment on lines
8
to
13
| requires-python = ">=3.11" | ||
| dependencies = [ | ||
| "asteval>=1.0.6", | ||
| "awscli>=1.44.58", | ||
| "distinctipy>=1.3.4", | ||
| "gdown>=5.2.0", |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the FoMo dataset.
It's missing GT orientation so we add some helpers to produce plausible orientation values. Not useful for evaluation, but helps orient things a bit for plotting.
Should be merged after #67