-
Notifications
You must be signed in to change notification settings - Fork 7
Add animal pipeline support to FMPose3DInference (and add test suite) #16
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
C-Achard
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good ! Should be easy to use downstream, definitely great there is some testing.
Some minor comments for future usage but nothing major.
|
|
||
| @staticmethod | ||
| def _map_keypoints(xy: np.ndarray) -> np.ndarray: | ||
| """Map quadruped80K keypoints to Animal3D 26-joint layout. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A bit of a high-level comment, would it be useful to minimally document the nomenclature somewhere ? In regards to DLC and all its functionalities "Animal3D" may be a bit ambiguous (and making it more specific could help with LLM parsing dataset structures)
|
|
||
| @staticmethod | ||
| def _map_keypoints(xy: np.ndarray) -> np.ndarray: | ||
| """Map quadruped80K keypoints to Animal3D 26-joint layout. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
NB: I don't mean this for standards like COCO, just internal names/our own nomenclature
| ] | ||
|
|
||
|
|
||
| def compute_limb_regularization_matrix( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Seems primarily intended for visualization / canonicalization but a note is that it may not be suitable for all poses (e.g. lying down or jumping) ? As it assumes a stable “up” limb direction. It can also become unstable across frames (which could cause flickering in some cases) as there is no temporal information, only a per frame computation with no prior.
Probably worth documenting these limitations for people so they use the raw pose or come up with their own logic for their data if the assumptions are different in their case, but not critical for now.
| inspected to choose pipeline components. Adding a third pipeline | ||
| means adding one branch here (or turning this into a registry). | ||
| """ | ||
| if model_cfg.model_type == "fmpose3d_animals": |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would using an Enum here make sense to avoid raw str checks ?
Summary
FMPose3DInferencewith full animal (26-joint Animal3D) pipeline support,including
SuperAnimalEstimator(DeepLabCut 2D),AnimalPostProcessor(limbregularisation), and the
for_animals()convenience constructor.tests/test_fmpose3d.py) covering all public andinternal components of the inference API with 78 tests (64 unit + 14 functional).
Changes
fmpose3d/fmpose3d.pySuperAnimalEstimator: wraps DeepLabCut SuperAnimal, maps quadruped80K keypointsto the 26-joint Animal3D layout via
_map_keypoints, handles missing detections(zero-fill fallback).
AnimalPostProcessor: applies limb regularisation (Rodrigues rotation aligningaverage limb direction to vertical) instead of root-zeroing / camera_to_world.
compute_limb_regularization_matrixandapply_limb_regularizationhelpers._default_componentsresolver that selects the correct estimator + postprocessorbased on
model_type.FMPose3DInference.for_animals()class method for one-line animal pipeline setup.tests/test_fmpose3d.py(new)_map_keypoints(direct mapping, interpolation, out-of-range,multi-individual), limb regularisation (identity, rotation, 180-degree, degenerate,
valid-rotation properties), both postprocessors (shape, mutation, camera_rotation
handling, all-zero pose),
_default_components,FMPose3DInferenceconstruction(
for_animals, custom injection,_resolve_pad),_ingest_input(arrays, paths,directories, 8 error cases),
_load_weightserror paths,pose_3dinput validation(wrong ndim, zero frames, 4D→first person, TTA path, progress callback, seed
reproducibility), dataclasses, mocked
SuperAnimalEstimator.predict(no-detectionand valid-detection branches).
root-zeroed, world z-floor, finite, seed reproducibility) and full animal pipeline
(DLC 2D → FMPose3D-animals 3D, shape checks, finite, magnitude sanity,
reproducibility). Marked
@pytest.mark.functionaland auto-skipped when pretrainedweights are not present.
pyproject.tomlfunctionalpytest marker.