Skip to content

feat(android): Camera Intrinsics #3806

Open
pvedula7 wants to merge 2 commits into
mrousavy:mainfrom
pvedula7:feat/android-intriniscs
Open

feat(android): Camera Intrinsics #3806
pvedula7 wants to merge 2 commits into
mrousavy:mainfrom
pvedula7:feat/android-intriniscs

Conversation

@pvedula7
Copy link
Copy Markdown

What

Adds Android support for Frame.cameraIntrinsicMatrix on Frame Outputs.

Android now honors the existing enableCameraMatrixDelivery option. When enabled, each HybridFrame attempts to resolve camera intrinsics from Camera2 metadata and exposes them through the existing Frame.cameraIntrinsicMatrix property. No new JS API is added.

Implementation

On Android, intrinsics are resolved in this order:

  1. Per-frame Camera2 metadata:
    CaptureResult.LENS_INTRINSIC_CALIBRATION
  2. Static camera metadata fallback:
    CameraCharacteristics.LENS_INTRINSIC_CALIBRATION

Camera2 exposes intrinsics as five calibration values:

[fx, fy, cx, cy, s]

VisionCamera converts those into the existing 3x3 matrix shape:

[ fx  s  cx ]
[  0 fy  cy ]
[  0  0   1 ]

serialized consistently with the existing native matrix array convention.

The Android implementation also applies CameraX’s ImageInfo.sensorToBufferTransformMatrix before exposing the matrix, so the returned values are in the Frame/buffer coordinate space expected by Frame.cameraIntrinsicMatrix.

This PR intentionally does not use CaptureResult.STATISTICS_LENS_INTRINSICS_SAMPLES. That key is API 35+, optional, and represents multiple intra-frame lens intrinsics samples rather than a single frame-level matrix. The existing JS API exposes a single number[] intrinsic matrix on Frame, so those samples would require a different API shape to represent correctly.

Validation

Android Kotlin compile:
./gradlew :react-native-vision-camera:compileDebugKotlin :react-native-vision-camera-barcode-scanner:compileDebugKotlin

Typecheck:
npm run typecheck --workspace react-native-vision-camera

Manual Android validation on Pixel 6a:
enableCameraMatrixDelivery: true returns a 9-number matrix.
Returned cx/cy are transformed into frame coordinates.
Unsupported or initially unavailable metadata keeps streaming and returns undefined.

@vercel
Copy link
Copy Markdown

vercel Bot commented Apr 28, 2026

@pvedula7 is attempting to deploy a commit to the Margelo Team on Vercel.

A member of the Team first needs to authorize it.

@mrousavy
Copy link
Copy Markdown
Owner

Thanks this is great stuff!! I'll look into this soon, for one I don't like the two paths in createUseCase (1 path is simpler, less branching), and also I don't like the Camera2Interop in the "higher level" code, ideally we'd hide this behind an extension to keep it safer and easier to maintain.

@pvedula7
Copy link
Copy Markdown
Author

Thanks this is great stuff!! I'll look into this soon, for one I don't like the two paths in createUseCase (1 path is simpler, less branching), and also I don't like the Camera2Interop in the "higher level" code, ideally we'd hide this behind an extension to keep it safer and easier to maintain.

Makes sense, agreed. Originally tried avoiding threadinging CameraInfo through every output because some modules like barcode scanner do not need it. Moved it into NativeCameraOutput.Config with a nullable default, so the createUseCase(...) method signature stays unchanged.

Moved Camera2-specific logic into small extensions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants