Open
Conversation
added 30 commits
March 13, 2026 21:02
Add support for Intel Image Processing Unit version 4 (IPU4), a specialized processor for computer vision and image processing tasks. This includes the ISYS (Image Input System) and PSYS (Processing System) components. The driver provides camera sensor input handling, video capture capabilities, and image processing pipeline management through the media controller framework. Source: https://github.com/intel/linux-intel-lts/tree/lts-v5.15.195-android_t-251103T063840Z/drivers/media/pci/intel
Introduce build system support for Intel IPU4 (Apollo Lake) and IPU4P (Ice Lake) imaging units. This adds the necessary Kconfig options to select between IPU generations and hardware platforms, and updates the Makefile to include the ipu4 directory in the build process.
Remove the external crlmodule header dependency from the test pattern generator. Replace with local control ID definitions to eliminate the dependency on external camera module drivers. The crlmodule is a legacy external camera driver library; this change allows IPU4 to operate independently without requiring it.
Remove #ifdef blocks for older kernel versions and legacy code paths that handled compatibility with kernel versions < 4.15. This simplification is valid for current kernel versions (>= 6.1). Also remove unused legacy features and conditional logic that is no longer needed in modern kernels.
simplify state handling Refactor the ISYS subdevice infrastructure to modernize state management and fix pixel order issues during cropping. The current implementation of format propagation relies on complex, manual pad interdependence checks (has_pad_interdep) and legacy routing logic. This patch replaces these custom checks with a streamlined propagation model that updates all source pads based on the sink configuration, aligning with modern V4L2 subdevice state handling. Key Changes: - Bayer Order Correction: Introduces a raw Bayer format lookup table and index-based calculation in the CSI-2 BE-SOC driver. This ensures that the MEDIA_BUS_FMT is correctly updated (e.g., swapping BGGR to GBRG) when an odd-numbered vertical crop is applied, preventing color corruption downstream. - Infrastructure Simplification: Removes has_pad_interdep and ipu_isys_subdev_has_route logic. The propagation logic now uniformly updates source pads, significantly reducing code complexity in ipu-isys-subdev.c. - V4L2 State Migration: Renames and updates v4l2_subdev_pad_config references to the modern v4l2_subdev_state API. - Cleanup: Removes deprecated #if blocks for IPU4/IPU4P and eliminates redundant conditional checks for n:n and 1:n pad mapping scenarios. This modernization improves maintainability and ensures hardware cropping behaves predictably across different Bayer patterns.
The current implementation uses find_vm_area() to retrieve the backing pages for DMA buffers. This is problematic because find_vm_area() is intended for kernel vmalloc-space management and does not provide a generic way to look up metadata based on a DMA handle (IOVA). Relying on it for DMA operations (mmap, get_sgtable) is brittle and restricts the driver to specific allocation types. Implement an internal VMA tracking mechanism in the IPU MMU driver to properly manage buffer metadata. This allows lookups by both virtual address and IOVA, ensuring that dma_map_ops callbacks can always find the original page list regardless of the memory's origin. - Add struct ipu_dma_vma to track IOVA, vaddr, size, and page pointers. - Guard the new vma_list with a mutex to handle concurrent allocations. - Update ipu_dma_alloc() to register allocations upon success. - Update ipu_dma_free(), mmap(), and get_sgtable() to use the new tracking list instead of find_vm_area(). This change removes the dependency on vmalloc internal helpers and ensures consistent behavior across different DMA mapping scenarios.
… API Rename CSI-2 specific fields in v4l2_mbus_frame_desc_entry to use more consistent naming: - .bus.csi2.channel -> .bus.csi2.vc (Virtual Channel) - .bus.csi2.data_type -> .bus.csi2.dt (Data Type) This aligns with the official CSI-2 terminology used throughout the media controller framework.
Replace PCI-specific DMA mask functions with the generic DMA API: - pci_set_dma_mask() + pci_set_consistent_dma_mask() -> dma_set_mask_and_coherent() The generic DMA API is preferred for portability and consistency when working with device-agnostic code.
Update driver code to use media_pad_remote_pad_first() instead of the deprecated media_entity_remote_pad() function.
Update driver code to use the media_entity_pipeline() accessor function instead of directly accessing entity.pipe. This provides proper encapsulation of the media pipeline implementation and aligns with the modern media controller framework API.
The current ISYS implementation treats the physical link as a single monolithic stream. When sensors multiplex multiple virtual channels (VC) over CSI-2, the media pipeline validation fails to distinguish between logical sub-streams, preventing independent streaming of VC-interleaved data. Introduce a custom pipeline validation mechanism that filters entities by Virtual Channel ID. Key changes: - Implement custom graph traversal in media_pipeline_walk_by_vc() to associate entities only if their VC ID matches the requested stream. - Add private V4L2 controls (V4L2_CID_IPU_QUERY_SUB_STREAM) to bridge metadata between the CSI-2 receiver and the video nodes. - Extend ipu_isys_pipeline to track per-substream configurations including resolution, data type, and VC index. - Ensure media_pipeline_stop is VC-aware to maintain correct reference counting across multiplexed pipelines. This enables support for multi-sensor setups and sophisticated ISP configurations where multiple logical streams share a single ISYS backend.
Modernize DMA buffer mapping by using the newer iosys_map API instead of the deprecated dma_buf_map type. This reflects the kernel's migration to a more generic memory mapping abstraction.
Update media graph walk initialization to pass the entity directly instead of the first pad. This aligns with the media controller API's preferred usage pattern where the starting entity encompasses all its pads.
…overy Add integration with the ipu-bridge module to enable automatic sensor discovery and attachment. This includes implementing async notifier callbacks for bound sensors and VCM instantiation support.
…ormats by mbus code Set V4L2_CAP_IO_MC to indicate media controller support and update format enumeration to filter by media bus code. This enables proper interaction with media control aware applications and ensures format enumeration respects sensor capabilities.
Add format descriptions for vector Bayer formats used by IPU hardware: - V4L2_PIX_FMT_*BGGR,GBRG,GRBG,RGGB*V32: 8-bit and 10-bit vector formats These formats are used for high-throughput image processing pipelines.
Fix format enumeration to use V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE instead of V4L2_BUF_TYPE_VIDEO_CAPTURE. Since the device is advertised as supporting multiplanar capture, format enumeration must match this buffer type to be consistent with userspace expectations.
…ESIZES) Add support for frame size enumeration by implementing the vidioc_enum_framesizes operation. This allows applications like libcamera to query supported frame resolutions for video capture, using a stepwise enumeration model.
…rent() Replace dma_alloc_attrs() calls with the simpler dma_alloc_coherent() API where no special DMA attributes are needed. This improves code clarity and reduces API surface complexity.
…erent() Replace dma_free_attrs() calls with the simpler dma_free_coherent() API where no special DMA attributes are needed. This improves code clarity and reduces API surface complexity.
…ping Replace deprecated dma_map_sg() and dma_sync_sg_for_device() calls with the modern dma_map_sgtable() and dma_sync_sgtable_for_device() API. This simplifies DMA scatter-gather table handling and uses the preferred kernel API. Replace the deprecated dma_map_sg() and dma_sync_sg_for_device() calls with dma_map_sgtable() and dma_sync_sgtable_for_device(). Link: https://github.com/intel/ipu6-drivers/blob/master/drivers/media/pci/intel/ipu-buttress.c#L778
…quency Simplify the retrieval of the CSI-2 link frequency by using the standard v4l2_get_link_freq() helper instead of manually querying V4L2_CID_LINK_FREQ controls and its menu items. Additionally, update the link_freq parameter type to s64 to match the helper's return type and improve error reporting in the debug logs. Link: https://github.com/intel/ipu6-drivers/blob/master/drivers/media/pci/intel/ipu-isys-csi2.c#L84
Replaces deprecated strlcpy() calls with strscpy(), which is the preferred string copy function in modern kernel code due to reduced attack surface.
Replace the deprecated no_llseek with noop_llseek in file operations for trace files. The noop_llseek is the preferred modern replacement.
Replace from_timer() macro with timer_container_of(), which is the new kernel API for extracting container structures from timer callbacks.
…c_try() Replace try_to_del_timer_sync() with timer_delete_sync_try(), which is the new kernel API for attempting to delete timers synchronously.
…ignature Adapt get_user_pages() call to the new kernel API that removed the vm_struct parameter from the function signature.
Update MODULE_IMPORT_NS() macros to use string literal arguments instead of bare symbols. This aligns with newer kernel conventions for namespace imports.
…ad_ops signature Adapt the set_routing callback implementation to match the updated v4l2_subdev_pad_ops signature, which now includes state and format whence parameters.
added 26 commits
March 13, 2026 21:02
…st() Replace get_user_pages() and manual follow_pfn() logic with the modern pin_user_pages_fast() API that is preferred for long-term page pins. The patch also refactors memory allocation to use kvzalloc() and simplifies the error path by utilizing unpin_user_pages().
Update CSI-2 link frequency initialization to align with the new v4l2_get_link_freq() function signature that requires a media_pad parameter instead of control handler.
Replace v4l2_async_subdev with the newer v4l2_async_connection type. This reflects the evolution of the V4L2 async framework to better represent connections between sensor and ISP components.
…parameter Add const qualifier to device_driver parameter in bus match function to align with the updated kernel device driver model API.
…ameter Adapt async notifier initialization to the new v4l2_async_nf_init() API that requires the v4l2_device handle for proper notifier registration.
…ange Update async notifier registration call to use the simplified v4l2_async_nf_register() API that no longer requires the v4l2_device parameter (obtained from notifier context).
Replace deprecated v4l2_subdev_get_try_format(), v4l2_subdev_get_try_crop(), and v4l2_subdev_get_try_compose() with the new v4l2_subdev_state_get_format/crop/compose() accessor API for accessing pad try state.
The 'routes' member in struct v4l2_subdev_routing is a __u64, not a pointer type. Direct array indexing is not possible. Cast the address to a local struct v4l2_subdev_route pointer to correctly access the routing table and check for active routes.
Add NULL pointer checks during driver cleanup to prevent dereferencing partially-initialized resources when initialization fails mid-way. Guard against uninitialized video queues and V4L2 subdevices.
Add NULL pointer checks before dereferencing format state pointers during subdevice open. This prevents crashes when format state accessors return NULL.
Add fallback logic for format state retrieval on stream-aware subdevices that do not have explicit routing configured. Return the active format when try state is unavailable to prevent NULL pointer dereferences.
…_init() Set default pad directions in ipu_isys_subdev_init() before media_entity_pads_init(). The media_entity_pads_init() function requires SINK/SOURCE flags to be set on each pad. Initialize pad directions before calling this function to ensure proper media framework integration.
Add NULL checks before cleaning up arrays that may not have been allocated. This prevents potential crashes when the isys_unregister_subdevices() cleanup path runs on devices where initialization failed or was incomplete.
Add helper functions and data structures needed for format validation in link_validate(): - Add ipu_isys_get_src_stream_by_src_pad() to extract source stream from routing information - Add video_stream_watermark struct for tracking stream QoS parameters - Add new fields to ipu_isys_video for format storage and stream tracking: pix_fmt, meta_fmt, csi2, stream, watermark, source_stream, vc, dt These provide the foundation for proper format negotiation between subdevices and video nodes, needed for kernel 6.18+ compatibility.
…support Remove V4L2_SUBDEV_FL_HAS_SUBSTREAMS from CSI-2 subDevs. In kernel 6.18+, the V4L2 core requires stream-aware subdevs to have proper routing configured, but IPU4 manages streams internally. The flag caused all set_fmt/get_fmt calls to be rejected with EINVAL. Fix format enumeration to use per-device format tables (av->pfmts) instead of global ipu_isys_pfmts, and add support for multiplanar buffer type V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE. Add helper functions to abstract format access by buffer type: - ipu_isys_get_format() - ipu_isys_get_data_size() - ipu_isys_get_bytes_per_line() - ipu_isys_get_frame_width() - ipu_isys_get_frame_height() These read from the appropriate format structure based on the active buffer queue type (single-planar, multi-planar, or metadata).
Replace routing-based link validation with proper format negotiation that validates actual format compatibility between subdevices. Add ipu_isys_get_isys_format() to look up pixelformat in the global format table with optional type filtering. Remove is_external() helper as it's no longer needed. Rewrite link_validate() to: - Get source subdev and attempt to retrieve V4L2 state - Lock state and retrieve source format from subdev - Validate format compatibility: width, height, mbus code must match - Return -EPIPE if format validation fails - Include debug logging for format mismatches This replaces the complex routing-based approach that only set up pipeline metadata without actual format validation, fixing format mismatches that prevented streaming.
Search per-device av->pfmts (packed formats) first for matching pixelformat before falling back to global ipu_isys_get_isys_format(). For CSI-2 capture with formats like SBGGR10P, the packed fourcc was not found in the global table, causing incorrect mbus code resolution and permanent format mismatch.
Count all video device entities during graph walk in ipu_isys_video_prepare_streaming() and increment nr_queues accordingly. Previously, nr_queues was only incremented in isa_link_validate(), causing direct CSI-2 capture pipelines to have nr_queues=0, which made the stream start hang indefinitely.
IPU4P uses non-contiguous firmware port numbering: array indices 0-4 map to firmware ports 3, 6, 7, 8, 9. The ACPI SSDB tables report firmware port numbers, but the driver used them as direct array indices into isys->csi2[], causing out-of-bounds access. Add ipu_isys_csi2_fw_port_to_index() to implement the inverse mapping and apply it in all code paths that convert firmware port numbers to array indices. Surface Pro 7 and Surface Book 3 sensor port assignments: - OV8865 (rear): fw port 3 -> array index 0 - OV7251 (IR): fw port 6 -> array index 1 - OV5693 (front): fw port 7 -> array index 2
Add CSI-2 port infrastructure to support additional ports and Surface Pro 7 multi-sensor configuration with rear, IR, and front cameras.
Use fwnode_graph_for_each_endpoint() to iterate over the endpoints connected to the IPU. This replaces the manual loop over a fixed number of ports (ISYS_MAX_PORTS), which was restrictive and required manual fwnode reference management. The change simplifies the logic by: - Removing the hardcoded ISYS_MAX_PORTS limit. - Utilizing the macro's built-in reference counting for endpoints. - Cleaning up the error paths by removing unnecessary 'goto' labels.
Retry ipu_buttress_authenticate() up to 3 times with short delays between attempts during video_open(). This improves reliability when hardware initialization timing varies across different platforms.
The Intel IPU4 and IPU7 drivers both export several symbols with identical names (e.g., ipu_buttress_auth_done). This causes a symbol collision during the modpost stage when both drivers are compiled into the same kernel image, which is common for generic distro-style kernels. Rename the IPU4 buttress symbols by adding an 'ipu4_' prefix. This uniquely identifies the IPU4-specific exports and allows the IPU4, IPU6, and IPU7 drivers to co-exist in a single build without namespace conflicts. The following symbols are renamed: - ipu_buttress_auth_done -> ipu4_buttress_auth_done - ipu_buttress_start_tsc_sync -> ipu4_buttress_start_tsc_sync - ipu_buttress_tsc_read -> ipu4_buttress_tsc_read - ipu_buttress_tsc_ticks_to_ns -> ipu4_buttress_tsc_ticks_to_ns
HACK: Rename "intel-ipu4-isys" to "intel-ipu6". The libcamera simple pipeline handler currently lacks a dedicated match for "intel-ipu4-isys", but it can successfully enumerate the device if it identifies as "intel-ipu6". This allows using the software ISP path as a temporary solution. Revert this once libcamera implements native IPU4 support.
Add and use cross-component debug traces (ov5693 + ipu-isys-csi2 + ipu-isys-video) to characterize the ov5693/IPU4P no-frame issue.
Findings from logs:
- Sensor stream control works: ov5693 writes stream register successfully and reads back 0x01 on stream-on, 0x00 on stream-off.
- ISYS firmware command path works: STREAM_OPEN_DONE, STREAM_START_AND_CAPTURE_ACK, and STREAM_CAPTURE_ACK are received.
- No CSI frame ingress is observed: no SOF/EOF/pin-ready activity; CSI state remains in_frame={0,0,0,0}, receiver_errors=0x0 through start/stop timeout paths.
- CSI reports only ULPS clock-lane exit once after external s_stream(1), suggesting lane transition without sustained payload/frame traffic.
- g_mbus_config lane query returns -EINVAL in this setup; driver falls back to hostdata lane configuration and still enables receiver.
Conclusion:
The failure is no longer in stream orchestration; it is in the sensor-to-CSI data path (or timing/PHY compatibility) where payload frames are not reaching the ISYS receiver.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR: linux-surface/linux-surface#2013
IPU4P driver for Ice Lake devices [1][2].
This IPU4P implementation is based on Intel’s downstream IPU4
driver [3]. ISYS capture works with libcamera [5] and has been tested on
Surface Pro 7 and Surface Book 3. The world-facing camera (ov8865)
works; the user-facing (ov5693) is still being debugged.
To confirm this implementation matches the official Intel downstream
source (excluding whitespace/formatting), follow these instructions to compare
the Intel LTS Reference against the Patched Stable Kernel: [4]
[1] https://github.com/ruslanbay/ipu4-drivers/tree/main/patches/kernel/v6.19
[2] https://github.com/ruslanbay/linux/commits/ipu4-6.19
[3] https://github.com/intel/linux-intel-lts/tree/lts-v5.15.195-android_t-251103T063840Z/drivers/media/pci/intel
[4] https://github.com/ruslanbay/ipu4-drivers/tree/b3d68c865eae49d68298b7d9d13a66e4efe9bb79?tab=readme-ov-file#misc
[5] https://github.com/ruslanbay/ipu4-drivers/tree/8c87ee77643fefc7896bc89b7530246606ed2bf7?tab=readme-ov-file#12-build-and-install-libcamera-repo