Skip to content

DCAR Report #2 ‐ 2025‐07‐29

Jump to bottom
Pierre Raybaut edited this page Jul 29, 2025 · 2 revisions

Report #2 - Date: 2025-07-25

Scope of the report

This report summarizes the progress made on the DataLab Core Architecture Redesign (DCAR) project since the last report (DCAR Report #1).

More specifically, this report covers the following tasks:

  • Task 1: Core Architecture Redesign
    • 1b. Decouple I/O features (including I/O plugins)
    • 1c. Redesign the API for the new core library
  • Task 2: Validation and Testing
    • 2a. Migrate validation infrastructure to the new core
    • 2b. Conduct extensive testing (core and application layers)
  • Task 3: Documentation and Training Materials
    • 3a. Write installation guides, API docs, user manuals

Milestone Status for Report #2

The following paragraphs summarize the status of each milestone at the time of this report.

✅ 1b. Decouple I/O features (excluding HDF5)

The I/O plugin system has been successfully migrated to sigima, allowing for flexible I/O operations.

✅ 1c. Redesign the API for the new core library

The API redesign is complete. The new object model for signals and images is fully independent from the cdl module which has been renamed to datalab (whereas the cdl Python package has been renamed to datalab-platform on PyPI but no release has been made yet).

The following actions have been completed:

  • Removed all dependencies on cdl
  • Removed all references to Conf:
    • Replaced Conf with a new option system (options singleton defined in sigima/config.py)
    • Contrary to DataLab's Conf which is a configuration management system based on a INI file serialization, this new option system is an in-memory configuration system that allows for dynamic updates and is not serialized to disk, which is more suitable for the new core library.
    • DataLab's Conf is now synchronized with Sigima's options system, allowing for a unified configuration management approach.
  • NumPy version was bumped to 1.22.0, due to vulnerabilities.
  • Clarified the role of the PREFIX attribute in BaseObj

The following architecture has been implemented:

sigima
├── data                # Package data
│   ├── logos
│   ├── tests
├── io                  # I/O operations and plugins
├── locale              # Localization files
├── objects             # Core objects (signals, images, ROI, etc.)
├── proc                # Computation functions that operate on objects
│   ├── image
│   ├── __init__.py
│   ├── base.py
│   ├── decorator.py
│   ├── signal.py
│   ├── validation.py   # Validation infrastructure for computation functions
├── tests
├── tools               # Algorithms and tools that operate on NumPy arrays
├── __init__.py
├── config.py           # Library options system
├── params.py           # Parameters (`DataSet` objects) for computation functions
├── worker.py           # Minimal interface for executing long-running operations

✅ 2a. Migrate validation infrastructure to the new core

The validation infrastructure has been successfully migrated to the new core library. This includes:

  • Moving the validation logic from cdl to sigima.proc.validation.
  • Moving the pytest fixtures and tests from cdl to sigima.tests.
  • Complete rewrite of the tests to use the new sigima API and, more importantly, to separate the interactive (GUI-based) tests from the core library tests (this required using pytest requests and fixtures to handle the GUI interactions).

ℹ️ This part of the migration was particularly challenging due to the need to adapt the tests to the new architecture and ensure compatibility with the updated API.

💡 At this stage, the new test infrastructure was not only an improvement in terms of organization and maintainability, but it also allowed for more comprehensive testing of the new features and functionality introduced in the sigima library. For example, we introduced the module sigima.tools.checks which provides a set of decorators for validating the input parameters (1D and 2D arrays) of the tools functions, thus extending the scope of validation within the library.

✅ 2b. Conduct extensive testing (core and application layers)

Extensive testing has been conducted on both the core and application layers.

Coverage reports show the following results:

Module Layer Coverage
sigima Core 80%
datalab Application 84%

Those coverage levels correspond to the scope of the DCAR project in the sense that it is the result of the mechanical migration of the code from cdl to sigima and datalab, and of some refactoring of the code to adapt it to the new architecture.

🟡 3a. Write installation guides, API docs, user manuals

Sigima's documentation has been initialized and is available in the docs directory of the sigima package. It includes the following sections:

Section Subtask Status Comment
Getting Started 3b. 🟡 Just initialized, no content yet: this is the scope of subtask 3b
API 3a. ✅ Done
Contributing 3a. ✅ Done
Release notes 3a. ✅ Done

To summarize, all the actions planned for subtask 3a have been completed, except for the "user manuals" which are not yet written (crash course to Sigima, tutorials, and examples).

Documentation is available on Read the Docs at sigima.readthedocs.io.

Code References for Report #2

Clone this wiki locally