Decision record - reference_integration scope

[PiotrKorkus]

Decision record for reference_integration scope

Review checklist

• Persistency @vinodreddy-g
• Logging @pawelrutkaq @rmaddikery @arsibo
• Lifecycle @pawelrutkaq @NicolasFussberger
• Kyron @pawelrutkaq
• Orchestration @vinodreddy-g
• Baselibs @4og
• Baselibs_rust @pawelrutkaq @4og
• Communication @LittleHuba
• FEO @johannes-esr
• Infrastructure @AlexanderLanin
• Technical Leads @antonkri @FScholPer @qor-lb

Every CFT shall provide their status - OK or NOK

[github-actions]

⚠️ Docs-as-Code version mismatch detected
Please check the CI build logs for details and align the documentation version with the Bazel dependency.

[pawelrutkaq]

Blocker as we need feedback gathering from all (Each CFT shall give OK/NOK)

[github-actions]

The created documentation from the pull request is available at: docu-html

[anmittag]

Assuming Agreeing is voting for Option 2. ETAS position is clear Option 2.

[AlexanderLanin]

Assuming Agreeing is voting for Option 2. ETAS position is clear Option 2.

Correct. The document states:

Decision: Option 2 has been most favored by the community.
We accept additional effort for the integration process in exchange for a better traceability.

---

Although I'm personally not sure this is extra effort, or better traceability, I favor option 2 for all the other reasons listed in the document :-)

[4og]

Unfortunately, I cannot agree with Option 2 as in my opinion not all viable alternatives have been considered. I will propose one below as Option 3. In general I agree with the strive towards a "full-stack" solution with common tooling and toolchain for all S-CORE modules (this already makes Option 1 non-viable as it dictates full independence). I understand that the main reasoning for Option 2 is to mitigate the risk that Minimal Version Selection (MVS) algorithm in Bazel brings when all modules are put together, and to have better controllability of the quality tooling used in the project. However, Option 2 has major flaws that made me think about alternatives that are good enough to mitigate these risks.

Flaws of Option 2

Flaw 1: Non-dev dependencies for quality tooling

Running quality tooling in the reference integration requires making quality tooling dependencies in modules non-dev-dependencies. This has been discussed in several calls already. @LittleHuba was mainly raising this issue. (For specifics on version selecting algorithm, see https://bazel.build/external/module#version-selection, there is also a link to Go's reasoning why MVS works this way).

Maybe non-dev dependencies are problematic because they increase the dependency footprint, which is an anti-pattern in software engineering when the goal is actually to reduce the number of dependencies. In production integration environments there are much more dependencies, not only coming from S-CORE modules, and the same MVS algorithm will be kicking in as it has to satisfy dependency chains of these now non-dev quality tools (this includes all transitive dependencies along the chain!) alongside everything else in those environments. This is not a theoretical issue. We are currently trying to incorporate S-CORE modules into our system and see how MVS is changing dependencies of non-S-CORE components and breaking the build.

I understand we can "hack" it and still make all quality dependencies dev-dependencies, and maintain patches in the reference integration to un-dev them. But I don't feel it's a good way forward as we're doing the reference integration not only for the sake of having it, but also to serve as a blueprint for actual automotive integration projects.

Flaw 2: Not all quality checks can be re-run generically

Re-running quality checks in the reference integration on the basis that Bazel chooses a different version set for dependencies may work for some work products, but not all. For example, some things may need to be done manually, such as verifying for AoU violations. And most importantly, a centralized quality pipeline will not cover module-specific checks that don't belong in a generalized setup. This is just one example, but let's say a security analysis may advise to run a fuzzer on a specific component. A module may have similar domain-specific quality requirements that cannot be centralized.

Flaw 3: Unclear feedback loop

What is the feedback loop when the reference integration discovers a failure because of the version selection?
I suppose the affected modules will need to fix the issue and release an update. Over time, this means everyone converges to use the same versions of dependencies, which makes the quality checks in the reference integration redundant as they end up repeating exactly what was already verified in the modules themselves with no added value. And here we come to Option 3.

Proposal: Option 3

The main idea is to do front-loading in modules themselves via organizational and technical means:

• Modules follow the S-CORE process.
• We agree across modules on what quality tooling and toolchains to use, but each module runs them in its own context with its own resolved dependency set (a module can still use something extra for module-specific use cases).
• Most importantly, we agree on dependency versions for the upcoming S-CORE release. Example: if score_logging is releasing v1.5 to the next S-CORE release, every other module that depends on logging has to migrate to v1.5 (see also backward compatibility note under General Guideline below).
• Modules minimize the number of non-dev and non-S-CORE dependencies in their MODULE.bazel files.

What checks remain in the reference integration:

• Consolidate safety artifacts from all module releases.
• Verify that the module safety package adheres to the S-CORE process (we can discuss if it can be done somewhere else, e.g. the Bazel registry).
• Run feature integration tests.
• Run consistency checks to discover mismatches in the dependency graph. The bazel mod command can help here.
  • If a mismatch is detected: affected modules will need to release an update.
  • Or depending on what kind of mismatch it is: provide safety argumentation why it's not a problem.

To avoid big-bangs shortly before a release and discover problems earlier:

• Module maintainers regularly publish release candidates of their modules to the reference integration, either in form of releasing to the registry or as git hashes. I believe this can be well automated, though the exact mechanism would need to be worked out.
• Immediate feedback to module owners in case of version mismatches, failed integration tests, or incomplete safety packages.

On the coordination cost: Option 3 requires tighter communication across modules. In practice this could be supported by a shared dependency manifest (a file listing agreed-upon dependency versions for the upcoming release, covering not only direct but also transitive dependencies) and a release planning process where module teams align on timelines and versions. I don't think this cost is much higher than with Option 2, because Option 2 also requires coordination when reference integration failures need to be triaged and fixed by the responsible module teams.

General Guideline: Strive for backward compatibility guarantees in all modules and clear deprecation policies. This is particularly important for Option 3 to work, because it reduces friction when modules update to agreed-upon dependency versions. It's also important in general as Bazel's version selection doesn't care about SemVer. It uses it (a relaxed version of it to be precise) for comparison only to know which version is newer or older during the resolution process. Bazel may very well choose for example major version 4.0 for all even if some module depends on 1.2. I know it can be selectively worked around with multiple_version_override, but I'm not sure we want to go this way.

I think with Option 3 we can avoid all three flaws mentioned above and make the integration technically simpler. Verifying quality where it originates, in the modules, is more reliable than repeating it in the reference integration. It's because modules have full knowledge of their own domain, their specific quality requirements, and the exact dependency context they were tested against.

I'm also sure we can brainstorm more options (e.g. like bumping dependencies to a consistent state and running checks in the scope of the module and not in the scope of the reference integration).

At this stage of the project I think it is more important to gather learnings about using Bazel with bzlmod in integration projects (not only in S-CORE itself, but also in series projects) rather than investing into a complicated integration setup as described in Option 2, which may turn non-viable. Depending on the learnings we can steer into one or another direction.

[pawelrutkaq]

So since I am only here a "move topic forward guy," I recirculate this again to TL: @anmittag @thilo-schmitt @FScholPer @qor-lb, please pick up in TL meeting how to move forward - for now, this is a blocker for 0.7 ;)

From tech opinion: This (Option 3) is viable, but part of the agreement would be module owners agree to run specific CI workflows (actions, whatever), i.e., on release, so we can gather consistent artefacts (to be defined by ref_int needs). This could not be anymore each one doing something a little bit different (ofc - running more, own checks always is okay).
Note: Using git hashes in ref int during CI(main) (milestone phase 0 - 6 weeks of dev) will not be able to provide a quality artefacts view of modules.