Integrate one-pipeline for standardized package deployment and end-2-end testing#321
Integrate one-pipeline for standardized package deployment and end-2-end testing#321pawelchcki merged 6 commits intomasterfrom
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #321 +/- ##
=======================================
Coverage 68.85% 68.85%
=======================================
Files 56 56
Lines 7471 7471
Branches 1058 1058
=======================================
Hits 5144 5144
Misses 1820 1820
Partials 507 507 🚀 New features to boost your workflow:
|
|
Keeping in draft until improvements to CI reliability land |
pawelchcki
force-pushed
the
pawel/one-pipeline
branch
from
89bce77 to
e4af4fb
Compare
pawelchcki
changed the title
pawelchcki
force-pushed
the
pawel/one-pipeline
branch
16 times, most recently
from
62153d7 to
418de04
Compare
pawelchcki
requested review from
cataphract,
Copilot and
xlamorlette-datadog
and removed request for
a team
There was a problem hiding this comment.
Pull request overview
Integrates DataDog’s shared one-pipeline GitLab template to standardize OCI packaging/publishing/promotion while tightening CI behavior (hardening scripts, expanding shellcheck scope, enabling auto-cancel, and expanding the RUM nginx test matrix).
Changes:
- Adds one-pipeline integration (locked include) and wires in
package-ocivia a repo-providedprepare-oci-package.sh. - Introduces SSI artifact assembly jobs (
ssi-build/ssi-build-all) to produce the directory layout consumed bypackage-oci. - Adds/updates requirements metadata and CI hardening (shellcheck scope, retries, auto-cancel, and nginx 1.29.5 in RUM matrices).
Reviewed changes
Copilot reviewed 11 out of 12 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
packaging/sources/requirements.json |
Adds native dependency requirements metadata for packaging/validation. |
metadata/requirements-block.json |
Adds an (empty) blocklist input for requirements validation. |
metadata/requirements-allow.json |
Adds allowlist test cases for requirements validation across arch/libc combinations. |
bin/README.md |
Updates release.py description to reflect current release flow. |
Makefile |
Clarifies architecture normalization expectations for CI vs build tooling. |
.gitlab/ssi-package.yml |
Adds jobs to assemble RUM module artifacts into OCI packaging inputs. |
.gitlab/prepare-oci-package.sh |
Adds packaging preparation script used by one-pipeline package-oci. |
.gitlab/one-pipeline.locked.yml |
Adds locked remote include for one-pipeline template. |
.gitlab/common.yml |
Moves git/submodule rewrite config into a hidden job to avoid leaking into included templates; applies it to repo-defined job templates. |
.gitlab/build-and-test-fast.yml |
Expands shellcheck coverage to include .gitlab and *.sh; adds nginx 1.29.5 to RUM matrix; retries coverage on script failures. |
.gitlab/build-and-test-all.yml |
Expands nginx RUM build/test matrices across additional nginx versions. |
.gitlab-ci.yml |
Enables auto-cancel + interruptible defaults; includes new SSI/one-pipeline configs; configures package-oci needs and requirements test variables. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
pawelchcki
force-pushed
the
pawel/one-pipeline
branch
from
418de04 to
69e91e6
Compare
pawelchcki
changed the title
pawelchcki
dismissed
robertomonteromiguel’s stale review
implemented, and roberto is not available
| tags: ["arch:amd64"] | ||
| script: | ||
| - find bin/ test/ example/ -type f -executable -not -name '*.py' | xargs shellcheck --exclude SC1071,SC1091,SC2317 | ||
| - find bin/ test/ example/ .gitlab/ \( -type f -executable -not -name '*.py' \) -o \( -name '*.sh' -type f \) | xargs shellcheck --exclude SC1071,SC1091,SC2317 |
There was a problem hiding this comment.
nit, opt: -type f could be factorized: -type f \( \( -executable -not -name '*.py' \) -o -name '*.sh' \)
| - "1.26.3" | ||
| - "1.27.5" | ||
| - "1.28.2" | ||
| - "1.29.5" |
There was a problem hiding this comment.
This is not needed. System tests use 1.29.6. See:
- https://github.com/DataDog/system-tests/blob/main/utils/build/docker/cpp_nginx/nginx.Dockerfile;
- https://github.com/DataDog/nginx-datadog/blob/master/.github/workflows/system-tests.yml#L19C16-L19C28.
(I should improve the documentation about this.)
The 1.29.5 line 62 should be removed.
Same below for tests lines 200-203 and 135-137.
| WAF: "ON" | ||
| tags: ["docker-in-docker:$ARCH"] | ||
| retry: | ||
| max: 2 |
There was a problem hiding this comment.
Thanks, this is indeed flaky!
(Actually, we are about to migrate to publishing to Datadog, see #324, but this should be completed a bit later.)
| exit 1 | ||
| fi | ||
|
|
||
| echo "Sources contents:" |
There was a problem hiding this comment.
nit: Maybe we could print out the content of sources before checking it contains a .so file, it could help debugging.
|
|
||
| echo "Copying sources from ${SOURCES_DIR} to sources/" | ||
| mkdir -p sources | ||
| cp -r "${SOURCES_DIR}/"* sources/ |
There was a problem hiding this comment.
nit: We could check that SOURCES_DIR is not empty before this cp command (because if it is empty, the * won't expand).
| # Assumes CWD is set by the package-oci job (a subdirectory of the repo root, | ||
| # so ../artifacts/ resolves to the ssi-build artifact directory). | ||
| # | ||
| # The ssi-build / ssi-build-all CI jobs collect RUM-enabled nginx modules from |
There was a problem hiding this comment.
opt, nit: To be a bit clearer, start this comment by stating something as: “This script is used by the ssi-build and ssi-build-all jobs that are specific to RUM” (all this package OCI stuff is new to me).
- Remove nginx 1.29.5 from fast CI matrices (kept in exhaustive) - Factorize find expression in shellcheck job - Improve prepare-oci-package.sh comments and add empty-dir guard - Add amd64 runner comment in ssi-package.yml
|
Thanks Xavier! |
Integrate the shared one-pipeline template to standardize OCI packaging, publishing, testing and promotion.
Replaces custom
ssi-package-merge/pushjobs withpackage-oci→ publish → promote flow viaprepare-oci-package.sh.Includes CI hardening: error validation in shell scripts, shellcheck coverage, auto-cancel pipelines, and nginx 1.29.5 in RUM matrix.