Skip to content

Commit 529e1fa

Browse files
committed
docs: M12.2 demos, assets, GitHub Pages index, regression tooling
- Demo segments 12–14: VHS tapes, narration, composed recordings, full concat - compose.sh/generate-all: VHS map for 12–14, fail loud on compose errors - docs/index.html: all 14 segments + full-demo anchors; README video table links - docs/assets: split panels, panels-index.json, split_composites.py (PNGs via LFS) - Regression/agent docs, SCRIPTS catalog, team onboarding, env wording updates - scripts: run-regression*.sh, archive README, common.sh updates - .gitignore: .cursor, demos .venv/manim media/VHS rendered/test-results Made-with: Cursor
1 parent 29611c9 commit 529e1fa

File tree

96 files changed

+2390
-87
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

96 files changed

+2390
-87
lines changed

.gitattributes

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1 +1,2 @@
11
docs/demos/recordings/*.mp4 filter=lfs diff=lfs merge=lfs -text
2+
docs/assets/**/*.png filter=lfs diff=lfs merge=lfs -text

.gitignore

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,13 @@ credentials.json
99

1010
# IDE
1111
.idea/
12+
.cursor/
13+
14+
# Demo toolchain — regenerable outputs (Manim, VHS, local test runs)
15+
docs/demos/.venv/
16+
docs/demos/animations/media/
17+
docs/demos/terminal/rendered/
18+
docs/demos/test-results/
1219
# .vscode/ checked in for launch configs (intercept step-debug)
1320
*.swp
1421
*.swo

AGENTS.md

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
# Agent instructions (tekton-dag)
2+
3+
## Regression — iterate until complete
4+
5+
Do **not** stop after a single partial test run. Follow **[docs/AGENT-REGRESSION.md](docs/AGENT-REGRESSION.md)**:
6+
7+
- Run **`bash scripts/run-regression-agent.sh`** (or **`run-regression-agent-full.sh`** if Results + DB must pass).
8+
- **Loop**: fix failures → re-run until **`regression exit code: 0`** and done criteria in the doc are met.
9+
10+
## Quick commands
11+
12+
| Intent | Command |
13+
|--------|---------|
14+
| Best effort for current env | `bash scripts/run-regression-agent.sh` |
15+
| Strict + Tekton Results | `bash scripts/run-regression-agent-full.sh` |
16+
| Timestamped log + correct exit code | `bash scripts/run-regression-stream.sh …` |
17+
18+
Python bootstrap: see [docs/REGRESSION.md](docs/REGRESSION.md).
19+
20+
## Cursor
21+
22+
Rule **regression-iterate** (always on) reinforces the same behavior in `.cursor/rules/regression-iterate.mdc`.

README.md

Lines changed: 29 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -4,17 +4,31 @@ Standalone Tekton pipeline system for **local development and proof-of-concept**
44

55
## Demo Videos
66

7-
🎬 **[Watch all videos on GitHub Pages →](https://jmjava.github.io/tekton-dag/)**
7+
🎬 **GitHub Pages (all segments + players):** [jmjava.github.io/tekton-dag/](https://jmjava.github.io/tekton-dag/)
88
*Publishing & 404 troubleshooting: [docs/GITHUB-PAGES.md](docs/GITHUB-PAGES.md).*
99

10-
| Video | Description | Duration |
11-
|-------|-------------|----------|
12-
| 📹 Architecture Overview | System architecture, DAG model, polyglot support, pipelines | 2m 40s |
13-
| 📹 Intercept Routing | PR vs normal traffic routing, header-based interception | 2m 4s |
14-
| 📹 Local Debugging | mirrord integration, IDE breakpoints, live cluster debugging | 2m 0s |
15-
| 📹 Multi-Team Helm | Helm chart deployment, team isolation, custom hooks | 2m 4s |
16-
17-
*All videos are generated programmatically from source files. See [Milestone 8](milestones/milestone-8.md) for details.*
10+
Each row links to the **in-browser player** on Pages (`#seg-…`) and to the **composed MP4** in the repo. Anchors match [docs/index.html](docs/index.html).
11+
12+
| # | Video | Description | Duration | Watch (Pages) | MP4 in repo |
13+
|---|-------|-------------|----------|---------------|-------------|
14+
| 01 | Architecture overview | System architecture, DAG model, polyglot support, pipelines | ~2m 46s | [▶ Pages](https://jmjava.github.io/tekton-dag/#seg-01) | [`01-architecture.mp4`](docs/demos/recordings/01-architecture.mp4) |
15+
| 02 | Quick start | Kind, Tekton, images, tasks (VHS) | ~1m 50s | [▶ Pages](https://jmjava.github.io/tekton-dag/#seg-02) | [`02-quickstart.mp4`](docs/demos/recordings/02-quickstart.mp4) |
16+
| 03 | Bootstrap dataflow | Stack bootstrap PipelineRun walkthrough | ~2m 30s | [▶ Pages](https://jmjava.github.io/tekton-dag/#seg-03) | [`03-bootstrap-dataflow.mp4`](docs/demos/recordings/03-bootstrap-dataflow.mp4) |
17+
| 04 | PR pipeline | PR flow, intercepts, tests (VHS) | ~2m 30s | [▶ Pages](https://jmjava.github.io/tekton-dag/#seg-04) | [`04-pr-pipeline.mp4`](docs/demos/recordings/04-pr-pipeline.mp4) |
18+
| 05 | Intercept routing | PR vs normal traffic routing, header-based interception | ~2m 6s | [▶ Pages](https://jmjava.github.io/tekton-dag/#seg-05) | [`05-intercept-routing.mp4`](docs/demos/recordings/05-intercept-routing.mp4) |
19+
| 06 | Local debugging | mirrord integration, IDE breakpoints, live cluster debugging | ~1m 57s | [▶ Pages](https://jmjava.github.io/tekton-dag/#seg-06) | [`06-local-debug.mp4`](docs/demos/recordings/06-local-debug.mp4) |
20+
| 07 | Orchestrator API | REST API, stacks, test plan, graph (VHS) | ~2m 14s | [▶ Pages](https://jmjava.github.io/tekton-dag/#seg-07) | [`07-orchestrator.mp4`](docs/demos/recordings/07-orchestrator.mp4) |
21+
| 08 | Multi-team Helm | Helm chart deployment, team isolation, custom hooks | ~1m 59s | [▶ Pages](https://jmjava.github.io/tekton-dag/#seg-08) | [`08-multi-team-helm.mp4`](docs/demos/recordings/08-multi-team-helm.mp4) |
22+
| 09 | Tekton Results | Results API and persisted history (VHS) | ~1m 40s | [▶ Pages](https://jmjava.github.io/tekton-dag/#seg-09) | [`09-results-db.mp4`](docs/demos/recordings/09-results-db.mp4) |
23+
| 10 | Newman / regression | API tests and local test tiers (VHS) | ~2m 0s | [▶ Pages](https://jmjava.github.io/tekton-dag/#seg-10) | [`10-newman-tests.mp4`](docs/demos/recordings/10-newman-tests.mp4) |
24+
| 11 | Test-trace graph | Blast radius, graph query, focused tests (mixed) | ~2m 0s | [▶ Pages](https://jmjava.github.io/tekton-dag/#seg-11) | [`11-test-trace-graph.mp4`](docs/demos/recordings/11-test-trace-graph.mp4) |
25+
| 12 | Regression suite (M12.2) | Full regression story + agent workflows | varies | [▶ Pages](https://jmjava.github.io/tekton-dag/#seg-12) | [`12-regression-suite.mp4`](docs/demos/recordings/12-regression-suite.mp4) |
26+
| 13 | Management GUI architecture (M12.2) | Vue, Flask, orchestrator | varies | [▶ Pages](https://jmjava.github.io/tekton-dag/#seg-13) | [`13-management-gui-architecture.mp4`](docs/demos/recordings/13-management-gui-architecture.mp4) |
27+
| 14 | GUI Tekton extension (M12.2) | Extending the GUI for Tekton | varies | [▶ Pages](https://jmjava.github.io/tekton-dag/#seg-14) | [`14-gui-tekton-extension.mp4`](docs/demos/recordings/14-gui-tekton-extension.mp4) |
28+
29+
**Full concat files:** [01–11 on Pages](https://jmjava.github.io/tekton-dag/#full-demo)[`full-demo.mp4`](docs/demos/recordings/full-demo.mp4) · [01–14 on Pages](https://jmjava.github.io/tekton-dag/#full-demo-m12-2)[`full-demo-with-m12-2.mp4`](docs/demos/recordings/full-demo-with-m12-2.mp4)
30+
31+
*All videos are generated programmatically — run [`docs/demos/generate-all.sh`](docs/demos/generate-all.sh). See [Milestone 8](milestones/milestone-8.md), [M12.2](milestones/milestone-12.2.md), and [docs/demos/README.md](docs/demos/README.md).*
1832

1933
---
2034

@@ -33,12 +47,14 @@ Standalone Tekton pipeline system for **local development and proof-of-concept**
3347
| [M10](milestones/milestone-10.md) | **Completed** | Multi-team scaling: orchestration service, Helm chart, ArgoCD, batched builds |
3448
| [M10.1](milestones/milestone-10-1.md) | **Completed** | Orchestration service testing: Postman/Newman (15 requests, 30 assertions), integration validation |
3549
| [M11](milestones/milestone-11.md) | **Completed** | Vue 3 Management GUI + Python/Flask backend (replaces `reporting-gui/`). Multi-team, multi-cluster, DAG visualization. 69 Playwright E2E tests, 56 pytest unit tests, Postman collection. |
36-
| [M12](milestones/milestone-12.md) | **Completed** | Architecture customization: shared Python package, Helm ConfigMap/PVC templates, parameterized pipelines (no hardcoded `localhost:5000`), `scripts/common.sh`, build image variants (Java 11/17/21, Node 18/20/22, Python 3.10–3.12, PHP 8.1–8.3), custom pipeline hook tasks (pre/post build/test), stack JSON schema, 62 orchestrator pytest tests, 14 shared-package tests. Full docs: CUSTOMIZATION.md, MAINTENANCE.md, Helm README. |
37-
| [M12.2](milestones/milestone-12.2.md) | **Completed** | Documentation sync and archive: README + docs index, obsolete session plan archived |
50+
| [M12](milestones/milestone-12.md) | **Completed** | Architecture customization: shared Python package, Helm ConfigMap/PVC templates, parameterized pipelines (no hardcoded `localhost:5000`), `scripts/common.sh`, build image variants (Java 11/17/21, Node 18/20/22, Python 3.10–3.12, PHP 8.1–8.3), custom pipeline hook tasks (pre/post build/test), stack JSON schema, 62 orchestrator pytest tests, 14 shared-package tests. Full docs: [CUSTOMIZATION.md](docs/CUSTOMIZATION.md), [TEAM-ONBOARDING-STACKS-AND-BAGGAGE.md](docs/TEAM-ONBOARDING-STACKS-AND-BAGGAGE.md), MAINTENANCE.md, Helm README. |
51+
| [M12.2](milestones/milestone-12.2.md) | **Partial** | **Part A done:** doc sync + archive. **Part B open:** regression + Management GUI [docs & demo plan](docs/TESTING-AND-REGRESSION-OVERVIEW.md) / [GUI extension](docs/MANAGEMENT-GUI-EXTENSION.md) / [video segments](docs/demos/segments-m12-2-regression-gui.md) |
3852

3953
Older milestones (M2, M3) are in [milestones/completed/](milestones/completed/).
4054

41-
**Next up:** Finish remaining [M8](milestones/milestone-8.md) items (VHS recordings, Slidev export, full demo concat) as needed; ongoing maintenance via [CUSTOMIZATION.md](docs/CUSTOMIZATION.md) and [MAINTENANCE.md](docs/MAINTENANCE.md).
55+
**Next up:** Finish remaining [M8](milestones/milestone-8.md) items (VHS recordings, Slidev export, full demo concat) as needed; **[M12.2 Part B](milestones/milestone-12.2.md)** regression + GUI [docs](docs/TESTING-AND-REGRESSION-OVERVIEW.md) and [demo segments](docs/demos/segments-m12-2-regression-gui.md); ongoing maintenance via [CUSTOMIZATION.md](docs/CUSTOMIZATION.md) and [MAINTENANCE.md](docs/MAINTENANCE.md).
56+
57+
**Regression (humans & Cursor agents):** run **`scripts/run-regression-agent.sh`** and iterate with fixes until green — see [AGENTS.md](AGENTS.md) and [docs/AGENT-REGRESSION.md](docs/AGENT-REGRESSION.md). Full tier list: [docs/REGRESSION.md](docs/REGRESSION.md).
4258

4359
---
4460

@@ -218,7 +234,7 @@ C4Container
218234

219235
Full diagram set: [docs/c4-diagrams.md](docs/c4-diagrams.md).
220236

221-
> **ArgoCD** is optional for local dev. In production, ArgoCD syncs the Helm chart per team via ApplicationSet. See [ArgoCD + Tekton architecture guide](docs/argocd-architecture-guide.md) and [argocd/applicationset.yaml](argocd/applicationset.yaml).
237+
> **ArgoCD** is optional for local dev. In a **production deployment**, ArgoCD syncs the Helm chart per team via ApplicationSet (separate from the validation cluster where pipelines run). See [ArgoCD + Tekton architecture guide](docs/argocd-architecture-guide.md), [Environments and clusters](docs/ENVIRONMENTS-AND-CLUSTERS.md), and [argocd/applicationset.yaml](argocd/applicationset.yaml).
222238
223239
---
224240

docs/AGENT-REGRESSION.md

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
# Agent playbook: regression until fully done
2+
3+
Human or **Cursor agent**: do not treat testing as complete after a single partial command (e.g. pytest only, or Newman only). **Iterate**: run → read output → fix → re-run **until** the target command exits **0** and the log matches the done criteria below.
4+
5+
## Target commands (pick one)
6+
7+
| Goal | Command (repo root) |
8+
|------|---------------------|
9+
| **Strict cluster** (orchestrator + `newman` + **required** `stack-dag-verify` PipelineRun) | `bash scripts/run-regression-stream.sh --cluster --require-dag-verify` |
10+
| **+ Tekton Results DB** (fails if Results API missing) | `bash scripts/run-regression-stream.sh --cluster --require-dag-verify --with-results-verify` |
11+
| **No cluster** (explicitly limited) | `bash scripts/run-regression-stream.sh --local-only` — then **state** that Tekton/Newman/Results were not run. |
12+
13+
Convenience (detects `kubectl`; uses strict cluster if context works, else local-only with a stderr banner):
14+
15+
```bash
16+
bash scripts/run-regression-agent.sh
17+
```
18+
19+
Bootstrap Python once if needed:
20+
21+
```bash
22+
python3 -m venv .venv && . .venv/bin/activate
23+
pip install -r orchestrator/requirements.txt -r management-gui/backend/requirements-dev.txt
24+
pip install -e 'libs/tekton-dag-common[test]' -e 'libs/baggage-python[test]'
25+
```
26+
27+
## Done criteria (all must hold for “cluster regression done”)
28+
29+
1. Final line: **`regression exit code: 0`** (use `run-regression-stream.sh`, not a broken pipe to `while read`).
30+
2. Log contains **either**:
31+
- **`Phase 2 PASSED`** (standalone `verify-dag-phase2`), **or**
32+
- **`Tekton DAG verify: will run inside run-full-test-and-verify-results.sh`** and that script completes successfully, **or**
33+
- You documented **why** the cluster cannot run Phase 2 (and you did not claim full cluster verification).
34+
3. **No unexplained `SKIP Tekton DAG verify`** when `stack-dag-verify` should exist — if you see it, fix install/namespace or pass `--require-dag-verify` and resolve the failure.
35+
4. Newman/orchestrator: no port-forward bind failure; if it fails, free port or set `ORCHESTRATOR_TEST_PORT`.
36+
37+
## Iteration loop (required agent behavior)
38+
39+
```
40+
repeat {
41+
run the chosen target command; capture full log
42+
if exit 0 and done criteria met → STOP (success)
43+
identify failing tier (pytest / Playwright / Phase 2 / Newman / Results)
44+
apply minimal fix (code, script, manifest, env, cluster)
45+
} until success or human abort
46+
```
47+
48+
Do **not** stop after only:
49+
50+
- Phase 1 + unit tests, or
51+
- Newman without a **Succeeded** `stack-dag-verify` path (see [REGRESSION.md](REGRESSION.md) “What actually runs Tekton PipelineRuns?”).
52+
53+
## References
54+
55+
- [REGRESSION.md](REGRESSION.md) — tiers, flags, env vars
56+
- [GITHUB-PAGES.md](GITHUB-PAGES.md) — if work touches Pages

docs/CUSTOMIZATION.md

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,8 @@
22

33
Practical recipes for extending teams, stacks, build images, registries, pipelines, and intercept behavior. Examples use paths from the main **tekton-dag** platform repository.
44

5+
**New team or new stack from scratch?** See [TEAM-ONBOARDING-STACKS-AND-BAGGAGE.md](TEAM-ONBOARDING-STACKS-AND-BAGGAGE.md) for which **baggage / header-forwarding library** to use per language and an end-to-end **stack creation checklist** (YAML, Helm, orchestrator).
6+
57
---
68

79
## 1. Add a new team
@@ -57,7 +59,8 @@ cp teams/squad-b/team.yaml helm/tekton-dag/raw/teams/squad-b/
5759

5860
## 2. Add an app to a stack
5961

60-
**Goal:** Register another service in the DAG (build, image, tests, downstream edges).
62+
**Goal:** Register another service in the DAG (build, image, tests, downstream edges).
63+
**Baggage:** add the matching library from `libs/` and set `propagation-role`—see [TEAM-ONBOARDING-STACKS-AND-BAGGAGE.md](TEAM-ONBOARDING-STACKS-AND-BAGGAGE.md).
6164

6265
Edit or create a stack file under `stacks/`. Each entry under `apps` needs a unique `name`, `repo`, `role`, `build` tool settings, and optional `downstream` / `tests`:
6366

docs/DAG-AND-PROPAGATION.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -94,3 +94,5 @@ So:
9494
| **“Which app has changes”** | The app whose PR triggered the run = the single **changed-app** (and thus in **build-apps**); that’s the app being intercepted for that run. |
9595

9696
For sequence diagrams that show intercept scenarios (originator only, forwarder only, terminal only, or multiple intercepted), see [docs/c4-diagrams.md](c4-diagrams.md) (“Dynamic Diagram: PR Intercept Scenarios”).
97+
98+
To implement propagation in application code and onboard a new stack, see [TEAM-ONBOARDING-STACKS-AND-BAGGAGE.md](TEAM-ONBOARDING-STACKS-AND-BAGGAGE.md).

docs/ENVIRONMENTS-AND-CLUSTERS.md

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
# Environments and clusters
2+
3+
This project’s docs and demos often talk about **two traffic paths** (PR vs non-PR) in a cluster. That is **not** the same as “this cluster is production.”
4+
5+
## How we name things
6+
7+
| Term | Meaning |
8+
|------|--------|
9+
| **Validation / pre-production cluster** | A **dedicated** Kubernetes cluster (or namespace slice) used to **build, deploy, and test** changes. It is usually **similar in shape** to production (same kinds of services, ingress, policies) so tests are realistic. **Pipelines and intercept demos run here.** |
10+
| **Production cluster** | Where **released** workloads run for real users. Code and images are **promoted** here **after** validation — it is a **separate** cluster (or strictly separated environment), not the same place PR pipelines mutate. |
11+
| **Baseline deployment** (or **mainline**) | The **non-PR** revision of a service **in the validation cluster** — the “steady” line already merged and deployed there. Demos that used to say “production” for this path mean **baseline**, not customer production. |
12+
| **PR deployment** | An **ephemeral** revision deployed **alongside** the baseline **in the validation cluster**, reachable when requests carry the dev-session header. |
13+
14+
So: **“normal traffic”** in intercept diagrams = traffic to the **baseline** pods in the **validation** environment, **unless** the document explicitly says it is about the production cluster.
15+
16+
## Scripts and promotion
17+
18+
- `scripts/promote-pipelines.sh` moves Tekton YAML between namespaces (e.g. test → prod-facing namespace). That is an **operator-controlled promotion step**; it does not imply that PR validation runs in the production cluster.
19+
- Regression and E2E scripts target **whatever cluster your kubeconfig points at** — treat that as validation unless you intentionally point at production (not recommended for destructive tests).
20+
21+
## Demos and narration
22+
23+
Demo videos and Manim scenes use **baseline / mainline** wording for the non-PR path so they stay accurate for teams that **only** run pipelines in a validation cluster and ship artifacts onward to production separately.

docs/GITHUB-PAGES.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,8 @@
66

77
- Workflow: [.github/workflows/pages.yml](../.github/workflows/pages.yml)
88
- **Artifact root** is the [`docs/`](../docs/) folder (so `docs/index.html` becomes the site homepage).
9+
- Demo page [`index.html`](index.html) embeds **all 14 segments** plus **full-demo** / **full-demo-with-m12-2**; deep-link with fragments e.g. `…/tekton-dag/#seg-07` (see root [README](../README.md) demo table).
10+
- Checkout uses **`lfs: true`** so large media tracked with Git LFS (e.g. demo MP4s under [`docs/demos/recordings/`](demos/recordings/)) are present in the deployed site. If videos are missing on Pages, confirm those files are committed and that LFS objects are pushed (`git lfs push --all origin main`).
911
- Repository **Settings → Pages**: source should be **GitHub Actions** (not “Deploy from a branch”).
1012

1113
## If the site returns 404

0 commit comments

Comments
 (0)