Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
97 commits
Select commit Hold shift + click to select a range
b85e466
spec-sheet: add envd_scalability scenarios
aljoscha May 8, 2026
11564b8
fix crdb cluster spec sheet
aljoscha May 8, 2026
939b73f
cluster-spec-sheet: add cluster_object_limits scenarios
aljoscha May 13, 2026
ecc42ab
cluster-spec-sheet: cap stalled lag for plot legibility; cap envd_sca…
aljoscha May 13, 2026
0caf98e
ci: add envd_scalability + cluster_object_limits to release-qualifica…
aljoscha May 13, 2026
db8f376
cluster-spec-sheet: rename environmentd-related groups for clarity
aljoscha May 13, 2026
ecd0f5c
cluster-spec-sheet: log environment id + key system limits at workflo…
aljoscha May 13, 2026
4052673
cluster-spec-sheet: fix pyright failure in log_environment_info
aljoscha May 15, 2026
8e239c5
cluster-spec-sheet: make bulk CREATE statements idempotent
aljoscha May 15, 2026
bdb6607
cluster-spec-sheet: cap envd_objects_scalability default at 30k
aljoscha May 15, 2026
b95ce1c
cluster-spec-sheet: bisect to refine cluster_object_limits cliff
aljoscha May 15, 2026
3fdc5d4
cluster-spec-sheet: skip unavailable cluster sizes gracefully
aljoscha May 15, 2026
589fafe
envd-ddl-scalability: add DDL scaling audit harness
aljoscha May 15, 2026
0cd8050
envd-ddl-scalability: harness fixes for local envd; notes on suspects
aljoscha May 15, 2026
85acbb4
envd-ddl-scalability: record findings from first profiling pass
aljoscha May 15, 2026
9ff71d4
envd-ddl-scalability: views + mvs padding results
aljoscha May 15, 2026
aebf44b
storage: scoped tracing spans for create_collections sub-phases
aljoscha May 15, 2026
376f724
audit: pinpoint storage::create_collections cost to persist init
aljoscha May 15, 2026
f69d91c
adapter: bucket user item counts to speed up validate_resource_limits
aljoscha May 15, 2026
9fca09f
catalog: avoid full OID scan on every allocation
aljoscha May 15, 2026
293a243
catalog: avoid O(N) row scan in TableTransaction::insert
aljoscha May 15, 2026
78f4851
envd-ddl-scalability: post-fix measurements
aljoscha May 15, 2026
c31cc64
envd-ddl-scalability: correct misleading post-fix claim
aljoscha May 15, 2026
8456618
envd-ddl-scalability: rank post-fix span growers + revised design
aljoscha May 15, 2026
12262da
design: catalog transactions in O(delta) instead of O(catalog size)
aljoscha May 15, 2026
81f9c3f
catalog: shared DurableCatalogData + overlay TableTransaction
aljoscha May 15, 2026
200be39
catalog: name- and OID-keyed indexes on DurableCatalogData
aljoscha May 15, 2026
9e4e116
catalog: OID allocator probes the shared oid_owner index
aljoscha May 15, 2026
cfae53b
catalog+adapter: own Transaction, carry DurableCatalogData across DDL…
aljoscha May 15, 2026
11be652
adapter: O(delta) timeline read-hold accounting in apply_catalog_impl…
aljoscha May 15, 2026
2ef53a2
storage-client: add get_collection_shard and contains_shard point APIs
aljoscha May 15, 2026
8ee4e18
envd-ddl-scalability: post-design measurements
aljoscha May 15, 2026
cdaa38e
persist: add bogo-consensus, an in-memory gRPC Consensus backend
aljoscha May 15, 2026
87ba252
bogo-consensus: raise gRPC message size cap; record probe results
aljoscha May 15, 2026
4fe0d58
bogo-consensus: raise HTTP/2 flow control windows; add microbench
aljoscha May 18, 2026
9286fa9
envd-ddl-scalability: bogo follow-up after window fix
aljoscha May 18, 2026
15f741c
bogo-consensus: fan out client RPCs across multiple gRPC channels
aljoscha May 18, 2026
2cc6046
envd-ddl-scalability: note multi-channel bogo client landed
aljoscha May 18, 2026
9dbb473
envd-ddl-scalability: bogo faster than CRDB end-to-end
aljoscha May 18, 2026
0349346
bogo-consensus: match postgres pool size (50) for channel fan-out
aljoscha May 18, 2026
d8e1fa6
persist: add mz_persist_external_op_latency_by_shard_kind
aljoscha May 18, 2026
0f7e253
envd-ddl-scalability: shard-attributed slope (catalog & txns CAS, not…
aljoscha May 18, 2026
382565a
envd-ddl-scalability: txns grows with N too; queue up persist-only mi…
aljoscha May 18, 2026
77d624f
envd-ddl-scalability: persist microbench must run against CRDB + bogo
aljoscha May 18, 2026
ddc8b30
persist-client: add persist_cas_bench example
aljoscha May 18, 2026
71ed282
envd-ddl-scalability: persist-only CAS microbench finds no slope
aljoscha May 18, 2026
00f5d90
envd-ddl-scalability: spine_flattened per DDL is the slope
aljoscha May 18, 2026
412184a
persist: add mz_persist_state_apply_latency_by_shard_kind
aljoscha May 18, 2026
5f34fec
envd-ddl-scalability: state-apply attribution closes the slope
aljoscha May 18, 2026
03f856c
persist: source-label apply_diff calls
aljoscha May 18, 2026
86b7f6f
envd-ddl-scalability: source-attributed apply_diff — it's GC
aljoscha May 18, 2026
f4015a1
persist: split consensus_cas timing into wire vs outer
aljoscha May 18, 2026
c0f76f1
bogo-consensus: O(1) state-gauge updates
aljoscha May 18, 2026
7c31f36
envd-ddl-scalability: CRDB-backed sanity check at N=5k/10k/15k
aljoscha May 18, 2026
30b8d5c
adapter: phase-level timing inside Catalog::transact
aljoscha May 18, 2026
98c971e
envd-ddl-scalability: phase-split shows slope is mostly outside transact
aljoscha May 18, 2026
1efac86
adapter: phase-level timing in Coordinator::catalog_transact_inner
aljoscha May 18, 2026
fd16c06
envd-ddl-scalability: builtin_table_update().execute is the slope owner
aljoscha May 18, 2026
5d2d138
coord: remove O(n) table advancement loop from group_commit
aljoscha Feb 19, 2026
a567472
envd-ddl-scalability: group_commit fix lands; coord_builtin_table_exe…
aljoscha May 18, 2026
3cbc0f7
adapter: phase-level timing in apply_catalog_implications
aljoscha May 18, 2026
c694ec8
envd-ddl-scalability: inner_controller_setup owns apply_catalog_impli…
aljoscha May 18, 2026
b4db6f1
adapter: split create_table_collections into sub-phases
aljoscha May 18, 2026
7491526
envd-ddl-scalability: storage.create_collections owns CREATE TABLE slope
aljoscha May 18, 2026
1c2f6ac
storage-controller: phase-level timing in create_collections_for_boot…
aljoscha May 18, 2026
31fac9a
envd-ddl-scalability: create_collections_for_bootstrap is flat; slope…
aljoscha May 18, 2026
3518388
catalog: phase-level timing in compare_and_append and sync_inner
aljoscha May 19, 2026
00d31c5
catalog: skip unconditional consolidate at end of sync_inner
aljoscha May 19, 2026
657a070
envd-ddl-scalability: phase 8+9 — consolidate fix lands
aljoscha May 19, 2026
00025cb
adapter: phase-level + per-kind timing in CatalogState::apply_updates
aljoscha May 19, 2026
ad197b0
catalog: switch Schema items/functions/types to imbl::OrdMap
aljoscha May 19, 2026
4b6f5d1
storage-client: switch StorageMetadata fields to imbl persistent coll…
aljoscha May 19, 2026
5e17d15
envd-ddl-scalability: phase 10/11/12 — apply_updates slope erased
aljoscha May 19, 2026
351ddb9
envd-ddl-scalability: sweep for the imbl-OrdMap-of-non-persistent-val…
aljoscha May 19, 2026
1a8446c
catalog: sweep remaining inline-in-imbl::OrdMap fields onto imbl pers…
aljoscha May 20, 2026
5edac54
envd-ddl-scalability: phase 13 sanity bench — sweep is no-regression …
aljoscha May 20, 2026
0a671b7
storage-collections: per-phase timing in prepare_state
aljoscha May 20, 2026
b74d189
storage-collections: chunk update_write_frontiers to release the lock
aljoscha May 20, 2026
00133fc
envd-ddl-scalability: phase 14 — prepare_state slope and chunked unlock
aljoscha May 20, 2026
5e405b4
envd-ddl-scalability: plan phase 15+ — extend to N=20k/25k, iterate
aljoscha May 20, 2026
0007b89
storage-collections: shared txns_upper, kill per-tick fanout
aljoscha May 20, 2026
542a946
envd-ddl-scalability: phase 15 writeup + 40Hz follow-up note
aljoscha May 21, 2026
cdb8484
storage-collections: count txns-upper advances and sweeps
aljoscha May 21, 2026
16f7822
envd-ddl-scalability: phase 16 — txns-upper frequency origin
aljoscha May 21, 2026
893ee2b
envd-ddl-scalability: phase 17 — high-N slope confirmation
aljoscha May 21, 2026
4a97b49
envd-ddl-scalability: phase 18 — bogo vs CRDB CAS attribution
aljoscha May 21, 2026
9385fef
envd-ddl-scalability: phase 19 — CRDB consensus CAS scaling root-cause
aljoscha May 21, 2026
d0a4f4b
envd-ddl-scalability: phase 20 — pool bump confirmation sweep
aljoscha May 21, 2026
690df83
envd-ddl-baseline: scaffold sibling investigation for base DDL latency
aljoscha May 21, 2026
e723f47
envd-ddl-baseline: iteration 1 — external-call inventory at N=0
aljoscha May 21, 2026
0310706
storage-collections: decompose open_data_handles into per-step sub-ph…
aljoscha May 21, 2026
5e7812c
persist-client: short-circuit upgrade_version when state is at build …
aljoscha May 21, 2026
b90a47e
storage-client: fast-path single-collection create_collections opens
aljoscha May 21, 2026
eb205c3
envd-ddl-baseline: iteration 2 — decompose open_data_handles, cut upg…
aljoscha May 21, 2026
67ae7f9
persist-client,storage-client: instrument open_critical_handle sub-CAS
aljoscha May 21, 2026
1d4a099
persist-client,storage-client: fold open_critical_handle's three CAS …
aljoscha May 21, 2026
1ad2565
envd-ddl-baseline: iteration 3 — fold redundant CAS in open_critical_…
aljoscha May 21, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ members = [
"src/aws-secrets-controller",
"src/aws-util",
"src/balancerd",
"src/bogo-consensus",
"src/build-info",
"src/build-tools",
"src/catalog",
Expand Down Expand Up @@ -137,6 +138,7 @@ default-members = [
"src/aws-secrets-controller",
"src/aws-util",
"src/balancerd",
"src/bogo-consensus",
"src/build-info",
"src/build-tools",
"src/catalog",
Expand Down
18 changes: 18 additions & 0 deletions bin/bogo-consensus
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
#!/usr/bin/env bash

# Copyright Materialize, Inc. and contributors. All rights reserved.
#
# Use of this software is governed by the Business Source License
# included in the LICENSE file at the root of this repository.
#
# As of the Change Date specified in that file, in accordance with
# the Business Source License, use of this software will be governed
# by the Apache License, Version 2.0.
#
# bogo-consensus — build and run the in-memory persist Consensus gRPC server.

set -euo pipefail

cd "$(dirname "$0")/.."

exec cargo run --bin mz-bogo-consensus -- "$@"
34 changes: 31 additions & 3 deletions ci/release-qualification/pipeline.template.yml
Original file line number Diff line number Diff line change
Expand Up @@ -694,8 +694,8 @@ steps:
agents:
queue: linux-aarch64-small

- id: cluster-spec-sheet-environmentd
label: "Cluster spec sheet: Environmentd (against Staging)"
- id: cluster-spec-sheet-envd-qps-scalability
label: "Cluster spec sheet: Envd QPS Scalability (against Staging)"
depends_on: build-aarch64
timeout_in_minutes: 3600
concurrency: 1
Expand All @@ -704,6 +704,34 @@ steps:
- ./ci/plugins/mzcompose:
composition: cluster-spec-sheet
run: default
args: [--cleanup, --target=cloud-staging, environmentd]
args: [--cleanup, --target=cloud-staging, envd_qps_scalability]
agents:
queue: linux-aarch64-small

- id: cluster-spec-sheet-envd-objects-scalability
label: "Cluster spec sheet: Envd Objects Scalability (against Staging)"
depends_on: build-aarch64
timeout_in_minutes: 3600
concurrency: 1
concurrency_group: 'cluster-spec-sheet-cluster'
plugins:
- ./ci/plugins/mzcompose:
composition: cluster-spec-sheet
run: default
args: [--cleanup, --target=cloud-staging, envd_objects_scalability]
agents:
queue: linux-aarch64-small

- id: cluster-spec-sheet-cluster-object-limits
label: "Cluster spec sheet: Cluster Object Limits (against Staging)"
depends_on: build-aarch64
timeout_in_minutes: 3600
concurrency: 1
concurrency_group: 'cluster-spec-sheet-cluster'
plugins:
- ./ci/plugins/mzcompose:
composition: cluster-spec-sheet
run: default
args: [--cleanup, --target=cloud-staging, cluster_object_limits]
agents:
queue: linux-aarch64-small
Loading
Loading