Skip to content

Improve final clause speed article #172

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
BorisTyshkevich merged 2 commits into from
Mar 17, 2026
Merged

Improve final clause speed article #172

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
36c0656
Improve detached parts KB for 23.10+ behavior and safe cleanup recipes
bvt123 Mar 2, 2026
d6c0647
Improve FINAL clause speed article with recent version changes
bvt123 Mar 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Empty file added content/AGENTS.md
View file
Open in desktop
Empty file.
48 changes: 46 additions & 2 deletions content/en/altinity-kb-queries-and-syntax/altinity-kb-final-clause-speed.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,21 @@ description: >
* Since 23.12 - final applied only for intersecting ranges of parts, see [https://github.com/ClickHouse/ClickHouse/pull/58120](https://github.com/ClickHouse/ClickHouse/pull/58120)
* Since 24.1 - final doesn't compare rows from the same part with level > 0, see [https://github.com/ClickHouse/ClickHouse/pull/58142](https://github.com/ClickHouse/ClickHouse/pull/58142)
* Since 24.1 - final use vertical algorithm (more cache friendly), see [https://github.com/ClickHouse/ClickHouse/pull/54366](https://github.com/ClickHouse/ClickHouse/pull/54366)
* Since 25.6 - final supports Additional Skip Indexes, see [https://github.com/ClickHouse/ClickHouse/pull/78350](https://github.com/ClickHouse/ClickHouse/pull/78350)

* Since 25.6 - final supports skip indexes (`use_skip_indexes_if_final=1` by default), see [https://github.com/ClickHouse/ClickHouse/pull/78350](https://github.com/ClickHouse/ClickHouse/pull/78350)
* Since 25.12 - `apply_prewhere_after_final` and `apply_row_policy_after_final` settings for correct PREWHERE/row policy handling with FINAL
* Since 26.2 - `enable_automatic_decision_for_merging_across_partitions_for_final=1` by default (auto-enables cross-partition optimization when safe)

### Related Settings

| Setting | Default | Since | Description |
|---------|---------|-------|-------------|
| `do_not_merge_across_partitions_select_final` | 0 | 20.10 | Skip cross-partition merging when partitions are pre-optimized |
| `max_final_threads` | 0 (auto) | 20.5 | Thread limit for FINAL processing |
| `enable_vertical_final` | 1 | 24.1 | Read columns in parallel from different parts |
| `use_skip_indexes_if_final` | 1 | 25.6 | Allow skip indexes with FINAL |
| `use_skip_indexes_if_final_exact_mode` | 1 | 25.6 | Rescan newer parts to ensure correctness with skip indexes |
| `apply_prewhere_after_final` | 0 | 25.12 | Apply PREWHERE after deduplication (needed when PREWHERE references non-PK columns) |
| `enable_automatic_decision_for_merging_across_partitions_for_final` | 1 | 26.2 | Auto-enable `do_not_merge_across_partitions_select_final` when partition key is in PK |

### Partitioning

Expand Down Expand Up @@ -122,6 +135,37 @@ SELECT sum(value) FROM uint64_table FINAL format JSON;
"bytes_read": 480675040
```

### Vertical FINAL Algorithm (24.1+)

When `enable_vertical_final=1` (default since 24.1), ClickHouse uses a different deduplication strategy:
- Marks duplicate rows as deleted instead of merging them immediately
- Filters deleted rows in a later processing step
- Reads different columns from different parts in parallel

This improves performance for queries that read only a subset of columns, as non-ORDER BY columns can be read independently from different parts.

### PREWHERE and Row Policies with FINAL (25.12+)

By default, PREWHERE and row policies are applied **before** FINAL deduplication. This can cause incorrect results when:
- PREWHERE references columns that differ across duplicate rows
- Row policies should filter based on the "winning" row values after deduplication

Use these settings when needed:
- `apply_prewhere_after_final=1` - Apply PREWHERE after deduplication
- `apply_row_policy_after_final=1` - Apply row policies after deduplication

Example problem: if you have `ReplacingMergeTree` with a `deleted` column and PREWHERE filters on it, without `apply_prewhere_after_final=1` you may get wrong results because PREWHERE sees rows before FINAL picks the winner.

### Performance Tuning

**For wide tables (many columns):**
- `enable_vertical_final=1` (default) reads only needed columns in parallel
- Ensure `max_final_threads` is not set to 1

**For partitioned tables:**
- Use `do_not_merge_across_partitions_select_final=1` when partitions are pre-optimized
- Since 26.2, `enable_automatic_decision_for_merging_across_partitions_for_final=1` (default) auto-enables this when partition key columns are included in PRIMARY KEY

**For tables with skip indexes:**
- Both `use_skip_indexes_if_final` and `use_skip_indexes_if_final_exact_mode` are enabled by default since 25.6
- Skip indexes on PRIMARY KEY columns have lower overhead (no extra rescan needed since 26.1)
Loading
Loading