feat: increase ClickHouse and Zookeeper memory to handle larger workl… #507
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…oads ClickHouse scaled 4x (1Gi→4Gi request, 2Gi→8Gi limit) to support increased trace volume. Zookeeper scaled proportionally (256Mi→1Gi, 512Mi→2Gi) to maintain coordination at new scale. Evidence of memory pressure: - 74 MEMORY_LIMIT_EXCEEDED errors in 15 seconds during merge operations - ClickHouse auto-lowered all caches to 1GiB due to "limited RAM" - max_server_memory_usage constrained to 1.80 GiB (90% of 2Gi limit) - Actual usage during merges: 1.61-1.95 GiB, consistently exceeding limits Supporting evidence: /tmp/clickhouse-memory-evidence.md Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Collaborator
Author
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
…oads
ClickHouse scaled 4x (1Gi→4Gi request, 2Gi→8Gi limit) to support increased trace volume. Zookeeper scaled proportionally (256Mi→1Gi, 512Mi→2Gi) to maintain coordination at new scale.
Evidence of memory pressure:
Supporting evidence: