Compaction concurrency is not respected for the same user

[friedrichg]

Describe the bug
When configuring concurrency to 1. compactors will run more than one compaction per compactor for the same user at the same time ignoring the concurrency configuration.
For example: 24h and 12h compaction at the same time.

This is surprising and can cause extra need of space in disks that is almost never used.

To Reproduce

• Configure compactor concurrency to 1 (Default)
• Use 2h,12h,24h blocks
• Put 6x2h blocks and 2x12 blocks in the bucket
• Watch the logs run compaction for both compaction levels at the same time

Expected behavior
it should only do one compaction at the time if concurrency is set to 1.
There should be an integration test for this

Environment:
ALL

[adamalexandru4]

Root cause

BucketCompactor.Compact() has a for loop that repeatedly calls grouper.Groups() after each successful compaction (shouldRerun=true). The grouper returns up to compactionConcurrency groups per call, but since the loop calls it multiple times, the effective limit is unbounded.

With concurrency=1 and blocks [6×2h, 2×12h]:

• Iteration 1: Groups() → 1 group (6×2h → 12h), compacts, shouldRerun=true
• Iteration 2: Groups() → 1 group (3×12h → 24h), compacts, shouldRerun=true
• Iteration 3: Groups() → 0 groups, loop breaks

Result: 2 compactions in one pass despite concurrency=1. Each downloads blocks to local disk, causing unexpected disk usage spikes.

Proposed fix

Track cumulative groups returned across Groups() calls within one Compact() invocation. Once the total reaches compactionConcurrency, return empty:

func (g *ShuffleShardingGrouper) Groups(blocks map[ulid.ULID]*metadata.Meta) ([]*compact.Group, error) {
    remainingConcurrency := g.compactionConcurrency - g.totalGroupsPlanned
    if remainingConcurrency <= 0 {
        return nil, nil
    }
    // ... existing logic, using remainingConcurrency as the limit ...
    g.totalGroupsPlanned += len(outGroups)
    return outGroups, nil
}

This is safe because the grouper is created fresh in each compactUser() call, so totalGroupsPlanned resets every pass.

This fix currently covers:

• ShuffleShardingGrouper (sharding_strategy: shuffle-sharding)
• PartitionCompactionGrouper (sharding_strategy: shuffle-sharding + compaction_strategy: partitioning)

Gap: The DefaultBlocksGrouperFactory path (used when sharding_strategy: default or sharding is disabled) delegates to the Thanos DefaultGrouper, which is not patched. This affects single-tenant / non-sharded setups. I don't have any plan here, waiting for your advice.

[friedrichg]

Do you want to try and fix it? @adamalexandru4 I can give you the issue