Skip to content

fix tests in BatchedAlignedSeriesReadChunkCompactionTest#16885

Open
kabo87777 wants to merge 1 commit intoapache:masterfrom
kabo87777:fix-3
Open

fix tests in BatchedAlignedSeriesReadChunkCompactionTest#16885
kabo87777 wants to merge 1 commit intoapache:masterfrom
kabo87777:fix-3

Conversation

@kabo87777
Copy link

Fix flaky tests in BatchedAlignedSeriesReadChunkCompactionTest

Description

This PR fixes 12 flaky tests in BatchedAlignedSeriesReadChunkCompactionTest that were failing intermittently due to non-deterministic iteration order of collections.

Root Cause

The getPaths() method in AbstractCompactionTest.java had two sources of non-determinism:

  1. HashSet usage: The method used HashSet<IFullPath> which does not guarantee iteration order
  2. Unsorted schemaMap.values(): The measurementSchemas list was created directly from schemaMap.values() without sorting, and since schemaMap is backed by a ConcurrentHashMap, the iteration order is non-deterministic

This caused AlignedFullPath objects to be created with measurements in different orders across test runs, leading to comparison failures when validating compaction results.

Fix

  1. Changed HashSet to LinkedHashSet to preserve insertion order
  2. Sorted measurementSchemas by measurement name before creating the list

Changes

File: iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/compaction/AbstractCompactionTest.java

// Before (non-deterministic):
Set<IFullPath> paths = new HashSet<>();
// ...
List<IMeasurementSchema> measurementSchemas = new ArrayList<>(schemaMap.values());

// After (deterministic):
Set<IFullPath> paths = new LinkedHashSet<>();
// ...
List<IMeasurementSchema> measurementSchemas =
    schemaMap.values().stream()
        .sorted(Comparator.comparing(IMeasurementSchema::getMeasurementName))
        .collect(Collectors.toList());

Tests Fixed

  • BatchedAlignedSeriesReadChunkCompactionTest#testCompactionWithDifferentCompressionTypeOrEncoding
  • BatchedAlignedSeriesReadChunkCompactionTest#testCompactionWithEmptyBatch
  • BatchedAlignedSeriesReadChunkCompactionTest#testSimpleCompactionByFlushChunk
  • BatchedAlignedSeriesReadChunkCompactionTest#testSimpleCompactionByFlushPage
  • BatchedAlignedSeriesReadChunkCompactionTest#testSimpleCompactionByWritePoint
  • BatchedAlignedSeriesReadChunkCompactionTest#testSimpleCompactionWithAllDeletedColumnByFlushChunk
  • BatchedAlignedSeriesReadChunkCompactionTest#testSimpleCompactionWithAllDeletedPageByFlushPage
  • BatchedAlignedSeriesReadChunkCompactionTest#testSimpleCompactionWithNotExistColumnByFlushChunk
  • BatchedAlignedSeriesReadChunkCompactionTest#testSimpleCompactionWithNullColumn
  • BatchedAlignedSeriesReadChunkCompactionTest#testSimpleCompactionWithNullColumnByFlushChunk
  • BatchedAlignedSeriesReadChunkCompactionTest#testSimpleCompactionWithPartialDeletedColumnByFlushChunk
  • BatchedAlignedSeriesReadChunkCompactionTest#testSimpleCompactionWithPartialDeletedPageByWritePoint

Verification

Verified using NonDex with 3 different random seeds - all tests pass consistently.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant