Skip to content

Testing zero-copy bugs fixes (not for merging)#1156

Closed
szetszwo wants to merge 173 commits intoapache:masterfrom
szetszwo:RATIS-2164
Closed

Testing zero-copy bugs fixes (not for merging)#1156
szetszwo wants to merge 173 commits intoapache:masterfrom
szetszwo:RATIS-2164

Conversation

@szetszwo
Copy link
Copy Markdown
Contributor

@szetszwo szetszwo commented Sep 28, 2024 •
edited
Loading

This fix will be split into multiple JIRAs: RATIS-2164, RATIS-2151, RATIS-2173

The following are the bugs found so far:

  1. LeakDetector: asserted allLeaks is non-empty but printed "allLeaks.size = 0"
    • Another bug: Tracks are added to the set before calling retain. Without calling retain at all, it is not a leak.
  2. SimpleTracing and AdvancedTracing: the methods should be synchronized.
    • Minor presentation problem: AdvancedTracing should have a single track list instead of retainsTraces and releaseTraces.
  3. GrpcClientProtocolService.UnorderedRequestStreamObserver.processClientRequest(..) should use try-finally.
  4. GrpcLogAppender.appendLog(..) calls release() incorrectly for exception.
  5. LogAppenderDefault.sendAppendEntriesWithRetries(..) calls release() incorrectly for exception.
  6. LogSegment cache can release an entry multiple times.
  7. LogSegment.loadCache(..) should call retain() for cache hit.
  8. SegmentedRaftLog.retainLog(..): between getting the entry and calling retain(), the entry can be released. The "fail to retain" exception, if there is any, can be ignored since It is the same as a cache miss. See RATIS-2159. TestRaftWithSimulatedRpc could "fail to retain". #1153
  9. SegmentedRaftLog.retainEntryWithData(..) should release for exception.
  10. Test bug: the log entries stored in SimpleStateMachine4Testing can be released.
  11. LogSegment: New entries can be added after EntryCache is closed.
  12. MemoryRaftLog has similar problems as in SegmentedRaftLog.
  13. SegmentedRaftLogWorker should clean up unfinished tasks in the queue after stopped running.

duongkame and others added 30 commits January 12, 2024 19:00
@duongkame
RATIS-1925. Support Zero-Copy in GrpcClientProtocolService (apache#1007)
84285d3
@ivandika3
RATIS-1998. Add watch request metrics (apache#1009)
59dfd35
@adoroszlai
RATIS-1989. Intermittent timeout in TestStreamObserverWithTimeout (ap…
2062058 
…ache#1012)
@szetszwo
RATIS-1987. Intermittent NPE in TestGrpcZeroCopy. (apache#1013)
863c2e8
@szetszwo
RATIS-1999. Move RaftServerJmxAdapter out from RaftServerImpl. (apach…
6671ef6 
…e#1010)
@szetszwo
RATIS-2001. TransactionContext can be wrongly reused. (apache#1015)
2448256
@adoroszlai
RATIS-2005. Fix URF_UNREAD_FIELD in MetricServerCallListener (apache#…
eae0d18 
…1018)
@adoroszlai
RATIS-2006. Fix DM_EXIT in filestore Client (apache#1019)
859a410
@adoroszlai
RATIS-2002. Fix findbugs warnings in LogSegmentPath (apache#1017)
14eb0bf
@duongkame
RATIS-1990. Refactor appendEntries processing to support reference co…
5d4ec69 
…unt (apache#1011)
@adoroszlai
RATIS-2003. Fix IT_NO_SUCH_ELEMENT in InstallSnapshotRequests (apache…
82f7b3b 
…#1016)
@adoroszlai
RATIS-2004. Fix EQ_COMPARETO_USE_OBJECT_EQUALS in LogSegment (apache#…
fc5c725 
…1020)
@duongkame
RATIS-2009. ReferenceCount should work for all LogEntry types. (apach…
82c31ea 
…e#1021)
@duongkame
RATIS-1934. Support Zero-Copy in GrpcServerProtocolService (apache#1014)
d459eb6
@adoroszlai
RATIS-2010. Remove dependency on spotbugs-annotations (apache#1022)
561cd6d
@szetszwo
RATIS-2013. OrderedAsync retry results an IllegalStateException in Gr…
c1e0ec2 
…pcClientProtocolService. (apache#1026)
@nandakumar131
RATIS-1972. Add junit 5 dependencies in ratis-server. (apache#1003)
0e24757
@szetszwo
RATIS-2012. Client should not retry after close. (apache#1025)
ea1341b
@ivandika3
RATIS-2008. Follower should recognize candidate if the candidate is t…
79923c2 
…he same peer as the current valid leader (apache#1024)
@duongkame
RATIS-2007. Zero-copy buffers are not released (apache#1027)
e2aaddc
@adoroszlai
RATIS-2015. Node.js 16 actions are deprecated (apache#1028)
a18d08c
@szetszwo
RATIS-2011. When a log entry is truncated, remove TransactionContext. (…
5560718 
…apache#1029)
@duongkame
RATIS-2017: checkAndEvictCache must be done in writeLock. (apache#1031)
7d587ec
@symious
RATIS-2016. Correct NotificationInstallSnapshot's index (apache#1030)
035cd96
@nandakumar131
RATIS-1974. Migrate ratis-metrics-default tests to Junit 5 (apache#1034)
9cd74a2
@duongkame
RATIS-2018. Zero-copy buffers are not released - 2nd chunk (apache#1032)
c4b10fa
@nandakumar131
RATIS-1973. Migrate ratis-examples tests to Junit 5. (apache#1035)
cf897f0
@OneSizeFitsQuorum
RATIS-2019 Fixed abnormal exit of StateMachineUpdater (apache#1033)
d2f1c57
@duongkame
RATIS-1978. Add tests assertions to verify all zero-copy messages are…
c46da22 
… released properly (apache#1023)
@nandakumar131
RATIS-1975. Migrate ratis-metrics-dropwizard3 tests to Junit 5. (apac…
a2cb49e 
…he#1037)
133tosakarin
133tosakarin reviewed Oct 7, 2024
View reviewed changes
Comment thread ratis-common/src/main/java/org/apache/ratis/util/DataBlockingQueue.java
private final Condition notEmpty = lock.newCondition();

private boolean closed = false;

Copy link
Copy Markdown
Contributor

@133tosakarin 133tosakarin Oct 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not use atomic?

Copy link
Copy Markdown
Contributor Author

@szetszwo szetszwo Oct 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is already a lock.

Copy link
Copy Markdown
Contributor

@133tosakarin 133tosakarin Oct 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. :)

133tosakarin
133tosakarin approved these changes Oct 7, 2024
View reviewed changes
Comment thread ratis-common/src/main/java/org/apache/ratis/util/DataBlockingQueue.java
private final Condition notEmpty = lock.newCondition();

private boolean closed = false;

Copy link
Copy Markdown
Contributor

@133tosakarin 133tosakarin Oct 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. :)

@szetszwo
Copy link
Copy Markdown
Contributor Author

szetszwo commented Oct 7, 2024

Finally, it is able to pass all the tests (with a few retries). Note that there are probably some other zero copy bugs. Will fix them separately.

@szetszwo szetszwo mentioned this pull request Oct 7, 2024
Merged
@szetszwo
Copy link
Copy Markdown
Contributor Author

szetszwo commented Oct 7, 2024 •
edited
Loading

This can pass all the tests (with a few retries). Since this change is quite big (56kB) and non-trivial, I will split this to a few JIRAs:

  1. The current JIRA RATIS-2164 for fixing LeakDetector. (I will leave this PR as-is and submit another PR.)
  2. RATIS-2151 for fixing gRPC.
  3. RATIS-2159 for fixing other non-gRPC cases.

I will see if (2) and (3) needed to be further split.

BTW, we should move LeakDetector enabling from MiniRaftClusterWithGrpc to MiniRaftCluster. It will be able to detect more failures.

@szetszwo szetszwo changed the title RATIS-2164. LeakDetector has a race condition. Testing zero-copy bugs fixes (not for merging) Oct 7, 2024
This was referenced Oct 7, 2024
Merged
Merged
Merged
Merged
Merged
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Mar 29, 2026

This PR has been marked as stale due to 60 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in ~30 days.

@github-actions github-actions Bot added the stale label Mar 29, 2026
@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 3, 2026

Thank you for your contribution. This PR is being closed due to inactivity. Please contact a maintainer if you would like to reopen it.

@github-actions github-actions Bot closed this May 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@133tosakarin 133tosakarin 133tosakarin approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

stale

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.