CASSANDRA-21066 Full Repair for Tracked Keyspaces #4565
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…aces This comes with lots of comments and a handful of integration tests, but the tests aren't passing and it coul use more unit test coverage. There's a refactor (introducing SyncTasks) that's only partially complete. The rest of this commit message is a description of how full repair is intended to work for tracked keyspaces. Tracked keyspaces cannot accept new data without first registering it in the log. Any unreconciled data that isn't present in the log will break read monotonicity, since mutation tracking uses a single data read and can only read reconcile mutation IDs that are present in the log. For more information about how bulk transfers work on tracked keyspaces, see TrackedImportTransfer. Full repair sync tasks also deliver data to replicas, and require integration with the log just like imports do. For more details on a read anomaly that could happen without integration with the bulk transfer machinery, see TrackedKeyspaceRepairSupportTest#testFullRepairPartiallyCompleteAnomaly. The general design of this integration is to give repair SyncTasks the same two-phase commit as import transfers, where we stream SSTables to a pending/ directory, then once sufficient streams complete successfully, we "activate" those streams and move them out of the pending directory and into the live set. The first step is to ensure that each SyncTask is aligned to a single Mutation Tracking shard, by splitting SyncTasks along the shard boundaries. Each SyncTask will then stream data within a single shard, and permit us to assign a single transfer ID to each SyncTask. Each participant in a repair may receive different SyncTasks (or none at all, if they're already in-sync). This means that TransferActivation needs to be made more flexible, and support a single TransferActivation with multiple plan IDs, or no plan IDs at all. This increase in flexibility has not yet been implemented. patch by Abe Ratnofsky; reviewed by Caleb Rackliffe and ? for CASSANDRA-21066
maedhroz
force-pushed
the
CASSANDRA-21066-maedhroz
branch
from
9edc6c2 to
d74f11b
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
WIP: Checkpointing progress for full repair support for tracked keyspaces
This comes with lots of comments and a handful of integration tests, but
the tests aren't passing and it coul use more unit test coverage.
There's a refactor (introducing SyncTasks) that's only partially
complete.
The rest of this commit message is a description of how full repair
is intended to work for tracked keyspaces.
Tracked keyspaces cannot accept new data without first registering it in
the log. Any unreconciled data that isn't present in the log will break
read monotonicity, since mutation tracking uses a single data read and
can only read reconcile mutation IDs that are present in the log. For
more information about how bulk transfers work on tracked keyspaces, see
TrackedImportTransfer.
Full repair sync tasks also deliver data to replicas, and require
integration with the log just like imports do. For more details on a
read anomaly that could happen without integration with the bulk
transfer machinery, see
TrackedKeyspaceRepairSupportTest#testFullRepairPartiallyCompleteAnomaly.
The general design of this integration is to give repair SyncTasks the
same two-phase commit as import transfers, where we stream SSTables to a
pending/ directory, then once sufficient streams complete successfully,
we "activate" those streams and move them out of the pending directory
and into the live set.
The first step is to ensure that each SyncTask is aligned to a single
Mutation Tracking shard, by splitting SyncTasks along the shard
boundaries. Each SyncTask will then stream data within a single shard,
and permit us to assign a single transfer ID to each SyncTask.
Each participant in a repair may receive different SyncTasks (or none at
all, if they're already in-sync). This means that TransferActivation
needs to be made more flexible, and support a single TransferActivation
with multiple plan IDs, or no plan IDs at all. This increase in
flexibility has not yet been implemented.