CASSSIDECAR-340: Add default process-based lifecycle provider to support the lifecycle API #277

pauloricardomg · 2025-11-07T21:57:58Z

Changes summary

Functionality

ProcessRuntimeConfiguration: represents the runtime configuration of a cassandra process, like environment variables, cassandra_home, cassandra_conf, etc. defined in sidecar.yaml
ProcessLifecycleProvider.start: constructs Cassandra runtime configuration from instance and global parameters and uses java ProcessBuilder API to start a Cassandra process writing a pidfile to the lifecycle state directory.
ProcessLifecycleProvider.stop: uses the pidfile to fetch process handle and send SIGKILL to stop the Cassandra process gracefully.
ProcessLifecycleProvider.isRunning: a Cassandra process is considered running if a PID file exists in the lifecycle state directory and a process with that PID is currently running.
InstanceMetadata and InstanceConfiguration: expose storageDir and instance-specific lifecycleOptions

Documentation

Add examples/lifecycle/README.md with an example on starting/stopping cassandra via lifecycle API
Update default sidecar.yaml with example and comments on how to enable ProcessLifecycleProvider

Infrastructure

Update build.gradle, integrationTestTask.gradle and gradle.properties to support running ProcessLifecycleProvider integration tests
Add scripts/build-cassandra-tarball.sh and update scripts/build-dtest-jars.sh` to build tarballs (requirement for running integration tests)
Update .circleci/config.yml to download cassandra tarballs for ProcessLifecycleProviderIntegrationTest and run integration tests for each supported version (4.0/5.0/5.1)

Testing

Add unit tests for ProcessLifecycleProvider to check runtime configuration is being constructed correctly and basic workflow (start/stop/status) with FakeLifecycleProvider
Add unit tests for ProcessRuntimeConfiguration to test start command construction and precondition validations
ProcessLifecycleProviderIntegrationTest: configures a local Cassandra install from a tarball in a temporary directory and run tests from LifecycleProviderIntegrationTester (start/stop/status)
- TestFileUtils: contains utilities to install the cassandra tarball for ProcessLifecycleProviderIntegrationTest

arjunashok

Added some comments. Also, please add an entry for these changes into Changes.txt.

arjunashok · 2025-11-28T04:21:33Z

examples/lifecycle/setup.sh

+echo "Creating configuration"
+cp -r ${CASSANDRA_HOME}/conf/* ${CASSANDRA_CONF}
+sed "s#\$cassandraHome#${CASSANDRA_HOME}#g" ${SIDECAR_YAML_TEMPLATE} > ${SIDECAR_YAML}
+sed -i "s#\$baseDir#${NODE_DIR}#g" ${SIDECAR_YAML}


This was complaining on a Mac. The format that worked was -

sed -i '' "s#\$baseDir#${NODE_DIR}#g" ${SIDECAR_YAML}

Good catch, fixed on 9d08a6d

arjunashok · 2025-11-28T04:44:01Z

server/src/main/java/org/apache/cassandra/sidecar/lifecycle/ProcessLifecycleProvider.java

+            String stdoutLocation = getStdoutLocation(runtimeConfig.instanceName());
+            String stderrLocation = getStderrLocation(runtimeConfig.instanceName());
+            String pidFileLocation = getPidFileLocation(runtimeConfig.instanceName());
+            ProcessBuilder processBuilder = runtimeConfig.buildStartCommand(pidFileLocation,


Seems like the additional files created here - stdout/stderr - are appended-to without rotation/cleanup and could result in disk space issues.

Also, there is a possibility of stale PID files if the process terminates abnormally. We would need some cleanup mechanism for such cases.

Seems like the additional files created here - stdout/stderr - are appended-to without rotation/cleanup and could result in disk space issues.

These files are just startup logs, not the entire Cassandra logs which are logged to $CASSANDRA_LOG_DIR with proper rotation. In order to not grow the scope of this PR while keeping the file sizes bounded I updated this patch to overwrite the pervious startup logs instead of appending indefinitely on c1eb011. If needed we can add append+rotation support later.

Also, there is a possibility of stale PID files if the process terminates abnormally. We would need some cleanup mechanism for such cases.

Good call. I've enhanced ProcessLifecycleProvider.isCassandraProcessRunning to auto-remove the PID file when the PID file exists but no Cassandra process is running, as well ensure the PID file is removed on ProcessLifecycleProvider.stop (0b65e3e).

arjunashok · 2025-12-01T23:37:49Z

server/src/main/java/org/apache/cassandra/sidecar/lifecycle/ProcessLifecycleProvider.java

+            LOG.info("Starting Cassandra instance {} with command: {}", runtimeConfig.instanceName(), processBuilder.command());
+
+            Process process = processBuilder.start();
+            process.waitFor(CASSANDRA_PROCESS_TIMEOUT_MS, TimeUnit.MILLISECONDS); // blocking call, make async?


As you have called out, consider using some sort of a periodic check for process, Instead of blocking the thread. Likewise for the stop process below.

Suggested change

process.waitFor(CASSANDRA_PROCESS_TIMEOUT_MS, TimeUnit.MILLISECONDS); // blocking call, make async?

vertx.setPeriodic(1000, timerId -> {

if (isCassandraProcessRunning(instance)) {

vertx.cancelTimer(timerId);

LOG.info("Started Cassandra instance {} with PID {}", runtimeConfig.instanceName(),

readPidFromFile(Path.of(pidFileLocation)));

} else if (System.currentTimeMillis() > startTime + CASSANDRA_PROCESS_TIMEOUT_MS) {

vertx.cancelTimer(timerId);

LOG.error("Failed to start Cassandra instance {} within timeout", runtimeConfig.instanceName());

// Handle timeout error

}

});

My understanding is that vertx.setPeriodic will return immediatelly right ? In which case, LifecycleProvider.start could return before the Cassandra process is initialized.

The issue is that LifecycleManager.submitStartTask assumes LifecycleProvider.start is synchronous and will only return after the process is guaranteed to be started. Are you suggesting making LifecycleProvider.start asynchronous and return a Future instead? Since this is a departure from the current synchronous design and may require many changes I would prefer to do this in a follow-up ticket if that works for you.

Yep, valid point about the architectural impact of going fully async. Follow-up ticket makes sense.

I was hoping to move away from blocking the worker thread from waiting on a kernel system call, towards a Java synchronization primitive (like CountDownLatch), which is more efficient and interruptible. Though I recognize this would be an intermediate step - if we're planning to eventually adopt an async interface. We'd need to weigh whether this intermediate improvement justifies the additional churn.

Additionally process.waitFor(timeout) returns false when it times out, but we don't check the return value. Wondering if it is worth detecting the status of the process start for better failure handling.

arjunashok · 2025-12-01T23:39:47Z

server/src/main/java/org/apache/cassandra/sidecar/lifecycle/ProcessLifecycleProvider.java

+            {
+                LOG.info("Stopping process of Cassandra instance {} with PID {}.", casCfg.instanceName(), pid);
+                CompletableFuture<ProcessHandle> terminationFuture = processHandle.get().onExit();
+                processHandle.get().destroy();  // blocking call, make async?


Consider a force destroy destroyForcibly() on timeout with appropriate logging.

done on 56ca1f6

server/src/main/java/org/apache/cassandra/sidecar/lifecycle/ProcessLifecycleProvider.java

arjunashok · 2025-12-01T23:46:29Z

server/src/main/java/org/apache/cassandra/sidecar/lifecycle/ProcessLifecycleProvider.java

+        this.lifecycleDir = params.get(OPT_STATE_DIR);
+        this.defaultCassandraHome = params.get(OPT_CASSANDRA_HOME);


Should we fail-fast if these directories do not exist or do not have appropriate permissions?

fixed on 369363a

arjunashok

Thanks for addressing the comments. Looks good. Added a comment to fix a spot bugs failure.

Please add an entry to Changes.txt, and share a link to the CI build.

arjunashok · 2025-12-15T07:33:43Z

server/src/main/java/org/apache/cassandra/sidecar/lifecycle/ProcessLifecycleProvider.java

+            ProcessBuilder pb = new ProcessBuilder("ps", "-p", String.valueOf(pid), "-o", "args=");
+            Process proc = pb.start();
+            try (BufferedReader reader = new BufferedReader(
+                    new InputStreamReader(proc.getInputStream())))


To address the spotbugs error

Suggested change

new InputStreamReader(proc.getInputStream())))

new InputStreamReader(proc.getInputStream(), StandardCharsets.UTF_8)))

updated on b643899

Empty string argument to -i works on both BSD and GNU sed.

- Verify process is Cassandra before considering it running - Auto-remove stale PID files when process missing or not Cassandra - Add cleanup in stopCassandra() and improved process validation - Add unit tests for stale PID scenarios

Co-authored-by: Arjun Ashok <arjun.diablo@gmail.com>

When stopping a Cassandra process, if the graceful destroy() does not complete within the timeout period, the code now calls destroyForcibly() to ensure the process is terminated. If destroyForcibly() also fails (returns false), an exception is thrown.

Validates that state directory and cassandra home exist and have appropriate permissions at construction time.

Configure -PtarballVersion for heavyweight integration tests to fix ProcessLifecycleProviderIntegrationTest which requires cassandra.test.tarball_path.

The sidecar.version resource file is generated during the jar task. Without running jar first, SidecarVersionProvider throws IllegalStateException when trying to read the missing resource. This aligns GitHub Actions with CircleCI which already includes jar before integrationTestHeavyWeight.

pauloricardomg · 2025-12-23T21:53:01Z

Thanks for the review @arjunashok - updated github actions to support new intergration tests for lifecycle provider. All the checks have passed.

arjunashok

LGTM, +1. Thanks for addressing the comments.

Couple of nits:

Consider addressing the comment here to handle timeout return value.
Add entry to Changes.txt

arjunashok reviewed Dec 2, 2025

View reviewed changes

pauloricardomg force-pushed the CASSSIDECAR-340 branch 4 times, most recently from 995176a to e025874 Compare December 5, 2025 23:39

arjunashok reviewed Dec 15, 2025

View reviewed changes

pauloricardomg force-pushed the CASSSIDECAR-340 branch 4 times, most recently from b071a11 to d7d8098 Compare December 22, 2025 19:17

pauloricardomg and others added 12 commits December 23, 2025 11:42

Add ProcessLifecycleProvider

9068785

Make sed command cross-compatible with macOS and Linux

5cbdacb

Empty string argument to -i works on both BSD and GNU sed.

Overwrite instead of appending startup logs to avoid unbounded growth

95ec387

Enhance stale PID file detection and cleanup

585c638

- Verify process is Cassandra before considering it running - Auto-remove stale PID files when process missing or not Cassandra - Add cleanup in stopCassandra() and improved process validation - Add unit tests for stale PID scenarios

Allow non-snapshot tarballs to be supplied

b7e8af2

Fix checkstyle violations

4927f3c

Apply suggestion from @arjunashok

e63bd29

Co-authored-by: Arjun Ashok <arjun.diablo@gmail.com>

Add fail-fast directory validation in constructor

369363a

Validates that state directory and cassandra home exist and have appropriate permissions at construction time.

Use explicit UTF-8 charset in InputStreamReader and Files.readString

b643899

Add CASSSIDECAR-340 entry to CHANGES.txt

e4baad9

Add cassandra tarball support to GitHub Actions heavyweight tests

f9a6c9f

Configure -PtarballVersion for heavyweight integration tests to fix ProcessLifecycleProviderIntegrationTest which requires cassandra.test.tarball_path.

pauloricardomg force-pushed the CASSSIDECAR-340 branch from d7d8098 to f9a6c9f Compare December 23, 2025 16:56

arjunashok approved these changes Jan 2, 2026

View reviewed changes

-            process.waitFor(CASSANDRA_PROCESS_TIMEOUT_MS, TimeUnit.MILLISECONDS); // blocking call, make async?
+        vertx.setPeriodic(1000, timerId -> {
+            if (isCassandraProcessRunning(instance)) {
+                vertx.cancelTimer(timerId);
+                LOG.info("Started Cassandra instance {} with PID {}", runtimeConfig.instanceName(),
+                         readPidFromFile(Path.of(pidFileLocation)));
+            } else if (System.currentTimeMillis() > startTime + CASSANDRA_PROCESS_TIMEOUT_MS) {
+                vertx.cancelTimer(timerId);
+                LOG.error("Failed to start Cassandra instance {} within timeout", runtimeConfig.instanceName());
+                // Handle timeout error
+            }
+        });

		this.lifecycleDir = params.get(OPT_STATE_DIR);
		this.defaultCassandraHome = params.get(OPT_CASSANDRA_HOME);

	new InputStreamReader(proc.getInputStream())))
	new InputStreamReader(proc.getInputStream(), StandardCharsets.UTF_8)))

CASSSIDECAR-340: Add default process-based lifecycle provider to support the lifecycle API #277

Are you sure you want to change the base?

CASSSIDECAR-340: Add default process-based lifecycle provider to support the lifecycle API #277

Uh oh!

Conversation

pauloricardomg commented Nov 7, 2025

Changes summary

Functionality

Documentation

Infrastructure

Testing

Uh oh!

arjunashok left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pauloricardomg Dec 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

arjunashok left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pauloricardomg commented Dec 23, 2025

Uh oh!

arjunashok left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

pauloricardomg Dec 5, 2025 •

edited

Loading