From 798f15f73d8c7716f480967b864c73a32e36f721 Mon Sep 17 00:00:00 2001
From: MOHITKOURAV01 <mohitdas852@gmail.com>
Date: Wed, 18 Mar 2026 13:04:47 +0530
Subject: [PATCH] [DOC] Update documentation for running tests with local
 Docker services

- Add 'Running Tests Against Local Services' section to docs/developer_setup.md
- Add local services testing instructions to CONTRIBUTING.md
- Document the Docker-based local test server setup (clone services repo,
  set permissions, start docker compose, set OPENML_USE_LOCAL_SERVICES env var)
- Include cleanup instructions

Fixes #1708
---
 CONTRIBUTING.md         | 42 ++++++++++++++++++++++
 docs/developer_setup.md | 78 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 120 insertions(+)

diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index d194525ef..17b7411e0 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -107,6 +107,48 @@ $env:OPENML_TEST_SERVER_ADMIN_KEY = "admin-key"
 export OPENML_TEST_SERVER_ADMIN_KEY="admin-key"
 ```
 
+#### Running Tests Against Local Services (Docker)
+
+The CI now runs tests against a local OpenML server using Docker instead of the remote test server.
+You can replicate this setup locally:
+
+1. **Clone the services repository** (from the parent directory of `openml-python`):
+   ```bash
+   git clone --depth 1 https://github.com/openml/services.git
+   cd services
+   ```
+
+2. **Set file permissions** required by the containers:
+   ```bash
+   chmod -R a+rw ./data
+   chmod -R a+rw ./logs
+   ```
+
+3. **Start the Docker services**:
+   ```bash
+   docker compose --profile rest-api --profile minio --profile evaluation-engine up -d
+   ```
+
+4. **Wait for the API to become healthy**:
+   ```bash
+   timeout 60s bash -c 'until [ "$(docker inspect -f {{.State.Health.Status}} openml-php-rest-api)" == "healthy" ]; do sleep 5; done'
+   ```
+
+5. **Run the tests** with the local services environment variable:
+   ```bash
+   export OPENML_USE_LOCAL_SERVICES="true"
+   pytest -n 4 --durations=20 --dist load -sv -m "not production_server"
+   ```
+
+6. **Cleanup** after testing:
+   ```bash
+   cd services
+   docker compose --profile rest-api --profile minio --profile evaluation-engine down
+   ```
+
+For more details, see the [Developer Setup Guide](https://openml.github.io/openml-python/main/developer_setup.html).
+
+
 ### Pull Request Checklist
 
 You can go to the `openml-python` GitHub repository to create the pull request by [comparing the branch](https://github.com/openml/openml-python/compare) from your fork with the `main` branch of the `openml-python` repository. When creating a pull request, make sure to follow the comments and structured provided by the template on GitHub.
diff --git a/docs/developer_setup.md b/docs/developer_setup.md
index 55a73fef9..e11d06d0f 100644
--- a/docs/developer_setup.md
+++ b/docs/developer_setup.md
@@ -193,6 +193,83 @@ Exclude production tests (local only):
 pytest -m "not production_server"
 ```
 
+### Running Tests Against Local Services
+
+The CI pipeline runs tests against a local OpenML server using Docker, instead of the remote
+test server. You can replicate this setup locally to avoid depending on the remote server.
+
+#### Prerequisites
+
+* **Docker**: Docker Desktop (ensure the daemon is running).
+* **Git**: To clone the services repository.
+
+#### 1. Clone the Services Repository
+
+```bash
+git clone --depth 1 https://github.com/openml/services.git
+cd services
+```
+
+#### 2. Configure File Permissions
+
+The containerized services need write access to the data and log directories:
+
+```bash
+chmod -R a+rw ./data
+chmod -R a+rw ./logs
+```
+
+#### 3. Start the Docker Services
+
+Launch the required service profiles (REST API, MinIO, and Evaluation Engine):
+
+```bash
+docker compose --profile rest-api --profile minio --profile evaluation-engine up -d
+```
+
+Wait for the PHP API to become healthy:
+
+```bash
+# Wait up to 60 seconds for the API to boot
+timeout 60s bash -c 'until [ "$(docker inspect -f {{.State.Health.Status}} openml-php-rest-api)" == "healthy" ]; do sleep 5; done'
+```
+
+Verify the services are running:
+
+```bash
+curl -sSfL http://localhost:8000/api/v1/xml/data/1 | head -n 15
+```
+
+#### 4. Run Tests Against Local Services
+
+Set the `OPENML_USE_LOCAL_SERVICES` environment variable and run pytest:
+
+=== "Linux/macOS"
+
+    ```bash
+    export OPENML_USE_LOCAL_SERVICES="true"
+    pytest -n 4 --durations=20 --dist load -sv -m "not production_server"
+    ```
+
+=== "Windows (PowerShell)"
+
+    ```shell
+    $env:OPENML_USE_LOCAL_SERVICES = "true"
+    pytest -n 4 --durations=20 --dist load -sv -m "not production_server"
+    ```
+
+When this environment variable is set, the test framework automatically redirects API calls
+to `http://localhost:8000` instead of the remote test server.
+
+#### 5. Cleanup
+
+After you're done testing, stop and remove the Docker containers:
+
+```bash
+cd services
+docker compose --profile rest-api --profile minio --profile evaluation-engine down
+```
+
 ### Admin Privilege Tests
 
 Certain tests require administrative privileges on the test server. These are skipped automatically unless an admin API key is provided via environment variables.
@@ -208,3 +285,4 @@ $env:OPENML_TEST_SERVER_ADMIN_KEY = "admin-key"
 ```bash
 export OPENML_TEST_SERVER_ADMIN_KEY="admin-key"
 ```
+