Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/dependabot.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
version: 2
updates:
- package-ecosystem: "pip" # Confusingly, pip is correct for poetry
- package-ecosystem: "pip" # Dependabot uses pip for uv-managed pyproject files
directory: "/" # Location of package manifests
schedule:
interval: "monthly"
Expand Down
19 changes: 6 additions & 13 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,26 +31,19 @@ jobs:
run: |
./start_services.sh --no-jupyter -d

- name: Install poetry
run: pipx install poetry
- name: Install uv
run: pipx install uv

- name: Verify MLflow versions match
run: ./scripts/check_mlflow_versions.sh

- name: Check poetry lock file
run: poetry check --lock

- name: Remove existing virtual environment
run: |
poetry env remove python || true

- name: Install dependencies
run: |
poetry cache clear --no-interaction --all .
poetry install --with test --no-cache
uv lock --check
uv sync --group test

- name: Run all tests
run: MLFLOW_TRACKING_URI=http://localhost:8080 poetry run pytest
run: MLFLOW_TRACKING_URI=http://localhost:8080 uv run pytest

- name: Stop MLflow server
run: docker compose down
Expand Down Expand Up @@ -79,7 +72,7 @@ jobs:

- name: Execute all notebooks
run: |
docker exec modelplane-jupyter-1 poetry run python /app/test_notebooks.py
docker exec modelplane-jupyter-1 uv run python /app/test_notebooks.py

- name: Stop MLflow server
run: docker compose down
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -11,3 +11,4 @@ secrets.toml
*.txt
*.json
*.jsonl
modelplane.egg-info/
13 changes: 6 additions & 7 deletions Dockerfile.jupyter
Original file line number Diff line number Diff line change
Expand Up @@ -4,22 +4,21 @@ ENV PATH="/root/.local/bin:$PATH"
# Used for the notebook server
WORKDIR /app

# pipx needed for poetry installation
# pipx needed for uv installation script
# ssh client needed for installing private modelbench dependencies
# git needed dvc
RUN apt-get update && apt-get install -y pipx openssh-client git && \
pipx install poetry
COPY pyproject.toml poetry.lock README.md ./
pipx install uv
COPY pyproject.toml uv.lock README.md ./

RUN mkdir -p /root/.ssh && chmod 700 /root/.ssh
RUN git config --global core.sshCommand "ssh -o UserKnownHostsFile=/root/.ssh/known_hosts -o ForwardAgent=yes"
RUN ssh-keyscan github.com > /root/.ssh/known_hosts

RUN poetry install --no-interaction --no-ansi --no-root
RUN uv sync --frozen --no-install-project

COPY src/ ./src/
RUN echo $(poetry env info --path)
RUN poetry install --no-interaction --no-ansi
RUN uv sync --frozen

EXPOSE 8888
CMD ["poetry", "run", "jupyter", "notebook", "--ip", "0.0.0.0", "--allow-root", "--no-browser", "--ServerApp.root_dir=/app/flightpaths"]
CMD ["uv", "run", "jupyter", "notebook", "--ip", "0.0.0.0", "--allow-root", "--no-browser", "--ServerApp.root_dir=/app/flightpaths"]
2 changes: 1 addition & 1 deletion Dockerfile.mlflow
Original file line number Diff line number Diff line change
Expand Up @@ -3,5 +3,5 @@ FROM ghcr.io/mlflow/mlflow:v3.7.0
# The base image does not include various dependencies that are needed for
# the MLflow server. We assume a postgres backend, so we need psycopg2.
# We also need boto3 for S3 support, and google-cloud-storage for GCS support.
RUN pip install mlflow[auth]==3.7.0 psycopg2-binary==2.9.11 boto3==1.42.43 \
RUN pip install mlflow[auth]==3.7.0 psycopg2-binary==2.9.11 boto3==1.42.51 \
google-cloud-storage==3.9.0
5 changes: 3 additions & 2 deletions Dockerfile.mockvllm
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,9 @@ WORKDIR /app

COPY tests/notebooks/mock_vllm_server.py .

# versions chosen to match whats in poetry.lock as of 2025-07-09
RUN pip install fastapi==0.115.12 uvicorn==0.34.3
# versions chosen to match whats in uv.lock as of 2026-02-18
# TODO: maybe match the uv.lock, though it's not functionally important here
RUN pip install fastapi==0.129.0 uvicorn==0.41.0

EXPOSE ${VLLM_PORT}

Expand Down
10 changes: 5 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,31 +79,31 @@ Annotator and SUT responses will be cached (locally) unless you pass the

## CLI

You can also interact with modelplane via CLI. Run `poetry run modelplane --help`
You can also interact with modelplane via CLI. Run `uv run modelplane --help`
for more details.

*Important:* You must set the `MLFLOW_TRACKING_URI` environmental variable.
For example, if you've brought up MLFlow using the fully local docker compose process above,
you could run:
```
MLFLOW_TRACKING_URI=http://localhost:8080 poetry run modelplane get-sut-responses --sut_id {sut_id} --prompts tests/data/prompts.csv --experiment expname
MLFLOW_TRACKING_URI=http://localhost:8080 uv run modelplane get-sut-responses --sut_id {sut_id} --prompts tests/data/prompts.csv --experiment expname
```
After running the command, you'd see the `run_id` in the output from mlflow,
or you can get the `run_id` via the MLFlow UI.

### Basic Annotations
```
MLFLOW_TRACKING_URI=http://localhost:8080 poetry run modelplane annotate --annotator_id {annotator_id} --experiment expname --response_run_id {run_id}
MLFLOW_TRACKING_URI=http://localhost:8080 uv run modelplane annotate --annotator_id {annotator_id} --experiment expname --response_run_id {run_id}
```

#### Private Ensemble
If you have access to the private annotator, you can run directly with:
```
MLFLOW_TRACKING_URI=http://localhost:8080 poetry run modelplane annotate --annotator_id safety-v1.1 --experiment expname --response_run_id {run_id}
MLFLOW_TRACKING_URI=http://localhost:8080 uv run modelplane annotate --annotator_id safety-v1.1 --experiment expname --response_run_id {run_id}
```


### Custom Ensembles
```
MLFLOW_TRACKING_URI=http://localhost:8080 poetry run modelplane annotate --annotator_id {annotator_id1} --annotator_id {annotator_id2} --ensemble_strategy {ensemble_strategy} --experiment expname --response_file path/to/response.csv
MLFLOW_TRACKING_URI=http://localhost:8080 uv run modelplane annotate --annotator_id {annotator_id1} --annotator_id {annotator_id2} --ensemble_strategy {ensemble_strategy} --experiment expname --response_file path/to/response.csv
```
Loading