Skip to content

Test DVC remote mechanism in CI#430

Closed
parmentf wants to merge 15 commits into
mainfrom
services/affiliation-rnsr/test-dvc-remote
Closed

Test DVC remote mechanism in CI#430
parmentf wants to merge 15 commits into
mainfrom
services/affiliation-rnsr/test-dvc-remote

Conversation

@parmentf
Copy link
Copy Markdown
Contributor

@parmentf parmentf commented May 13, 2026

A small change to trigger the Github Actions.
Aim: check if all services are touched by the #429 problem.

@parmentf parmentf self-assigned this May 13, 2026
@parmentf parmentf added documentation Improvements or additions to documentation test Improve, fix or add tests labels May 13, 2026
@parmentf
Copy link
Copy Markdown
Contributor Author

Same problem. @revolj your Dockerfile is right. The problem is reproductible. :(

@parmentf
Copy link
Copy Markdown
Contributor Author

Copilot explanation:

I'll help you find a solution for the failing job. Let me start by retrieving the logs for that job.Now let me get the workflow file to understand the context better:## Solution for Job 75754848427

Root Cause: The dvc pull command is failing during Docker image build because:

  1. Missing DVC cache files: The required cache file d84a9289507bae25372f1b7ae169ef4f.dir is missing from both local cache and remote WebDAV storage
  2. WebDAV timeout issues: Multiple read timeouts when attempting to fetch from the remote storage (webdavs)
  3. Checkout failure: DVC cannot checkout the models directory without the cache files

Key Error:

dvc.exceptions.CheckoutError: Checkout failed for following targets:
models
Is your cache up to date?

Recommended Solutions

Option 1: Add DVC Cache Retry Logic (Recommended)

Modify the Dockerfile to add timeout and retry configuration for DVC pull:

RUN dvc pull -v \
    --remote-config timeout=120 \
    --remote-config connect-timeout=60 \
    --retry 3

Or update .dvc/config to include timeouts:

['remote "storage"']
    url = webdavs://...
    timeout = 120
    connect_timeout = 60

Option 2: Skip Models During Build (If Optional)

If the models directory is not critical for the build:

RUN dvc pull -v --force || true

Or configure .dvcignore to exclude it from pulls if not needed during build.

Option 3: Pre-cache in Docker Build Context

Push DVC cache to a persistent layer before pulling:

COPY .dvc/cache /dvc/.dvc/cache
RUN dvc pull -v

Option 4: Increase Build Timeout

The WebDAV server may need more time. Increase Docker's timeout in your CI/CD or build script to accommodate slow remote storage responses.

Immediate Action: Check that the WebDAV storage service (WEBDAV_URL) is responsive and that the cache file hasn't been purged from remote storage. The repeated 30-second timeouts suggest network or server latency issues.

@parmentf
Copy link
Copy Markdown
Contributor Author

La suggestion de copilot ne tient pas la route: les options n'existent même pas pour la commande dvc pull.

@parmentf
Copy link
Copy Markdown
Contributor Author

Le test qui avait échoué mercredi dernier a réussi simplement en le relançant (mais il a mis 23 minutes au lieu de 5).

@parmentf parmentf closed this May 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation test Improve, fix or add tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant