Skip to content

DEVOPS-678: use new Artifactory virtual channels in rattler recipe #83

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sebhmg merged 8 commits into from
Mar 19, 2025
Merged

DEVOPS-678: use new Artifactory virtual channels in rattler recipe #83

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
a4726de
[DEVOPS-678] remove now unused workflow to build a conda recipe
sebhmg Mar 19, 2025
8b5ce66
[DEVOPS-678] accept conda channels as input besides repo names
sebhmg Mar 19, 2025
db93e95
[DEVOPS-678] use repo.prefix.dev instead of anaconda
sebhmg Mar 19, 2025
fb8c20f
[DEVOPS-678] use flexible channel priority
sebhmg Mar 19, 2025
e83dfa5
[DEVOPS-678] minor: adjust input parameter descriptions
sebhmg Mar 19, 2025
0fd1832
[DEVOPS-678] use strict channel priority
sebhmg Mar 19, 2025
2a525bd
[DEVOPS-678] use step output rather than env var
sebhmg Mar 19, 2025
228e7a7
[DEVOPS-678] fix wrong variable name
sebhmg Mar 19, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/reusable-jira-issue_to_jira.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ on:
required: true
type: string
components:
description: 'JIRA components (eg. [{"name": "geoh5py"}])'
description: 'JIRA components (e.g. [{"name": "geoh5py"}])'
required: false
type: string

Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/reusable-pre_commit.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ on:
required: false
type: string
os:
description: 'Matrix of OS to run against (eg. ["ubuntu-latest", "macos-latest"])'
description: 'List of OS to run against (e.g. ["ubuntu-latest", "macos-latest"])'
required: false
type: string
default: "['ubuntu-latest']"
Expand Down
7 changes: 3 additions & 4 deletions .github/workflows/reusable-python-build_poetry_package.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ on:
required: true
type: string
python-version:
description: 'Python version to use (eg: 3.10)'
description: 'Python version to use (e.g. 3.10)'
required: true
type: string
lfs:
Expand All @@ -20,13 +20,12 @@ on:
description: 'Version tag of the package to build'
required: true
type: string

virtual-repo-names:
description: 'Matrix of virtual repository names to publish on (eg. ["public-dev-pypi"])'
description: 'List of repository names to publish on (e.g. ["public-dev-pypi"])'
required: true
type: string
os:
description: 'OS to build against (eg. "ubuntu-latest")'
description: 'OS to build against (e.g. "ubuntu-latest")'
required: true
type: string
timeout-minutes:
Expand Down
7 changes: 3 additions & 4 deletions .github/workflows/reusable-python-build_setuptools_package.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ on:
required: true
type: string
python-version:
description: 'Python version to use (eg: 3.10)'
description: 'Python version to use (e.g. 3.10)'
required: true
type: string
lfs:
Expand All @@ -20,13 +20,12 @@ on:
description: 'Version tag of the package to build'
required: true
type: string

virtual-repo-names:
description: 'Matrix of virtual repository names to publish on (eg. ["public-dev-pypi"])'
description: 'List of repository names to publish on (e.g. ["public-dev-pypi"])'
required: true
type: string
os:
description: 'OS to build against (eg. "ubuntu-latest")'
description: 'OS to build against (e.g. "ubuntu-latest")'
required: true
type: string
timeout-minutes:
Expand Down
162 changes: 0 additions & 162 deletions .github/workflows/reusable-python-publish_conda_package.yml
View file
Open in desktop

This file was deleted.

8 changes: 4 additions & 4 deletions .github/workflows/reusable-python-publish_pypi_package.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ on:
workflow_call:
inputs:
package-manager:
description: 'Name of the package manager (eg. "poetry")'
description: 'Name of the package manager (e.g. "poetry")'
required: true
type: string
default: 'poetry'
Expand All @@ -13,7 +13,7 @@ on:
required: true
type: string
python-version:
description: 'Python version to use (eg: 3.10)'
description: 'Python version to use (e.g. 3.10)'
required: true
type: string
lfs:
Expand All @@ -26,11 +26,11 @@ on:
required: true
type: string
virtual-repo-names:
description: 'Matrix of virtual repository names to publish on (eg. ["public-pypi-dev"])'
description: 'List of virtual repository names to publish on (e.g. ["public-pypi-dev"])'
required: true
type: string
os:
description: 'OS to build against (eg. "ubuntu-latest")'
description: 'OS to build against (e.g. "ubuntu-latest")'
required: false
type: string
default: 'ubuntu-latest'
Expand Down
52 changes: 31 additions & 21 deletions .github/workflows/reusable-python-publish_rattler_package.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,24 +8,28 @@ on:
required: true
type: string
python-version:
description: 'Python version to use (eg: 3.10)'
description: 'Python version to use (e.g. 3.10)'
required: true
type: string
lfs:
description: 'Boolean to indicate if Github LFS is needed'
required: false
type: boolean
default: true
conda-channels:
description: 'List of conda channels to pull from (e.g. ["conda-forge", "pytorch"])'
required: true
type: string
source-repo-names:
description: 'Matrix of repository names to publish on (eg. ["public-conda-dev-local", "conda-forge-dev-remote"])'
description: 'List of repository names to pull from (e.g. ["geology-conda-dev", "geophysics-conda-dev"])'
required: true
type: string
publish-repo-names:
description: 'Matrix of virtual repository names to publish on (eg. ["public-conda-dev"])'
description: 'List of repository names to publish on (e.g. ["public-conda-dev"])'
required: true
type: string
os:
description: 'OS to build against (eg. "ubuntu-latest")'
description: 'OS to build against (e.g. "ubuntu-latest")'
required: false
type: string
default: 'ubuntu-latest'
Expand All @@ -47,6 +51,11 @@ defaults:
run:
shell: 'bash -l {0}'

env:
# cover for both conda and mamba
CONDA_CHANNEL_ALIAS: https://repo.prefix.dev
MAMBA_CHANNEL_ALIAS: https://repo.prefix.dev

jobs:
build_rattler_package:
name: Build package with rattler
Expand All @@ -67,31 +76,29 @@ jobs:
echo "Rattler recipe not found"
exit 1
fi
- name: Auths with JFrog channels
- name: Configure channels and Auth
id: config-channels
run: |
BUILDS_ARGS=""
BUILD_ARGS=""
RATTLER_AUTH_FILE=${{ runner.temp }}/credentials.json

repository_list=$(echo '${{ inputs.source-repo-names }}' | jq -r '.[]')
repository_count=$(echo '${{ inputs.source-repo-names }}' | jq '. | length')

# Check if there is more than one element
if [ "$repository_count" -ge 1 ]; then
repository_list=$(echo '${{ inputs.source-repo-names }}' | jq -r '.[]' | xargs)
if [ -n "$repository_list" ]; then
echo '{"${{ secrets.JFROG_ARTIFACTORY_URL }}": {"BasicHTTP": {"username": "github", "password":"${{ secrets.JFROG_ARTIFACTORY_TOKEN }}"} } }' > "$RATTLER_AUTH_FILE"
else
BUILDS_ARGS=" --channel conda-forge"
fi
echo "RATTLER_AUTH_FILE=${RATTLER_AUTH_FILE}" >> "$GITHUB_ENV"

# Loop through the repository names and add them to the command
for repo_name in ${repository_list}; do
BUILDS_ARGS+=" --channel https://${{ secrets.JFROG_ARTIFACTORY_URL }}/artifactory/api/conda/${repo_name}"
BUILD_ARGS+=" --channel https://${{ secrets.JFROG_ARTIFACTORY_URL }}/artifactory/api/conda/${repo_name}"
done

BUILDS_ARGS+=" --output-dir ${{ runner.temp }}/output"
BUILDS_ARGS+=" --channel-priority disabled"
channel_list=$(echo '${{ inputs.conda-channels }}' | jq -r '.[]' | xargs)
for channel_name in ${channel_list}; do
BUILD_ARGS+=" --channel ${channel_name}"
done

echo "RATTLER_AUTH_FILE=${RATTLER_AUTH_FILE}" >> "$GITHUB_ENV"
echo "BUILDS_ARGS=$BUILDS_ARGS" >> "$GITHUB_ENV"
echo "BUILD_ARGS=$BUILD_ARGS" >> "$GITHUB_OUTPUT"
- name: Check RATTLER_AUTH_FILE content
run: |
echo "RATTLER_AUTH_FILE=${{ env.RATTLER_AUTH_FILE }}"
Expand All @@ -104,7 +111,10 @@ jobs:
uses: prefix-dev/rattler-build-action@v0.2.32
with:
rattler-build-version: v0.34.1
build-args: ${{ env.BUILDS_ARGS }}
build-args: >-
--output-dir ${{ runner.temp }}/output
--channel-priority strict
${{ steps.config-channels.outputs.BUILD_ARGS }}
recipe-path: ${{ github.workspace}}/recipe.yaml
env:
RATTLER_AUTH_FILE: ${{ runner.temp }}/credentials.json
Expand All @@ -123,7 +133,7 @@ jobs:
needs: build_rattler_package
strategy:
matrix:
virtual-repo-name: ${{ fromJson(inputs.publish-repo-names) }}
publish-repo-name: ${{ fromJson(inputs.publish-repo-names) }}
steps:
- name: Download build artifact
uses: actions/download-artifact@v4
Expand All @@ -134,7 +144,7 @@ jobs:
uses: MiraGeoscience/CI-tools/.github/actions/reusable-python-publish_to_artifactory@main
with:
build-dir-path: 'build-dir/noarch'
artifactory-dir-path: ${{ matrix.virtual-repo-name}}/noarch
artifactory-dir-path: ${{ matrix.publish-repo-name}}/noarch
JFROG_ARTIFACTORY_URL: ${{ secrets.JFROG_ARTIFACTORY_URL }}
JFROG_ARTIFACTORY_TOKEN: ${{ secrets.JFROG_ARTIFACTORY_TOKEN }}
add_release_asset:
Expand Down
Loading