Skip to content

ENH: Transition package build test GHA to modern remote infrastructure #53

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jhlegarreta wants to merge 2 commits into
base: main
Choose a base branch
from
Open

ENH: Transition package build test GHA to modern remote infrastructure #53

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
9ecf568
ENH: Transition project configuration to `pyproject.toml`
jhlegarreta Feb 6, 2025
51389b4
ENH: Transition package build test GHA to modern remote infrastructure
jhlegarreta Feb 6, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
371 changes: 6 additions & 365 deletions .github/workflows/build-test-package.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,370 +2,11 @@ name: Build, test, package

on: [push,pull_request]

env:
itk-git-tag: "171fb2ba33a87041f99328a2f26612ff33aa9cc8"
itk-wheel-tag: "v5.3rc04.post3"
opencl-icd-loader-git-tag: "v2021.04.29"
opencl-headers-git-tag: "v2021.04.29"

jobs:
build-cxx:
runs-on: ${{ matrix.os }}
strategy:
max-parallel: 3
matrix:
os: [ubuntu-20.04, windows-2019, macos-11]
include:
- os: ubuntu-20.04
c-compiler: "gcc"
cxx-compiler: "g++"
cmake-build-type: "MinSizeRel"
- os: windows-2019
c-compiler: "cl.exe"
cxx-compiler: "cl.exe"
cmake-build-type: "Release"
- os: macos-11
c-compiler: "clang"
cxx-compiler: "clang++"
cmake-build-type: "MinSizeRel"

steps:
- uses: actions/checkout@v2

- name: Set up Python 3.8
uses: actions/setup-python@v2
with:
python-version: 3.8

- name: Install build dependencies
run: |
python -m pip install --upgrade pip
python -m pip install ninja

- name: Get specific version of CMake, Ninja
uses: lukka/get-cmake@v3.21.2

- name: Download OpenCL-SDK
if: matrix.os == 'macos-11'
run: |
cd ..
git clone --recursive https://github.com/KhronosGroup/OpenCL-SDK.git
Copy link
Copy Markdown
Member

@dzenanz dzenanz Feb 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This CI is somewhat customized. I don't think we can use the standard remote action in this repo.

cd OpenCL-SDK
shell: bash

- name: Download OpenCL-ICD-Loader
run: |
cd ..
git clone https://github.com/KhronosGroup/OpenCL-ICD-Loader
pushd OpenCL-ICD-Loader
git checkout ${{ env.opencl-icd-loader-git-tag }}
popd
pushd OpenCL-ICD-Loader/inc
git clone https://github.com/KhronosGroup/OpenCL-Headers
pushd OpenCL-Headers
git checkout ${{ env.opencl-headers-git-tag }}
popd
cp -r OpenCL-Headers/CL ./
popd
shell: bash

- name: Install pocl
if: matrix.os != 'windows-2019'
run: |
sudo conda config --add channels conda-forge
sudo conda config --set channel_priority strict
sudo conda install -c conda-forge pocl
shell: bash

- name: Download ITK
run: |
cd ..
git clone https://github.com/InsightSoftwareConsortium/ITK.git
cd ITK
git checkout ${{ env.itk-git-tag }}

- name: Build OpenCL-SDK
if: matrix.os == 'macos-11'
run: |
cd ..
mkdir OpenCL-SDK-build
cd OpenCL-SDK-build
cmake -DCMAKE_C_COMPILER:FILEPATH="${{ matrix.c-compiler }}" -DCMAKE_CXX_COMPILER="${{ matrix.cxx-compiler }}" -DBUILD_SHARED_LIBS:BOOL=ON -DCMAKE_BUILD_TYPE:STRING=${{ matrix.cmake-build-type }} -GNinja ../OpenCL-SDK
sudo cmake --build . --target install
shell: bash

- name: Build OpenCL-ICD-Loader
if: matrix.os != 'windows-2019'
run: |
cd ..
mkdir OpenCL-ICD-Loader-build
cd OpenCL-ICD-Loader-build
cmake -DCMAKE_C_COMPILER:FILEPATH="${{ matrix.c-compiler }}" -DBUILD_SHARED_LIBS:BOOL=ON -DCMAKE_CXX_COMPILER="${{ matrix.cxx-compiler }}" -DCMAKE_BUILD_TYPE:STRING=${{ matrix.cmake-build-type }} -DBUILD_TESTING:BOOL=OFF -GNinja ../OpenCL-ICD-Loader
sudo cmake --build . --target install

- name: Build OpenCL-ICD-Loader
if: matrix.os == 'windows-2019'
run: |
cd ..
mkdir OpenCL-ICD-Loader-build
cd OpenCL-ICD-Loader-build
call "C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Auxiliary\Build\vcvars64.bat"
cmake -DCMAKE_C_COMPILER:FILEPATH="${{ matrix.c-compiler }}" -DBUILD_SHARED_LIBS:BOOL=ON -DCMAKE_CXX_COMPILER="${{ matrix.cxx-compiler }}" -DCMAKE_BUILD_TYPE:STRING=${{ matrix.cmake-build-type }} -DBUILD_TESTING:BOOL=OFF -GNinja ../OpenCL-ICD-Loader
cmake --build . --target install
shell: cmd

- name: Build ITK
if: matrix.os != 'windows-2019'
run: |
cd ..
mkdir ITK-build
cd ITK-build
cmake -DCMAKE_C_COMPILER:FILEPATH="${{ matrix.c-compiler }}" -DBUILD_SHARED_LIBS:BOOL=ON -DCMAKE_CXX_COMPILER="${{ matrix.cxx-compiler }}" -DCMAKE_BUILD_TYPE:STRING=${{ matrix.cmake-build-type }} -DBUILD_TESTING:BOOL=OFF -GNinja ../ITK
ninja

- name: Build ITK
if: matrix.os == 'windows-2019'
run: |
cd ..
mkdir ITK-build
cd ITK-build
call "C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Auxiliary\Build\vcvars64.bat"
cmake -DCMAKE_C_COMPILER:FILEPATH="${{ matrix.c-compiler }}" -DBUILD_SHARED_LIBS:BOOL=ON -DCMAKE_CXX_COMPILER="${{ matrix.cxx-compiler }}" -DCMAKE_BUILD_TYPE:STRING=${{ matrix.cmake-build-type }} -DBUILD_TESTING:BOOL=OFF -GNinja ../ITK
ninja
shell: cmd

- name: Fetch CTest driver script
run: |
curl -L https://raw.githubusercontent.com/InsightSoftwareConsortium/ITK/dashboard/itk_common.cmake -O

- name: Configure CTest script
run: |
operating_system="${{ matrix.os }}"
cat > dashboard.cmake << EOF
set(CTEST_SITE "GitHubActions")
file(TO_CMAKE_PATH "\$ENV{GITHUB_WORKSPACE}/.." CTEST_DASHBOARD_ROOT)
file(TO_CMAKE_PATH "\$ENV{GITHUB_WORKSPACE}/" CTEST_SOURCE_DIRECTORY)
file(TO_CMAKE_PATH "\$ENV{GITHUB_WORKSPACE}/../build" CTEST_BINARY_DIRECTORY)
set(dashboard_source_name "${GITHUB_REPOSITORY}")
if(ENV{GITHUB_REF} MATCHES "master")
set(branch "-master")
set(dashboard_model "Continuous")
else()
set(branch "-${GITHUB_REF}")
set(dashboard_model "Experimental")
endif()
set(CTEST_BUILD_NAME "${GITHUB_REPOSITORY}-${operating_system}-\${branch}")
set(CTEST_UPDATE_VERSION_ONLY 1)
set(CTEST_TEST_ARGS \${CTEST_TEST_ARGS} PARALLEL_LEVEL \${PARALLEL_LEVEL})
set(CTEST_BUILD_CONFIGURATION "Release")
set(CTEST_CMAKE_GENERATOR "Ninja")
set(CTEST_CUSTOM_WARNING_EXCEPTION
\${CTEST_CUSTOM_WARNING_EXCEPTION}
# macOS Azure VM Warning
"ld: warning: text-based stub file"
)
set(dashboard_no_clean 1)
set(ENV{CC} ${{ matrix.c-compiler }})
set(ENV{CXX} ${{ matrix.cxx-compiler }})
if(WIN32)
set(ENV{PATH} "\${CTEST_DASHBOARD_ROOT}/ITK-build/bin;\$ENV{PATH}")
endif()
file(TO_CMAKE_PATH "\${CTEST_DASHBOARD_ROOT}/OpenCL-ICD-Loader/inc" OpenCL_INCLUDE_DIR)
find_library(OpenCL_LIBRARY OpenCL PATHS \${CTEST_DASHBOARD_ROOT}/OpenCL-ICD-Loader-build REQUIRED)
set(dashboard_cache "
OpenCL_INCLUDE_DIR:PATH=\${OpenCL_INCLUDE_DIR}
OpenCL_LIBRARY:FILEPATH=\${OpenCL_LIBRARY}
ITK_DIR:PATH=\${CTEST_DASHBOARD_ROOT}/ITK-build
BUILD_TESTING:BOOL=ON
")
string(TIMESTAMP build_date "%Y-%m-%d")
message("CDash Build Identifier: \${build_date} \${CTEST_BUILD_NAME}")
message("CTEST_SITE = \${CTEST_SITE}")
message("dashboard_cache = \${dashboard_cache}")
include(\${CTEST_SCRIPT_DIRECTORY}/itk_common.cmake)
EOF
cat dashboard.cmake
shell: bash

- name: Build and test
if: matrix.os != 'windows-2019'
run: |
ctest --output-on-failure -j 2 -VV -S dashboard.cmake -R "Doxygen|KWStyle"

- name: Build and test
if: matrix.os == 'windows-2019'
run: |
call "C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Auxiliary\Build\vcvars64.bat"
ctest --output-on-failure -j 2 -VV -S dashboard.cmake -R "Doxygen|KWStyle"
shell: cmd

build-windows-python-packages:
runs-on: windows-2019
strategy:
max-parallel: 2
matrix:
python-version-minor: [7, 8, 9, 10]
include:
- c-compiler: "cl.exe"
cxx-compiler: "cl.exe"
cmake-build-type: "MinSizeRel"

steps:
- name: Get specific version of CMake, Ninja
uses: lukka/get-cmake@v3.21.2

- uses: actions/checkout@v2
with:
path: "im"

- name: 'Install Python'
run: |
$pythonArch = "64"
$pythonVersion = "3.${{ matrix.python-version-minor }}"
iex ((new-object net.webclient).DownloadString('https://raw.githubusercontent.com/scikit-build/scikit-ci-addons/master/windows/install-python.ps1'))

- name: 'Fetch build dependencies'
run: |
mv im ../../
cd ../../im
curl -L "https://github.com/InsightSoftwareConsortium/ITKPythonBuilds/releases/download/${{ env.itk-wheel-tag }}/ITKPythonBuilds-windows.zip" -o "ITKPythonBuilds-windows.zip"
7z x ITKPythonBuilds-windows.zip -o/c/P -aoa -r
curl -L "https://data.kitware.com/api/v1/file/5c0ad59d8d777f2179dd3e9c/download" -o "doxygen-1.8.11.windows.bin.zip"
7z x doxygen-1.8.11.windows.bin.zip -o/c/P/doxygen -aoa -r
curl -L "https://data.kitware.com/api/v1/file/5bbf87ba8d777f06b91f27d6/download/grep-win.zip" -o "grep-win.zip"
7z x grep-win.zip -o/c/P/grep -aoa -r
shell: bash

- name: Download OpenCL-ICD-Loader
run: |
cd ..
git clone https://github.com/KhronosGroup/OpenCL-ICD-Loader
pushd OpenCL-ICD-Loader
git checkout ${{ env.opencl-icd-loader-git-tag }}
popd
pushd OpenCL-ICD-Loader/inc
git clone https://github.com/KhronosGroup/OpenCL-Headers
pushd OpenCL-Headers
git checkout ${{ env.opencl-headers-git-tag }}
popd
cp -r OpenCL-Headers/CL ./
popd
shell: bash

- name: Build OpenCL-ICD-Loader
run: |
cd ..
mkdir OpenCL-ICD-Loader-build
cd OpenCL-ICD-Loader-build
call "C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Auxiliary\Build\vcvars64.bat"
cmake -DCMAKE_C_COMPILER:FILEPATH="${{ matrix.c-compiler }}" -DBUILD_SHARED_LIBS:BOOL=ON -DCMAKE_CXX_COMPILER="${{ matrix.cxx-compiler }}" -DCMAKE_BUILD_TYPE:STRING=${{ matrix.cmake-build-type }} -DBUILD_TESTING:BOOL=OFF -GNinja ../OpenCL-ICD-Loader
cmake --build . --target install
shell: cmd

- name: 'Build 🐍 Python 📦 package'
run: |
cd ../../im
call "C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Auxiliary\Build\vcvars64.bat"
set PATH=C:\P\grep;%PATH%
set CC=cl.exe
set CXX=cl.exe
C:\Python3${{ matrix.python-version-minor }}-x64\python.exe C:\P\IPP\scripts\windows_build_module_wheels.py --py-envs "3${{ matrix.python-version-minor }}-x64" --lib-paths "${{ github.workspace}}/../OpenCL-ICD-Loader-build/${{ matrix.cmake-build-type }};${{ github.workspace}}/../OpenCL-ICD-Loader-build/" --no-cleanup -- "-DOpenCL_INCLUDE_DIR:PATH=${{ github.workspace}}/../OpenCL-ICD-Loader/inc" "-DOpenCL_LIBRARY:FILEPATH=${{ github.workspace}}/../OpenCL-ICD-Loader-build/OpenCL.lib"
shell: cmd

- name: Publish Python package as GitHub Artifact
uses: actions/upload-artifact@v1
with:
name: WindowsWheel3.${{ matrix.python-version-minor }}
path: ../../im/dist

build-linux-opencl-python-packages:
runs-on: ubuntu-20.04
strategy:
max-parallel: 2
matrix:
python-version: [37, 38, 39, 310]

steps:
- uses: actions/checkout@v2

- name: 'Free up disk space'
run: |
# Workaround for https://github.com/actions/virtual-environments/issues/709
df -h
sudo apt-get clean
sudo rm -rf "/usr/local/share/boost"
sudo rm -rf "$AGENT_TOOLSDIRECTORY"
df -h

- name: 'Build 🐍 Python 📦 package'
run: |
export ITK_PACKAGE_VERSION=${{ env.itk-wheel-tag }}
export TARBALL_SPECIALIZATION="-manylinux_2_28"
./wrapping/dockcross-manylinux-download-cache.sh
./wrapping/dockcross-manylinux-build-module-wheels-opencl.sh cp${{ matrix.python-version }}

- name: Publish Python package as GitHub Artifact
uses: actions/upload-artifact@v1
with:
name: LinuxOpenCLWheel${{ matrix.python-version }}
path: dist

build-macos-opencl-python-packages:
runs-on: macos-11
strategy:
max-parallel: 2

steps:
- uses: actions/checkout@v2

- name: 'Specific XCode version'
run: |
sudo xcode-select -s "/Applications/Xcode_13.2.1.app"

- name: Get specific version of CMake, Ninja
uses: lukka/get-cmake@v3.21.2

- name: 'Fetch build script'
run: |
curl -L https://raw.githubusercontent.com/InsightSoftwareConsortium/ITKPythonPackage/master/scripts/macpython-download-cache-and-build-module-wheels.sh -O
chmod u+x macpython-download-cache-and-build-module-wheels.sh

- name: 'Build 🐍 Python 📦 package'
run: |
export ITK_PACKAGE_VERSION=${{ env.itk-wheel-tag }}
export MACOSX_DEPLOYMENT_TARGET=10.11
./macpython-download-cache-and-build-module-wheels.sh

- name: Publish Python package as GitHub Artifact
uses: actions/upload-artifact@v1
with:
name: MacOSOpenCLWheels
path: dist

publish-python-packages-to-pypi:
needs:
- build-linux-opencl-python-packages
- build-macos-opencl-python-packages
- build-windows-python-packages
runs-on: ubuntu-20.04

steps:
- name: Download Python Packages
uses: actions/download-artifact@v2

- name: Prepare packages for upload
run: |
ls -R
for d in */; do
mv ${d}/*.whl .
done
mkdir dist
mv *.whl dist/
ls dist
cxx-build-workflow:
uses: InsightSoftwareConsortium/ITKRemoteModuleBuildTestPackageAction/.github/workflows/build-test-cxx.yml@v5.4.0

- name: Publish 🐍 Python 📦 package to PyPI
if: github.event_name == 'push' && startsWith(github.event.ref, 'refs/tags')
uses: pypa/gh-action-pypi-publish@v1.5.1
with:
user: __token__
password: ${{ secrets.pypi_password }}
python-build-workflow:
uses: InsightSoftwareConsortium/ITKRemoteModuleBuildTestPackageAction/.github/workflows/build-test-package-python.yml@v5.4.0
secrets:
pypi_password: ${{ secrets.pypi_password }}
Loading