Python based build scripts #299
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Content from docs/Quick_start_guide.rst was moved to ITK respository under Documentation/docs/learn/python_quick_start.md
- Addresses issue #282 by silencing non-critical warnings that obscure error identification during developer reviews.
This is the initial refactoring of bash script to python to support windows builds consistently.
```bash black $(git ls-files "*.py") pyupgrade --py39-plus $(git ls-files "*.py") flynt $(git ls-files "*.py") black $(git ls-files "*.py") ```
Remove unused imports
venv is a common name for the directory or the current environment. Use virtualenv_exe to more clearly indicate the program used to create the environment.
More clearly differentiate between the python packaging dir and the resulting checked out ITK source code
Linux, Macos, and Windows have similar arguments for configuring each phase of the builds. This is an initial stage for drying (donot repeat yourself) out the code. Preparing to re-use common implementations for all 3 platforms.
Give a name to the location where files are are generated and built.
Refactor Windows build scripts to use `dotenv` and update f-string path handling for consistency"
Single wheel was hard-coded as disabled in mac and linux. It was defaulted to false and un-modified since 2017. Removing makes windows match mac and linux.
…y, and consistency Refactored build scripts to replace `os` and `shutil` with `pathlib`. Improved path handling with clean, consistent usage of `Path`. Encapsulated repetitive logic and introduced utility functions like `_which` and `_remove_tree`. Enhanced readability, modularity, and maintainability while maintaining previous functionality.
…e consistency Simplified `build_wheels` method by removing `cleanup` argument and improving default handling for `cmake_options`. Standardized `itk_source_dir` variable naming and removed unused `env_file` reference. Adjusted type handling for path checks across scripts.
…ipts Refactored all instances of `check_call` to use the new `echo_check_call` utility function for improved command visibility and debugging. Updated imports across scripts and added the implementation of `echo_check_call` in `wheel_builder_utils`.
Refactored `CMakeLists.txt` by extracting repeated logic into new modular scripts: `ITKPythonPackage_common.cmake`, `ITKPythonPackage_SuperBuild.cmake`, and `ITKPythonPackage_BuildWheels.cmake`. Consolidated logic and improved readability while maintaining existing functionality. Breaking the very large CMakeLists.txt file into smaller files makes deciphering the different build phases easier.
- Moved `detect_platform` and `which_required` to `wheel_builder_utils.py` for reuse. - Updated imports in `generate_build_environment.py` to reference the shared utility functions.
Pixi is now used to install relevant versions of python needed for making wheels.
Default to /work on linux
This script automates paths and configurations for building Python wheels, including validations and environment-specific builds.
Member
Author
|
@thewtex This is what I have been working on as I try to understand the process for building and maintaining python packages. My goal is to be able to build private Python wheels that are constructed ONLY FOR MY MACHINE with "-mtune=native -march=native" from the HEAD of a private ITK branch. The refactorings are made with the hope that the eventually inside of ITK we can do the following: I am also trying to be backwards compatible for supporting external module builds. I have most of the core components working, but have been struggling to ensure backwards compatibility because there are so many intermediate layers that depend on implicit values (i.e. matching tags an release names across different repos). |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
TODO FIX DOCUMENTATION: https://itkpythonpackage.readthedocs.io/en/latest/Build_ITK_Python_packages.html
How to use the tarball/zipfile cache from ITKPythonBuilds needed for dockcross-manylinux-download-cache.sh? What needs to be included?
Remove SKBUILD cmake items that are unused (perhaps this needs to be fixed? unknown use case for this)
The entire process for creating python modules should be driven by a consistent selection of build options. The previous Python package generation infrastructure had many default values configured at various locations across several scripts. The values used to make the Python packages required that default values (i.e., the ITK TAG) was hard-coded in several places so that matching values existed during the various phases of processing (in different container/host environments and in different scripts.
This PR employs an overall strategy where all the candidate configuration choices are defaulted to previous behavior.
The virtual environments use PIXI to consolidate and manage the environments needed to create the packages consistently across all the platforms with similar tooling.
A secondary strategy is to prefer early failure to continued processing with defaults. Each stage of the process should be explicitly and completely defined.
ITKPythonPackage/CMakeLists.txt is called from an environment where the following variables exist:
ITK_GIT_TAG = ONLY used to represent the version of ITK to checkout. "HEAD" could be used THIS CONCEPT IS CONFOUNDED WITH ITK_PACAKAGE_VERSION (which should be pep-440 compliant, but v6.0b01 is not conformant). ITK_PACKAGE_VERSION (set from ITKRemoteModuleBuildTestPackageAction:itk-wheel-tag) ITKPYTHONPACKAGE_TAG (set from ITKRemoteModuleBuildTestPackageAction:itk-python-package-tag) ITKPYTHONPACKAGE_ORG (set from ITKRemoteModuleBuildTestPackageAction:itk-python-package-org) ITK_MODULE_PREQ (set ITKRemoteModuleBuildTestPackageAction:itk-module-deps)Some useful command lines at the heart of building packages
SDKROOT CXX CC
cd Dashboard/src/ITKPythonPackage
bash scripts/dockcross-manylinux-build-wheels.sh
Dashboard/src/ITKPythonPackage
cd Dashboard/src/ITKPythonPackage
TODO: setup CXX CC to be the pixi compilers for XCode 16 rather than XCode 17.
bash scripts/macpython-build-wheels.sh
python3 scripts/build_wheels.py --py-envs 3.13 --build-dir-root /Users/johnsonhj/Dashboard/src/IP-bld
/Users/johnsonhj/Dashboard/src/ITKPythonPackage/scripts/build_wheels.py --py-envs 3.9 3.10 3.11 --build-dir-root /Users/johnsonhj/Dashboard/src/ITKPP-bld --build-itk-tarball-cache --itk-git-tag v6.0b02
PSUEDO_CODE ./scripts/dockcross-manylinux-build-wheels.sh - Set environmental variables from build/package.env - run container that generates dockcross runner script build/runner_dockcross-*-*.sh - run build/runner_dockcross-*-*.sh from the directory that is to be manipulated in the cross env (i.e. source directory) run container where HOST_PWD is mounted inside the running container at /work HOST ~/.ssh is mounted to the inside running container at ~/.ssh HOST UserID & GroupID's are mapped inside the running container (i.e., files generated will have host users' permissions) FINAL_ARGS (usually other for alternate mount points) <usually a script to be run inside the running container to do the payload of work>docker run -i -t -v
pwd:/io dockerio.io/pypa/manylinux_2_28_x86_64:20250913-6ea98badockcross/manylinux_2_28-x64:20250913-6ea98ba
docker run -ti --name dockcross_22377 -v /home/johnsonhj/Dashboard/src/IPPP:/work -v /home/johnsonhj/.ssh:/home/johnsonhj/.ssh -e BUILDER_UID=1218096 -e BUILDER_GID=100 -e BUILDER_USER=johnsonhj -e BUILDER_GROUP=users -v /home/johnsonhj/Dashboard/src/IPPP/dist:/work/dist/ -v/home/johnsonhj/Dashboard/src/ITK:/work/ITK-source/ITK --env-file /home/johnsonhj/Dashboard/src/IPPP/build/package.env dockcross/manylinux_2_28-x64:20250913-6ea98ba /bin/bash
- [ ] dockcross-manylinux-download-cache.sh is curl downloaded from @ ITKPYTHONPACKAGE_TAG
- [ ] dockcross-manylinux-build-wheels.sh
export ITK_GIT_TAG=main; export ITKPYTHONPACKAGE_ORG=InsightSoftwareConsortium; export