From e0e23b14b3b16f897c2f2441cac96581d6dd3a01 Mon Sep 17 00:00:00 2001 From: Savannah Ostrowski Date: Mon, 27 Jan 2025 02:36:01 +0000 Subject: [PATCH 01/14] Add Removing the LLVM requirement for JIT builds PEP --- peps/pep-NNNN.rst | 247 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 247 insertions(+) create mode 100644 peps/pep-NNNN.rst diff --git a/peps/pep-NNNN.rst b/peps/pep-NNNN.rst new file mode 100644 index 00000000000..087371218c0 --- /dev/null +++ b/peps/pep-NNNN.rst @@ -0,0 +1,247 @@ +PEP: NNNN +Title: Removing the LLVM requirement for JIT builds +Author: Savannah Ostrowski +Discussions-To: TODO +Status: Draft +Type: Standards Track +Created: 26-Jan-2025 +Python-Version: 3.14 + +Abstract +======== + +Since Python 3.13, CPython has been able to be configured and built with an +experimental just-in-time (JIT) compiler via the ``--enable-experimental-jit``` +flag on Linux and Mac and ``--experimental-jit`` on Windows. To build CPython with +the JIT enabled, users are required to have LLVM installed on their machine +(initially, with LLVM 16 but more recently, with LLVM 19). LLVM is responsible +for generating stencils that are essential to our copy-and-patch JIT (see :pep:`744`). + +This PEP proposes removing the LLVM build-time dependency for JIT-enabled builds +by hosting the generated stencils in the CPython repository to simplify the +contributor experience and address concerns raised at the Python Core Developer +Sprint in September 2024. That said, there is a clear tradeoff to consider, as +improved developer experience does come at the cost of increased repository +size. + +It is important to note that this PEP is not a proposal to accept or reject the +JIT itself but rather to determine whether the build-time dependency on LLVM is +acceptable for JIT builds moving forward. If this PEP is rejected, we will +proceed with the status quo, retaining the LLVM build-time requirement. While +this dependency has served the JIT development process effectively thus far, it +introduces setup complexity and additional challenges that this PEP seeks to +alleviate. + +Motivation +======== + +At the Python Core Developer Sprint that took place in September 2024, there was +discussion about the next steps for the JIT (a related discussion also took +place `on GitHub `__). As part +of that discussion, there was also a clear appetite for removing the LLVM +requirement for JIT builds in preparation for shipping the JIT off by default in +3.14. The consensus at the sprint was that it would be sufficient to provide +pre-generated stencils for non-debug builds for Tier 1 platforms and that +checking these files into the CPython repo would be adequate for the limited +number of platforms (though more options have been explored; see `Rejected +Ideas`__). + +Currently, building CPython with `the JIT requires LLVM +` as a +build-time dependency. Despite not being exposed to end users, this dependency +is suboptimal. Requiring LLVM adds a setup burden for developers and those who +wish to build CPython with the JIT enabled. Depending on the operating system, +the version of LLVM shipped with the OS may differ from that required by our JIT +builds, which introduces additional complexity to troubleshoot and resolve. With +few core developers currently contributing to and maintaining the JIT, we also +want to make sure that the friction to work on JIT-related code is minimized as +much as possible. + +With the proposed approach, hosting pre-compiled stencils for supported +architectures (listed below in `Specification`__) can be generated in advance, +stored in a central location, and automatically used during builds. This +approach ensures reproducible builds, making the JIT a more stable and +sustainable part of CPython’s future. + +Rationale +======== + +This PEP proposes checking JIT stencils directly into the CPython repo as the +best path forward for eliminating our build-time dependency on LLVM. + +This approach: - Provides the best end-to-end experience for those looking to +build CPython with the JIT - Lessens the barrier to entry for those looking to +contribute to the JIT - Ensures builds remain reproducible and consistent across +platforms without relying on external infrastructure or download mechanisms - +Eliminates variability introduced by network conditions or potential +discrepancies between hosted files and the CPython repository state, and - +Subjects stencils to the same review processes we have for all other JIT-related +code + +However, this approach result in a slight increase in overall +repository size. Comparing repo growth on commits over the past 90 days, the +difference between the actual commits and the same commits with stencils added +amounts to a difference of 0.03MB per stencil file. This is a small increase in +the context of the overall repository size, which has grown by 2.55MB in the +same time period. For six stencil files, this amounts to an upper bound of 0.18MB. + +These stencils could become larger in the future with changes to register +allocation, which would introduce 5-6 variants per instruction in each stencil +file (5-6x larger). However, if we ended up going this route, there are +additional modifications we could make to stencil files that could help +counteract this size increase (e.g., stripping comments, minimizing the +stencils). + +Specification +======== + +This specification outlines the proposed changes to remove the build-time +dependency on LLVM and the contributor experience if this PEP is accepted. + +Repository changes +------------------ + +The CPython repository would now host the pre-compiled JIT stencils in a new +subdirectory in ``Tools/jit`` called ``stencils/``. At present, the JIT is tested +and built for six platforms, so to start, we’d check in six stencil files. In +the future, we may check in additional stencil files if support for additional +platforms is desired or relevant. + +.. code-block:: bash + + cpython/ + Tools/ + jit/ + stencils/ + aarch64-apple-darwin.h + aarch64-unknown-linux-gnu.h + i686-pc-windows-msvc.h + x86_64-apple-darwin.h + x86_64-pc-windows-msvc.h + x86_64-pc-linux-gnu.h + +Workflow +-------- + +To cover the workflow changes, this section can be divided into two parts - +building CPython with the JIT enabled and working on the JIT’s implementation. + +Building CPython with the JIT +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Precompiled JIT stencil files will be stored in the ``Tools/jit/stencils`` +directory, with each file name corresponding to its target triple as outlined +above. At build time, we determine whether to use the checked in stencils or to +generate a new stencil for the user’s platform. Specifically, for contributors +with LLVM installed, the ``build.py`` script in ``Tools/jit/stencils`` will allow +them to regenerate the stencil for their platform. Those without LLVM can rely +on the precompiled stencil files directly from the repository. + +Working on the JIT’s implementation (or touching JIT files) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +In CI, stencil files will be automatically validated and updated when changes +are made to JIT-related files. When a pull request is opened that touches these +files, the ``jit.yml`` workflow, which builds and tests our builds, will run as +usual. + +However, as part of this, we will introduce a new step that diffs the current +stencils in the repo against those generated in CI. If there is a diff for a +platform’s stencil file, a patch file for the updated stencil is generated and +the step will fail. Each patches is uploaded to GitHub Actions. After CI is +finished running across all platforms, the patches are aggregated into a single +patch file for convenience. You can download this aggregated patch, apply it +locally, and commit the updated stencils back to your branch. Then, the +subsequent CI run will pass. + +Reference Implementation +======================== + +Key parts of the reference implementation include: + +- | CI |_: The CI workflow responsible for building, testing, and now generating +stencil patches. + +- | jit_stencils |_: The directory where stencils are stored. + +- | targets |_: The code to compile and parse the templates at build time. + +.. | CI | replace:: ``.github/workflows/jit.yml`` +.. _CI: https://github.com/python/cpython/blob/main/.github/workflows/jit.yml + +.. | jit_stencils | replace:: ``Tools/jit/stencils`` +.. _jit_stencils: https://github.com/python/cpython/blob/main/Tools/jit/stencils + +.. | targets | replace:: ``Tools/jit/_targets`` +.. _targets: https://github.com/python/cpython/blob/main/Tools/jit/_targets.py + +Ignoring the stencils themselves and any necessary JIT README changes, the +changes to the source code to support reproducible stencil generation and +hosting are minimal (around 150 lines of changes). + +Rejected Ideas +============== + +Several alternative approaches were considered as part of the research and +exploration for this PEP. However, the ideas below either involve +infrastructural cost, maintenance burden, or a worse overall developer +experience. + +Using Git submodules +-------------------- + +Git submodules are a poor developer experience for hosting stencils because they +create a different kind of undesirable friction. For instance, any +updates to the JIT would necessitate regenerating the stencils and committing +them to a separate repository. This introduces a convoluted process: you must +update the stencils in the submodule repository, commit those changes, and then +update the submodule reference in the main CPython repository. This disconnect +adds unnecessary complexity and overhead, making the process brittle and +error-prone for contributors and maintainers. + +Using Git subtrees +------------------ + +When using subtrees, the embedded repository becomes part of the main +repository, similar to what’s being proposed in this PEP. However, subtrees +require additional tooling and steps for maintenance, which adds unnecessary +complexity to workflows. + +Hosting in a separate repository +------------------------------- +While splitting JIT stencils into a separate repository avoids the storage +overhead associated with hosting the stencils, it adds complexity to the build +process. Additional tooling would be required to fetch the stencils and +potentially create additional and unnecessary failure points in the workflow. +This separation also makes it harder to ensure consistency between the stencils +and the CPython source tree, as updates must be coordinated across the +repositories. Finally, this approach introduces an attack vector, as external +repositories are less integrated with our workflows, making provenance and +integrity harder to guarantee. + +Hosting in cloud storage +------------------------ + +Hosting stencils in cloud storage like S3 buckets or GitHub raw storage +introduces external dependencies, potentially complicating offline development +workflows. Also, depending on the provider, this type of hosting comes with +additional cost, which we’d like to avoid. + +Using Git LFS +-------------- + +Git LFS adds tool dependency for contributors, complicating the development +workflow, especially for those who may not already use Git LFS. Git LFS does not +work well with offline workflows since files managed by LFS require an internet +connection to fetch when checking out specific commits, which is disruptive for +even basic Git workflows. + +Maintain the status quo with LLVM as a build-time dependency +------------------------------------------------------------ + +Retaining LLVM as a build-time dependency upholds the existing barriers to +adoption and contribution. Ultimately, this option fails to address the core +challenges of accessibility and simplicity, and fails to eliminate the +dependency which was deemed undesirable at the Python Core Developer Sprint in +the fall (the impetus for this PEP), making it a poor long-term solution. + From ca32139340b2192c32e7df2da0be16b22d327c62 Mon Sep 17 00:00:00 2001 From: Savannah Ostrowski Date: Mon, 27 Jan 2025 02:37:07 +0000 Subject: [PATCH 02/14] Add reference implementation link --- peps/pep-NNNN.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-NNNN.rst b/peps/pep-NNNN.rst index 087371218c0..b22668d8e82 100644 --- a/peps/pep-NNNN.rst +++ b/peps/pep-NNNN.rst @@ -157,7 +157,7 @@ subsequent CI run will pass. Reference Implementation ======================== -Key parts of the reference implementation include: +Key parts of the `reference implementation `__ include: - | CI |_: The CI workflow responsible for building, testing, and now generating stencil patches. From 0b5c7883c7c6c8d8d59857f6ba3a288822b9790e Mon Sep 17 00:00:00 2001 From: Savannah Ostrowski Date: Mon, 27 Jan 2025 03:05:20 +0000 Subject: [PATCH 03/14] Fix linting --- .github/CODEOWNERS | 1 + peps/pep-0744.rst | 2 +- peps/{pep-NNNN.rst => pep-0774.rst} | 49 +++++++++++++++-------------- 3 files changed, 28 insertions(+), 24 deletions(-) rename peps/{pep-NNNN.rst => pep-0774.rst} (91%) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 6fa815b1920..6a5d31df5ab 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -650,6 +650,7 @@ peps/pep-0769.rst @facundobatista peps/pep-0770.rst @sethmlarson @brettcannon peps/pep-0771.rst @pradyunsg peps/pep-0773.rst @zooba +peps/pep-0774.rst @savannahostrowski # ... peps/pep-0777.rst @warsaw # ... diff --git a/peps/pep-0744.rst b/peps/pep-0744.rst index 9a0ff3de2e8..f03405bdde0 100644 --- a/peps/pep-0744.rst +++ b/peps/pep-0744.rst @@ -1,7 +1,7 @@ PEP: 744 Title: JIT Compilation Author: Brandt Bucher , - Savannah Ostrowski , + Savannah Ostrowski , Discussions-To: https://discuss.python.org/t/pep-744-jit-compilation/50756 Status: Draft Type: Informational diff --git a/peps/pep-NNNN.rst b/peps/pep-0774.rst similarity index 91% rename from peps/pep-NNNN.rst rename to peps/pep-0774.rst index b22668d8e82..b5d7b16ce52 100644 --- a/peps/pep-NNNN.rst +++ b/peps/pep-0774.rst @@ -1,7 +1,6 @@ -PEP: NNNN +PEP: 774 Title: Removing the LLVM requirement for JIT builds Author: Savannah Ostrowski -Discussions-To: TODO Status: Draft Type: Standards Track Created: 26-Jan-2025 @@ -11,7 +10,7 @@ Abstract ======== Since Python 3.13, CPython has been able to be configured and built with an -experimental just-in-time (JIT) compiler via the ``--enable-experimental-jit``` +experimental just-in-time (JIT) compiler via the ``--enable-experimental-jit`` flag on Linux and Mac and ``--experimental-jit`` on Windows. To build CPython with the JIT enabled, users are required to have LLVM installed on their machine (initially, with LLVM 16 but more recently, with LLVM 19). LLVM is responsible @@ -33,21 +32,21 @@ introduces setup complexity and additional challenges that this PEP seeks to alleviate. Motivation -======== +========== At the Python Core Developer Sprint that took place in September 2024, there was -discussion about the next steps for the JIT (a related discussion also took -place `on GitHub `__). As part +discussion about the next steps for the JIT - a related discussion also took +place on `GitHub `__. As part of that discussion, there was also a clear appetite for removing the LLVM requirement for JIT builds in preparation for shipping the JIT off by default in 3.14. The consensus at the sprint was that it would be sufficient to provide pre-generated stencils for non-debug builds for Tier 1 platforms and that checking these files into the CPython repo would be adequate for the limited -number of platforms (though more options have been explored; see `Rejected -Ideas`__). +number of platforms (though more options have been explored; see Rejected +Ideas). Currently, building CPython with `the JIT requires LLVM -` as a +`__ as a build-time dependency. Despite not being exposed to end users, this dependency is suboptimal. Requiring LLVM adds a setup burden for developers and those who wish to build CPython with the JIT enabled. Depending on the operating system, @@ -58,13 +57,12 @@ want to make sure that the friction to work on JIT-related code is minimized as much as possible. With the proposed approach, hosting pre-compiled stencils for supported -architectures (listed below in `Specification`__) can be generated in advance, -stored in a central location, and automatically used during builds. This -approach ensures reproducible builds, making the JIT a more stable and -sustainable part of CPython’s future. +architectures can be generated in advance, stored in a central location, and +automatically used during builds. This approach ensures reproducible builds, +making the JIT a more stable and sustainable part of CPython’s future. Rationale -======== +========= This PEP proposes checking JIT stencils directly into the CPython repo as the best path forward for eliminating our build-time dependency on LLVM. @@ -93,7 +91,7 @@ counteract this size increase (e.g., stripping comments, minimizing the stencils). Specification -======== +============= This specification outlines the proposed changes to remove the build-time dependency on LLVM and the contributor experience if this PEP is accepted. @@ -159,20 +157,19 @@ Reference Implementation Key parts of the `reference implementation `__ include: -- | CI |_: The CI workflow responsible for building, testing, and now generating -stencil patches. +- |CI|_: The CI workflow responsible for generating stencil patches. -- | jit_stencils |_: The directory where stencils are stored. +- |jit_stencils|_: The directory where stencils are stored. -- | targets |_: The code to compile and parse the templates at build time. +- |targets|_: The code to compile and parse the templates at build time. -.. | CI | replace:: ``.github/workflows/jit.yml`` +.. |CI| replace:: ``.github/workflows/jit.yml`` .. _CI: https://github.com/python/cpython/blob/main/.github/workflows/jit.yml -.. | jit_stencils | replace:: ``Tools/jit/stencils`` +.. |jit_stencils| replace:: ``Tools/jit/stencils`` .. _jit_stencils: https://github.com/python/cpython/blob/main/Tools/jit/stencils -.. | targets | replace:: ``Tools/jit/_targets`` +.. |targets| replace:: ``Tools/jit/_targets`` .. _targets: https://github.com/python/cpython/blob/main/Tools/jit/_targets.py Ignoring the stencils themselves and any necessary JIT README changes, the @@ -208,7 +205,8 @@ require additional tooling and steps for maintenance, which adds unnecessary complexity to workflows. Hosting in a separate repository -------------------------------- +-------------------------------- + While splitting JIT stencils into a separate repository avoids the storage overhead associated with hosting the stencils, it adds complexity to the build process. Additional tooling would be required to fetch the stencils and @@ -245,3 +243,8 @@ challenges of accessibility and simplicity, and fails to eliminate the dependency which was deemed undesirable at the Python Core Developer Sprint in the fall (the impetus for this PEP), making it a poor long-term solution. +Copyright +========= + +This document is placed in the public domain or under the +CC0-1.0-Universal license, whichever is more permissive. From eb5919560bec52d638b900ef5a6f0693c197c789 Mon Sep 17 00:00:00 2001 From: Savannah Ostrowski Date: Mon, 27 Jan 2025 03:09:06 +0000 Subject: [PATCH 04/14] Format bullets --- peps/pep-0774.rst | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/peps/pep-0774.rst b/peps/pep-0774.rst index b5d7b16ce52..75aa52c8c05 100644 --- a/peps/pep-0774.rst +++ b/peps/pep-0774.rst @@ -67,13 +67,16 @@ Rationale This PEP proposes checking JIT stencils directly into the CPython repo as the best path forward for eliminating our build-time dependency on LLVM. -This approach: - Provides the best end-to-end experience for those looking to -build CPython with the JIT - Lessens the barrier to entry for those looking to -contribute to the JIT - Ensures builds remain reproducible and consistent across -platforms without relying on external infrastructure or download mechanisms - -Eliminates variability introduced by network conditions or potential -discrepancies between hosted files and the CPython repository state, and - -Subjects stencils to the same review processes we have for all other JIT-related +This approach: +- Provides the best end-to-end experience for those looking to +build CPython with the JIT +- Lessens the barrier to entry for those looking to +contribute to the JIT +- Ensures builds remain reproducible and consistent across +platforms without relying on external infrastructure or download mechanisms +- Eliminates variability introduced by network conditions or potential +discrepancies between hosted files and the CPython repository state, and +- Subjects stencils to the same review processes we have for all other JIT-related code However, this approach result in a slight increase in overall From a7675d15e7aa2f22bff3c9488c7e77b92407bd1e Mon Sep 17 00:00:00 2001 From: Savannah Ostrowski Date: Mon, 27 Jan 2025 03:10:43 +0000 Subject: [PATCH 05/14] Fix typo --- peps/pep-0774.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0774.rst b/peps/pep-0774.rst index 75aa52c8c05..eebde0f2ac1 100644 --- a/peps/pep-0774.rst +++ b/peps/pep-0774.rst @@ -79,7 +79,7 @@ discrepancies between hosted files and the CPython repository state, and - Subjects stencils to the same review processes we have for all other JIT-related code -However, this approach result in a slight increase in overall +However, this approach does result in a slight increase in overall repository size. Comparing repo growth on commits over the past 90 days, the difference between the actual commits and the same commits with stencils added amounts to a difference of 0.03MB per stencil file. This is a small increase in From bcb03130ef26dc0e9bfac19a33b089bf88dd7748 Mon Sep 17 00:00:00 2001 From: Savannah Ostrowski Date: Mon, 27 Jan 2025 03:13:06 +0000 Subject: [PATCH 06/14] Fix bullets --- peps/pep-0774.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/peps/pep-0774.rst b/peps/pep-0774.rst index eebde0f2ac1..fadcfdf24fd 100644 --- a/peps/pep-0774.rst +++ b/peps/pep-0774.rst @@ -68,15 +68,15 @@ This PEP proposes checking JIT stencils directly into the CPython repo as the best path forward for eliminating our build-time dependency on LLVM. This approach: -- Provides the best end-to-end experience for those looking to +* Provides the best end-to-end experience for those looking to build CPython with the JIT -- Lessens the barrier to entry for those looking to +* Lessens the barrier to entry for those looking to contribute to the JIT -- Ensures builds remain reproducible and consistent across +* Ensures builds remain reproducible and consistent across platforms without relying on external infrastructure or download mechanisms -- Eliminates variability introduced by network conditions or potential +* Eliminates variability introduced by network conditions or potential discrepancies between hosted files and the CPython repository state, and -- Subjects stencils to the same review processes we have for all other JIT-related +* Subjects stencils to the same review processes we have for all other JIT-related code However, this approach does result in a slight increase in overall From eb03372ff33905bac7b52b47c92766ea5cb44658 Mon Sep 17 00:00:00 2001 From: Savannah Ostrowski Date: Mon, 27 Jan 2025 03:30:28 +0000 Subject: [PATCH 07/14] Update bullets --- peps/pep-0774.rst | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/peps/pep-0774.rst b/peps/pep-0774.rst index fadcfdf24fd..8f90fe1a472 100644 --- a/peps/pep-0774.rst +++ b/peps/pep-0774.rst @@ -68,16 +68,16 @@ This PEP proposes checking JIT stencils directly into the CPython repo as the best path forward for eliminating our build-time dependency on LLVM. This approach: -* Provides the best end-to-end experience for those looking to -build CPython with the JIT -* Lessens the barrier to entry for those looking to -contribute to the JIT -* Ensures builds remain reproducible and consistent across -platforms without relying on external infrastructure or download mechanisms + +* Provides the best end-to-end experience for those looking to build CPython + with the JIT +* Lessens the barrier to entry for those looking to contribute to the JIT +* Ensures builds remain reproducible and consistent across platforms without + relying on external infrastructure or download mechanisms * Eliminates variability introduced by network conditions or potential -discrepancies between hosted files and the CPython repository state, and + discrepancies between hosted files and the CPython repository state, and * Subjects stencils to the same review processes we have for all other JIT-related -code + code However, this approach does result in a slight increase in overall repository size. Comparing repo growth on commits over the past 90 days, the From 19cad2282623bc9b4967ea093602ceae05d07b34 Mon Sep 17 00:00:00 2001 From: Savannah Ostrowski Date: Mon, 27 Jan 2025 07:58:35 -0800 Subject: [PATCH 08/14] Apply suggestions from code review Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> --- peps/pep-0774.rst | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/peps/pep-0774.rst b/peps/pep-0774.rst index 8f90fe1a472..1f7cb94f12c 100644 --- a/peps/pep-0774.rst +++ b/peps/pep-0774.rst @@ -42,8 +42,8 @@ requirement for JIT builds in preparation for shipping the JIT off by default in 3.14. The consensus at the sprint was that it would be sufficient to provide pre-generated stencils for non-debug builds for Tier 1 platforms and that checking these files into the CPython repo would be adequate for the limited -number of platforms (though more options have been explored; see Rejected -Ideas). +number of platforms (though more options have been explored; see `Rejected +Ideas `__). Currently, building CPython with `the JIT requires LLVM `__ as a @@ -82,9 +82,9 @@ This approach: However, this approach does result in a slight increase in overall repository size. Comparing repo growth on commits over the past 90 days, the difference between the actual commits and the same commits with stencils added -amounts to a difference of 0.03MB per stencil file. This is a small increase in -the context of the overall repository size, which has grown by 2.55MB in the -same time period. For six stencil files, this amounts to an upper bound of 0.18MB. +amounts to a difference of 0.03 MB per stencil file. This is a small increase in +the context of the overall repository size, which has grown by 2.55 MB in the +same time period. For six stencil files, this amounts to an upper bound of 0.18 MB. These stencils could become larger in the future with changes to register allocation, which would introduce 5-6 variants per instruction in each stencil @@ -149,7 +149,7 @@ usual. However, as part of this, we will introduce a new step that diffs the current stencils in the repo against those generated in CI. If there is a diff for a platform’s stencil file, a patch file for the updated stencil is generated and -the step will fail. Each patches is uploaded to GitHub Actions. After CI is +the step will fail. Each patch is uploaded to GitHub Actions. After CI is finished running across all platforms, the patches are aggregated into a single patch file for convenience. You can download this aggregated patch, apply it locally, and commit the updated stencils back to your branch. Then, the @@ -179,6 +179,8 @@ Ignoring the stencils themselves and any necessary JIT README changes, the changes to the source code to support reproducible stencil generation and hosting are minimal (around 150 lines of changes). +.. _PEP 774 Rejected Ideas: + Rejected Ideas ============== @@ -229,9 +231,9 @@ workflows. Also, depending on the provider, this type of hosting comes with additional cost, which we’d like to avoid. Using Git LFS --------------- +------------- -Git LFS adds tool dependency for contributors, complicating the development +Git Large File Storage (LFS) adds a tool dependency for contributors, complicating the development workflow, especially for those who may not already use Git LFS. Git LFS does not work well with offline workflows since files managed by LFS require an internet connection to fetch when checking out specific commits, which is disruptive for From 3c361f354ca57ad8a63c645ec477384547a1fca8 Mon Sep 17 00:00:00 2001 From: Savannah Ostrowski Date: Mon, 27 Jan 2025 16:29:55 +0000 Subject: [PATCH 09/14] Add stencil size and LFS cost --- peps/pep-0774.rst | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/peps/pep-0774.rst b/peps/pep-0774.rst index 1f7cb94f12c..453cf5dd631 100644 --- a/peps/pep-0774.rst +++ b/peps/pep-0774.rst @@ -85,6 +85,7 @@ difference between the actual commits and the same commits with stencils added amounts to a difference of 0.03 MB per stencil file. This is a small increase in the context of the overall repository size, which has grown by 2.55 MB in the same time period. For six stencil files, this amounts to an upper bound of 0.18 MB. +The current total size of the stencil files for all six platforms is 7.2MB. These stencils could become larger in the future with changes to register allocation, which would introduce 5-6 variants per instruction in each stencil @@ -233,11 +234,14 @@ additional cost, which we’d like to avoid. Using Git LFS ------------- -Git Large File Storage (LFS) adds a tool dependency for contributors, complicating the development -workflow, especially for those who may not already use Git LFS. Git LFS does not -work well with offline workflows since files managed by LFS require an internet -connection to fetch when checking out specific commits, which is disruptive for -even basic Git workflows. +Git Large File Storage (LFS) adds a tool dependency for contributors, +complicating the development workflow, especially for those who may not already +use Git LFS. Git LFS does not work well with offline workflows since files +managed by LFS require an internet connection to fetch when checking out +specific commits, which is disruptive for even basic Git workflows. Git LFS has +some free quota but there are `additional +costs `__. +for exceeding that quota which are also undesirable. Maintain the status quo with LLVM as a build-time dependency ------------------------------------------------------------ From b1b86d7c8832812baa62fce78584e1992bc7d3e9 Mon Sep 17 00:00:00 2001 From: Savannah Ostrowski Date: Mon, 27 Jan 2025 08:55:37 -0800 Subject: [PATCH 10/14] Apply suggestions from code review Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> --- peps/pep-0774.rst | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/peps/pep-0774.rst b/peps/pep-0774.rst index 453cf5dd631..6210d7245dc 100644 --- a/peps/pep-0774.rst +++ b/peps/pep-0774.rst @@ -3,7 +3,7 @@ Title: Removing the LLVM requirement for JIT builds Author: Savannah Ostrowski Status: Draft Type: Standards Track -Created: 26-Jan-2025 +Created: 27-Jan-2025 Python-Version: 3.14 Abstract @@ -43,7 +43,7 @@ requirement for JIT builds in preparation for shipping the JIT off by default in pre-generated stencils for non-debug builds for Tier 1 platforms and that checking these files into the CPython repo would be adequate for the limited number of platforms (though more options have been explored; see `Rejected -Ideas `__). +Ideas`_). Currently, building CPython with `the JIT requires LLVM `__ as a @@ -109,7 +109,7 @@ and built for six platforms, so to start, we’d check in six stencil files. In the future, we may check in additional stencil files if support for additional platforms is desired or relevant. -.. code-block:: bash +.. code-block:: text cpython/ Tools/ @@ -125,8 +125,8 @@ platforms is desired or relevant. Workflow -------- -To cover the workflow changes, this section can be divided into two parts - -building CPython with the JIT enabled and working on the JIT’s implementation. +The workflow changes can be split into two parts, namely +building CPython with the JIT enabled and working on the JIT's implementation. Building CPython with the JIT ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -142,7 +142,7 @@ on the precompiled stencil files directly from the repository. Working on the JIT’s implementation (or touching JIT files) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -In CI, stencil files will be automatically validated and updated when changes +In continuous integration (CI), stencil files will be automatically validated and updated when changes are made to JIT-related files. When a pull request is opened that touches these files, the ``jit.yml`` workflow, which builds and tests our builds, will run as usual. @@ -180,8 +180,6 @@ Ignoring the stencils themselves and any necessary JIT README changes, the changes to the source code to support reproducible stencil generation and hosting are minimal (around 150 lines of changes). -.. _PEP 774 Rejected Ideas: - Rejected Ideas ============== From 3f4376f19780f9c9a775e2d1123d50f8c631d9fd Mon Sep 17 00:00:00 2001 From: Savannah Ostrowski Date: Mon, 27 Jan 2025 08:55:58 -0800 Subject: [PATCH 11/14] Apply suggestions from code review Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> --- peps/pep-0774.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0774.rst b/peps/pep-0774.rst index 6210d7245dc..99d8b6fcc67 100644 --- a/peps/pep-0774.rst +++ b/peps/pep-0774.rst @@ -225,7 +225,7 @@ Hosting in cloud storage ------------------------ Hosting stencils in cloud storage like S3 buckets or GitHub raw storage -introduces external dependencies, potentially complicating offline development +introduces external dependencies, complicating offline development workflows. Also, depending on the provider, this type of hosting comes with additional cost, which we’d like to avoid. From e18438e45c177283565ac1dc8c7ae2d194c1172f Mon Sep 17 00:00:00 2001 From: Savannah Ostrowski Date: Mon, 27 Jan 2025 17:04:45 +0000 Subject: [PATCH 12/14] Fix apostrophes --- peps/pep-0774.rst | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/peps/pep-0774.rst b/peps/pep-0774.rst index 453cf5dd631..887fc60ba04 100644 --- a/peps/pep-0774.rst +++ b/peps/pep-0774.rst @@ -59,7 +59,7 @@ much as possible. With the proposed approach, hosting pre-compiled stencils for supported architectures can be generated in advance, stored in a central location, and automatically used during builds. This approach ensures reproducible builds, -making the JIT a more stable and sustainable part of CPython’s future. +making the JIT a more stable and sustainable part of CPython's future. Rationale ========= @@ -105,7 +105,7 @@ Repository changes The CPython repository would now host the pre-compiled JIT stencils in a new subdirectory in ``Tools/jit`` called ``stencils/``. At present, the JIT is tested -and built for six platforms, so to start, we’d check in six stencil files. In +and built for six platforms, so to start, we'd check in six stencil files. In the future, we may check in additional stencil files if support for additional platforms is desired or relevant. @@ -126,7 +126,7 @@ Workflow -------- To cover the workflow changes, this section can be divided into two parts - -building CPython with the JIT enabled and working on the JIT’s implementation. +building CPython with the JIT enabled and working on the JIT's implementation. Building CPython with the JIT ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -134,12 +134,12 @@ Building CPython with the JIT Precompiled JIT stencil files will be stored in the ``Tools/jit/stencils`` directory, with each file name corresponding to its target triple as outlined above. At build time, we determine whether to use the checked in stencils or to -generate a new stencil for the user’s platform. Specifically, for contributors +generate a new stencil for the user's platform. Specifically, for contributors with LLVM installed, the ``build.py`` script in ``Tools/jit/stencils`` will allow them to regenerate the stencil for their platform. Those without LLVM can rely on the precompiled stencil files directly from the repository. -Working on the JIT’s implementation (or touching JIT files) +Working on the JIT's implementation (or touching JIT files) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ In CI, stencil files will be automatically validated and updated when changes @@ -149,7 +149,7 @@ usual. However, as part of this, we will introduce a new step that diffs the current stencils in the repo against those generated in CI. If there is a diff for a -platform’s stencil file, a patch file for the updated stencil is generated and +platform's stencil file, a patch file for the updated stencil is generated and the step will fail. Each patch is uploaded to GitHub Actions. After CI is finished running across all platforms, the patches are aggregated into a single patch file for convenience. You can download this aggregated patch, apply it @@ -206,7 +206,7 @@ Using Git subtrees ------------------ When using subtrees, the embedded repository becomes part of the main -repository, similar to what’s being proposed in this PEP. However, subtrees +repository, similar to what's being proposed in this PEP. However, subtrees require additional tooling and steps for maintenance, which adds unnecessary complexity to workflows. @@ -229,7 +229,7 @@ Hosting in cloud storage Hosting stencils in cloud storage like S3 buckets or GitHub raw storage introduces external dependencies, potentially complicating offline development workflows. Also, depending on the provider, this type of hosting comes with -additional cost, which we’d like to avoid. +additional cost, which we'd like to avoid. Using Git LFS ------------- From 1c646d889acc1f2f963ac9f31e4f9894f2e493ca Mon Sep 17 00:00:00 2001 From: Savannah Ostrowski Date: Mon, 27 Jan 2025 19:58:37 +0000 Subject: [PATCH 13/14] Explain stencils more clearly --- peps/pep-0774.rst | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/peps/pep-0774.rst b/peps/pep-0774.rst index 530e2b856e4..4a68bde2930 100644 --- a/peps/pep-0774.rst +++ b/peps/pep-0774.rst @@ -15,13 +15,16 @@ flag on Linux and Mac and ``--experimental-jit`` on Windows. To build CPython wi the JIT enabled, users are required to have LLVM installed on their machine (initially, with LLVM 16 but more recently, with LLVM 19). LLVM is responsible for generating stencils that are essential to our copy-and-patch JIT (see :pep:`744`). +These stencils are predefined, architecture-specific templates that are used +to generate machine code at runtime. This PEP proposes removing the LLVM build-time dependency for JIT-enabled builds -by hosting the generated stencils in the CPython repository to simplify the -contributor experience and address concerns raised at the Python Core Developer -Sprint in September 2024. That said, there is a clear tradeoff to consider, as -improved developer experience does come at the cost of increased repository -size. +by hosting the generated stencils in the CPython repository. This approach +allows us to leverage the checked-in stencils for supported platforms at build +time, simplifying the contributor experience and address concerns raised at the +Python Core Developer Sprint in September 2024. That said, there is a clear +tradeoff to consider, as improved developer experience does come at the cost of +increased repository size. It is important to note that this PEP is not a proposal to accept or reject the JIT itself but rather to determine whether the build-time dependency on LLVM is From 0cd997da400e91572cc5fc5629aa3e6726193ce7 Mon Sep 17 00:00:00 2001 From: Savannah Ostrowski Date: Mon, 27 Jan 2025 12:29:08 -0800 Subject: [PATCH 14/14] Update pep-0774.rst Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> --- peps/pep-0774.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0774.rst b/peps/pep-0774.rst index 4a68bde2930..e6d171d1749 100644 --- a/peps/pep-0774.rst +++ b/peps/pep-0774.rst @@ -88,7 +88,7 @@ difference between the actual commits and the same commits with stencils added amounts to a difference of 0.03 MB per stencil file. This is a small increase in the context of the overall repository size, which has grown by 2.55 MB in the same time period. For six stencil files, this amounts to an upper bound of 0.18 MB. -The current total size of the stencil files for all six platforms is 7.2MB. +The current total size of the stencil files for all six platforms is 7.2 MB. These stencils could become larger in the future with changes to register allocation, which would introduce 5-6 variants per instruction in each stencil