Skip to content

PEP 776: Remove section on synchronous loading limit #4333

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
hugovk merged 1 commit into from
Mar 31, 2025
Merged

PEP 776: Remove section on synchronous loading limit #4333

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 6 additions & 34 deletions peps/pep-0776.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -347,49 +347,21 @@ In the long term, we hope to implement stack switching ``stdin`` devices, but
that is out of scope for this PEP.


Dynamic Libraries
-----------------

Main Thread Synchronous Loading Limit
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In the main browser thread, a dynamic library can only be loaded synchronously
if it is at most 4 kilobytes. This excludes most nontrivial dynamic libraries.
This limit is not present in Node.js and can be avoided by using a web worker. If
stack switching is available, then it is possible to make ``dlopen()`` stack
switch in order to instantiate a dynamic library synchronously.

To avoid the synchronous loading limit, Pyodide currently preloads all dynamic
libraries present in a wheel when installing the wheel (or on startup). This is
a significant disadvantage with packages like SciPy that include a very large
number of shared libraries that are expected to be only loaded on demand.
Pyodide will implement a solution based on stack switching as it becomes more
widely available in runtimes.

Emscripten Python only loads extension module dynamic libraries when they are
imported. This approach is simpler and more efficient when it works. The web
example runs in a web worker and the cli runner runs in Node so neither of these
have the synchronous loading limit. We will continue with this approach in
Emscripten Python.

In the long run, we hope to implement a stack switching ``dlopen``, but that is
out of scope for this PEP.

Missing RPATH Support
~~~~~~~~~~~~~~~~~~~~~
---------------------

Another important limitation of the Emscripten dynamic loader is that it does
not currently have RPATH support. Pyodide's present workaround is as follows:
An important limitation of the Emscripten dynamic loader is that it does not
currently have RPATH support. Pyodide's present workaround is as follows:
``auditwheel-emscripten`` places shared library dependencies that are vendored
into a package in a ``${package}.libs`` folder, following auditwheel's
convention. Pyodide patches the dynamic loader to treat this ``${package}.libs``
folder as if it were on the RPATH of all of the dynamic libraries in the wheel.

In Emscripten 4.0.5, we have updated the shared object file format, ``wasm-ld``
and ``emcc`` to accept an ``-rpath`` flag. We are still working on updating the
dynamic loader to respect the rpath, but we expect this will be finished in the
next Emscripten release. Pyodide will then switch to using the RPATH and drop
the patch on the dynamic loader.
dynamic loader to respect the rpath, but we expect this will be finished soon.
Pyodide will then switch to using the RPATH and drop the patch on the dynamic
loader.

Emscripten Python currently uses the unpatched dynamic loader and so cannot load
extension modules that depend on vendored dynamic libraries via DT_NEEDED.
Expand Down