Use python binary instead of libpython when it's a PIE

[tkf]

Presumably, this would let us share precompilation cache between PyCall and PyJulia in more situations [1]: conda-forge/python-feedstock#222 (comment)

[1] Not all situation. For example, non-PIE statically linked python still won't work.

[stevengj]

You can't dlopen an executable, can you? Oh, I see that this is indeed possible for PIE executables on some platforms. Not sure if this is safe with python but I I guess we could give it a try.

Does ccall(("PyFoo", "/path/to/python"), ...) work directly, or do we need to explicitly call dlopen and work with the library handle?

[isuruf]

Does ccall(("PyFoo", "/path/to/python"), ...) work directly, or do we need to explicitly call dlopen and work with the library handle?

Yes, it works. dlopen can't be used directly because julia appends a .so to the path.

[tkf]

Hmm... So Py_GetVersion works but Py_InitializeEx fails here https://github.com/python/cpython/blob/v3.7.1/Python/sysmodule.c#L2292

Any guess why?

(@isuruf BTW, it looks like I don't need .so if I pass absolute path to dlopen. Checked with Julia 1.0.1 and master.)

I created a conda environment with create --prefix py defaults::python which installs python 3.7.1-h0371630_3 and then run:

julia> using Libdl

julia> const libpath = abspath("py/bin/python")
"/home/takafumi/.julia/dev/_wt/PyCall/pie/py/bin/python"

julia> const wPYTHONHOME = Base.cconvert(Cwstring, string(abspath("py"), ':', abspath("py")));

julia> const wpyprogramname = Base.cconvert(Cwstring, libpath);

julia> h = Libdl.dlopen(libpath, Libdl.RTLD_LAZY|Libdl.RTLD_DEEPBIND|Libdl.RTLD_GLOBAL)
Ptr{Nothing} @0x0000000002b6ab10

julia> unsafe_string(ccall((:Py_GetVersion, libpath), Ptr{UInt8}, ()))
"3.7.1 (default, Oct 23 2018, 19:19:42) \n[GCC 7.3.0]"

julia> ccall((:Py_SetProgramName, libpath), Cvoid, (Ptr{Cwchar_t},), wpyprogramname)

julia> ccall((:Py_SetPythonHome, libpath), Cvoid, (Ptr{Cwchar_t},), wPYTHONHOME)

julia> ccall((:Py_InitializeEx, libpath), Cvoid, (Cint,), 0)

signal (11): Segmentation fault
in expression starting at no file:0
fileno_unlocked at /usr/lib/libc.so.6 (unknown line)
_PySys_BeginInit at /tmp/build/80754af9/python_1540319607830/work/Python/sysmodule.c:2292
_Py_InitializeCore_impl at /tmp/build/80754af9/python_1540319607830/work/Python/pylifecycle.c:753
_Py_InitializeCore at /tmp/build/80754af9/python_1540319607830/work/Python/pylifecycle.c:859
_Py_InitializeFromConfig at /tmp/build/80754af9/python_1540319607830/work/Python/pylifecycle.c:1002
Py_InitializeEx at /tmp/build/80754af9/python_1540319607830/work/Python/pylifecycle.c:1034
top-level scope at ./none:0
jl_fptr_trampoline at /buildworker/worker/package_linux64/build/src/gf.c:1831
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:807
jl_toplevel_eval_in at /buildworker/worker/package_linux64/build/src/builtins.c:622
eval at ./boot.jl:319
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2184
eval_user_input at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/REPL/src/REPL.jl:85
macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/REPL/src/REPL.jl:117 [inlined]
#28 at ./task.jl:259
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2184
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1537 [inlined]
start_task at /buildworker/worker/package_linux64/build/src/task.c:268
unknown function (ip: 0xffffffffffffffff)
Allocations: 882346 (Pool: 882247; Big: 99); GC: 0

[isuruf]

Can you check what Py_IsInitialized gives you?

[tkf]

OTOH Py_InitializeEx works with /usr/bin/python in Arch Linux.

[tkf]

@isuruf Py_IsInitialized gives me 0

julia> using Libdl

julia> pyhome = abspath("py")
"/home/takafumi/.julia/dev/_wt/PyCall/pie/py"

julia> const libpath = "$pyhome/bin/python"
"/home/takafumi/.julia/dev/_wt/PyCall/pie/py/bin/python"

julia> const wPYTHONHOME = Base.cconvert(Cwstring, string(pyhome, ':', pyhome));

julia> const wpyprogramname = Base.cconvert(Cwstring, libpath);

julia> h = Libdl.dlopen(libpath, Libdl.RTLD_LAZY|Libdl.RTLD_DEEPBIND|Libdl.RTLD_GLOBAL)
Ptr{Nothing} @0x00000000026dabc0

julia> unsafe_string(ccall((:Py_GetVersion, libpath), Ptr{UInt8}, ()))
"3.7.1 (default, Oct 23 2018, 19:19:42) \n[GCC 7.3.0]"

julia> ccall((:Py_IsInitialized, libpath), Cint, ())
0

julia> ccall((:Py_SetProgramName, libpath), Cvoid, (Ptr{Cwchar_t},), wpyprogramname)

julia> ccall((:Py_SetPythonHome, libpath), Cvoid, (Ptr{Cwchar_t},), wPYTHONHOME)

julia> ccall((:Py_InitializeEx, libpath), Cvoid, (Cint,), 0)

signal (11): Segmentation fault
in expression starting at no file:0
fileno_unlocked at /usr/lib/libc.so.6 (unknown line)
_PySys_BeginInit at /tmp/build/80754af9/python_1540319607830/work/Python/sysmodule.c:2292

[tkf]

With /usr/bin/python:

julia> using Libdl

julia> pyhome = "/usr"
"/usr"

julia> const libpath = "$pyhome/bin/python"
"/usr/bin/python"

julia> const wPYTHONHOME = Base.cconvert(Cwstring, string(pyhome, ':', pyhome));

julia> const wpyprogramname = Base.cconvert(Cwstring, libpath);

julia> h = Libdl.dlopen(libpath, Libdl.RTLD_LAZY|Libdl.RTLD_DEEPBIND|Libdl.RTLD_GLOBAL)
Ptr{Nothing} @0x00000000024d8410

julia> unsafe_string(ccall((:Py_GetVersion, libpath), Ptr{UInt8}, ()))
"3.7.0 (default, Jul 15 2018, 10:44:58) \n[GCC 8.1.1 20180531]"

julia> ccall((:Py_IsInitialized, libpath), Cint, ())
0

julia> ccall((:Py_SetProgramName, libpath), Cvoid, (Ptr{Cwchar_t},), wpyprogramname)

julia> ccall((:Py_SetPythonHome, libpath), Cvoid, (Ptr{Cwchar_t},), wPYTHONHOME)

julia> ccall((:Py_InitializeEx, libpath), Cvoid, (Cint,), 0)

julia> ccall((:Py_IsInitialized, libpath), Cint, ())
1

[isuruf]

Okay. Looks like this idea won't work

[isuruf]

julia> using Libdl

julia> const libpython = "/usr/bin/python"
"/usr/bin/python"

julia> const libpython_handle = Libdl.dlopen(libpython, Libdl.RTLD_LOCAL)
Ptr{Nothing} @0x000000000118c890

julia> using PyCall

julia> include("/home/isuru/.julia/packages/PyCall/0jMpb/test/runtests.jl")
┌ Info: Python version 2.7.15-rc1 from /usr/bin/python, PYTHONHOME=/usr:/usr
│ ENV[PYTHONPATH]=
│ ENV[PYTHONHOME]=
└ ENV[PYTHONEXECUTABLE]=
Test Summary: | Pass  Total
PyCall        |  432    432
Test Summary: | Pass  Total
pydef         |    6      6
Test Summary: | Pass  Total
callback      |    3      3
Test Summary: | Pass  Total
pycall!       |   16     16
Test.DefaultTestSet("pycall!", Any[DefaultTestSet("basics", Any[], 8, false), DefaultTestSet("kwargs", Any[], 8, false)], 0, false)

julia> using Pkg

julia> Pkg.dir("PyCall")
┌ Warning: `Pkg.dir(pkgname, paths...)` is deprecated; instead, do `import PyCall; joinpath(dirname(pathof(PyCall)), "..", paths...)`.
└ @ Pkg.API /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/API.jl:454
"/home/isuru/.julia/packages/PyCall/0jMpb/"

$ cat ~/.julia/packages/PyCall/0jMpb/deps/deps.jl 
const python = "/usr/bin/python"
const pyprogramname = "/usr/bin/python"
const wpyprogramname = Base.cconvert(Cwstring, "/usr/bin/python")
const pyversion_build = v"2.7.15-rc1"
const PYTHONHOME = "/usr:/usr"
const wPYTHONHOME = Base.cconvert(Cwstring, "/usr:/usr")

const libpython = "/usr/bin/python"

"True if we are using the Python distribution in the Conda package."
const conda = false

[tkf]

Cool! But do you need to pass RTLD_LOCAL? IIUC it would make PyCall incompatible with wheels, right?

There is one situation where extensions that are linked in this way can fail to work: if a host program (e.g., apache2) uses dlopen() to load a module (e.g., mod_wsgi) that embeds the CPython interpreter, and the host program does not pass the RTLD_GLOBAL flag to dlopen(), then the embedded CPython will be unable to load any extension modules that do not themselves link explicitly to libpythonX.Y.so.1. Fortunately, apache2 does set the RTLD_GLOBAL flag, as do all the other programs that embed-CPython-via-a-dlopened-plugin that we could locate, so this does not seem to be a serious problem in practice. The incompatibility with Debian/Ubuntu is more of an issue than the theoretical incompatibility with a rather obscure corner case.
--- https://www.python.org/dev/peps/pep-0513/#libpythonx-y-so-1

[tkf]

What is the Linux distribution you are using?

[isuruf]

Ubuntu 18.04

[isuruf]

Libdl.RTLD_LAZY|Libdl.RTLD_GLOBAL works, but Libdl.RTLD_DEEPBIND doesn't

[tkf]

@isuruf Thanks! Yeah I just figured that out too :)

[tkf]

Looks like RTLD_DEEPBIND was added due to #189.