oneAPI-aware MPI

[michel2323]

I can only test this on Aurora at ANL. I could provide an MPICH_jll with support. How should I proceed?

[michel2323]

I have no idea what is failing here. Anyhow, before I add MPI.has_oneapi(), I want to leave all this to the user. It works on Aurora, so there's that. Can we merge this?

[lcw]

The failure looks to be unrelated. Is is possible for you to test your changes by adding a buildkite pipeline? For an example, you can see how oneAPI.jl runs their tests: https://github.com/JuliaGPU/oneAPI.jl/blob/a00fad6d0532ab7548f236b9293e9dac5845fd0e/.buildkite/pipeline.yml

[michel2323]

I've now tried building MPICH with ze support, and I'm running into all sorts of issues I shouldn't have to.

I've added a has_oneapi() method, which returns true only if the environment variable is explicitly set. At this point, I'd expect the user to explicitly want to be in for a ride. I'll discuss an MPICH build with the MPICH team here at ANL.

[lcw]

Sounds good! Let us know what the MPICH team says. I am happy to merge if building a oneAPI-aware MPICH is too much trouble.

[michel2323]

@lcw, builds are currently failing on hardware without native FP64 support, such as the integrated GPU used in our Buildkite oneAPI runner. The MPICH OpenCL implementation has extensive FP64 dependencies; when ocloc attempts to compile these for non-FP64 devices, it triggers a segfault. This persists even when applying ocloc emulation flags.

I don't have a Intel GPU with FP64 support that I could add to buildkite. Given these hardware constraints, I suggest we merge this as an experimental feature for now.