Skip to content

[VL] Isolate /usr/local from native builds on macOS#12105

Open
jackylee-ch wants to merge 1 commit into
apache:mainfrom
jackylee-ch:fix-macos-isolate-usr-local
Open

[VL] Isolate /usr/local from native builds on macOS#12105
jackylee-ch wants to merge 1 commit into
apache:mainfrom
jackylee-ch:fix-macos-isolate-usr-local

Conversation

@jackylee-ch
Copy link
Copy Markdown
Contributor

What changes are proposed in this pull request?

On Homebrew-based macOS, /usr/local accumulates leftover headers,
libraries, and CMake packages from prior Velox/Gluten builds (often
root-owned after past sudo make install runs from cmake_install
in the bootstrap). When the native build includes /usr/local/include
and /usr/local/lib/cmake in its CMake search paths it picks up stale
or mismatched versions of re2, glog, fmt, etc., shadowing Velox's
deps-install copies and producing confusing link or compile errors.

Concrete failure modes that show up before this fix:

  • Fresh build picks up /usr/local/lib/cmake/glog from a prior
    cmake --install and links a stale glog into libvelox.dylib
    alongside the bundled glog from deps-install.
  • /usr/local/include shadows Cellar headers (e.g. fmt) and yields
    static_assert failures because Velox's bundled fmt advanced past
    what /usr/local/include/fmt provides.
  • Link errors against an old re2 left over from an aborted Velox
    bootstrap.

This PR forces CMake to ignore /usr/local when building native deps
on macOS, in three places that drive the third-party / Velox / Gluten
CMake configures:

File / Function Change
dev/build-helper-functions.sh::cmake_install adds -DCMAKE_NO_SYSTEM_FROM_IMPORTED=ON, -DCMAKE_IGNORE_PREFIX_PATH=/usr/local, -DCMAKE_IGNORE_PATH, -DCMAKE_SYSTEM_IGNORE_PATH for the third-party CMake invocations driven by the Arrow/Velox bootstrap
ep/build-velox/src/build-velox.sh::compile same flags when configuring Velox itself, plus suppresses two Apple-Clang-only warnings that fire under -Werror on macOS but are benign noise: -Wno-inconsistent-missing-override and -Wno-macro-redefined
dev/builddeps-veloxbe.sh::build_gluten_cpp same isolation flags plus the same warning suppressions for the Gluten CMake configure

All blocks are gated on [[ "$(uname)" == "Darwin" ]] / if [ $OS == 'Darwin' ]. Linux build/link semantics are unchanged.

Without this isolation, the macOS native build is non-deterministic
depending on what's left over in /usr/local; with it, the build only
reads from the well-defined Homebrew Cellar prefix (/opt/homebrew)
and Velox's deps-install.

How was this patch tested?

  • macOS 14 arm64 with Apple Clang 17 and a /usr/local populated
    with stale glog/re2/fmt installs from a prior session: with these
    flags applied, the build no longer references /usr/local/include
    or /usr/local/lib in any compile or link command (verified via
    _build/release/build.ninja), and produces clean libvelox.dylib
    • libgluten.dylib that pass the Gluten CPP ctest matrix.
  • Spark 3.5 + Velox backend Java JNI canary suites
    (VeloxBloomFilterTest, ColumnarBatchTest, VeloxListenerApiTest,
    OnHeapFileSystemTest, ArrowColumnVectorTest) all pass on the
    resulting build.
  • Linux x86_64 Ubuntu 22.04 build green; all isolation blocks are
    Darwin-gated. Local Ubuntu build verified clean.

Was this patch authored or co-authored using generative AI tooling?

co-auth: Claude (Sonnet/Opus) via Claude Code 1.x

… macOS

On Homebrew-based macOS, `/usr/local` accumulates leftover headers,
libraries, and CMake packages from prior Velox/Gluten builds (often
root-owned after past `sudo make install` runs from `cmake_install`
in the bootstrap script). When the native build includes
`/usr/local/include` and `/usr/local/lib/cmake` in its CMake search
paths it picks up stale or mismatched versions of re2, glog, fmt,
etc., shadowing Velox's `deps-install` copies and producing confusing
link or compile errors.

Concrete failure modes seen:
- Fresh build picks up `/usr/local/lib/cmake/glog` from a prior
  `cmake --install` and links a stale glog into libvelox.dylib
  alongside the bundled glog from `deps-install`.
- `/usr/local/include` shadows Cellar headers (e.g. fmt) and yields
  `static_assert` failures because Velox's bundled fmt advanced past
  what `/usr/local/include/fmt` provides.
- Link errors against an old re2 left over from an aborted Velox
  bootstrap.

Force CMake to ignore `/usr/local` when building native deps on macOS
in three places that drive third-party / Velox / Gluten CMake configs:

- `dev/build-helper-functions.sh::cmake_install`: adds
  `-DCMAKE_NO_SYSTEM_FROM_IMPORTED=ON`,
  `-DCMAKE_IGNORE_PREFIX_PATH=/usr/local`, `-DCMAKE_IGNORE_PATH`,
  `-DCMAKE_SYSTEM_IGNORE_PATH` for the third-party CMake invocations
  driven by the Arrow/Velox bootstrap.

- `ep/build-velox/src/build-velox.sh::compile`: same flags when
  configuring Velox itself, plus suppress two Apple-Clang-only
  warnings that fire under `-Werror` on macOS but are benign noise:
  `-Wno-inconsistent-missing-override` and `-Wno-macro-redefined`.

- `dev/builddeps-veloxbe.sh::build_gluten_cpp`: same isolation flags
  plus the same warning suppressions for the Gluten CMake configure.

All blocks are gated on `[[ "$(uname)" == "Darwin" ]]` /
`if [ $OS == 'Darwin' ]`. Linux paths are unchanged. Without this
isolation, the macOS native build is non-deterministic depending on
what's left over in `/usr/local`; with it the build only reads from
the well-defined Homebrew Cellar prefix (`/opt/homebrew`) and Velox's
`deps-install`.

Verification:
- macOS 14 arm64 with Apple Clang 17 and a `/usr/local` populated
  with stale glog/re2/fmt installs from a prior session: with these
  flags applied, the build no longer references `/usr/local/include`
  or `/usr/local/lib` in any compile or link command (verified via
  `_build/release/build.ninja`), and produces clean libvelox.dylib +
  libgluten.dylib that pass the gluten cpp ctest matrix.
- Linux x86_64 build green; all blocks are Darwin-gated.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant