CMake / Ninja / Make build graph interrogation

[Notgnoshi]

• What's the object file that corresponds to a given source file?
• What things trigger CMake reconfiguration?
• What files get regenerated after a CMake reconfigure?
• If I change a file, how many things will get rebuilt? What are they? Can I tell this without doing an incremental build?
• I did an incremental build, but lots more things rebuilt than I expected - why?
• What is the header dependency graph for each TU?
• What tools can I used to interrogate my header dependency graph?
• How can I find headers that aren't needed?
• How can I detect circular header inclusion?

[Notgnoshi]

What's the object file that corresponds to a given source file?

ninja -t list all | grep example.cpp

What's the dependency graph of a given object file?

ninja -t deps <target>

where is one of the object file paths from above.

Documentation might lead you to look at ninja -d inputs, but I think that’s probably not the result most developers are looking for. I think that gives you the build ordering dependency graph, but not the “Here’s what source files that make up my TU” dependency graph.

You might be tempted to look at ninja -t graph , but the generated graphs are huge and take long enough to render with dot that I gave up.

What source files transitively depend on a given header?

grep -r header.hpp . --include='*.d'

in the build directory.

to clean it up a bit:

grep --include=*.d -l -r header.hpp . | xargs -l head -1 | cut -d: -f1 | sort -u

What things trigger a CMake reconfigure?

See the RERUN_CMAKE rule in build.ninja

grep RERUN_CMAKE build.ninja | cut -d'|' -f2 | tr ' ' '\n' | sort -u

What's the command used to build a given object file?

• look at compile_commands.json
• ninja -t commands <target> - and possibly combine with tail -1

[Notgnoshi]

Use compdb -p ./build list >compile_commands.json to generate a compilation database that includes header files.

This can help when you're editing a header-only portion of a library that doesn't have source files for clangd to scrape compiler arguments from, which would otherwise cause clangd to throw errors on your header until you open a source file that includes it.

[Notgnoshi]

#include dependency graph tools

• ninja -t deps <target>
• https://includeguardian.io/instructions
• https://github.com/bkryza/clang-include-graph
  • Preferred over my own tool https://github.com/Notgnoshi/includegraph
• https://github.com/andy31415/igraph
  • config file looks complex, but possibly pretty powerful?
• https://www.flourish.org/cinclude2dot/
  • Perl script to parse your code, probably breaks with cross-compiler
• Doxygen INCLUDE_GRAPH and INCLUDED_BY_GRAPH
• https://github.com/include-what-you-use/include-what-you-use
  • unsure if this has a side-effect-free mode for interrogation
  • docs: https://github.com/include-what-you-use/include-what-you-use/tree/master/docs

[Notgnoshi]

Build profiling tools

• compile-time-perf
• externis
• -ftime-trace (requires clang)
  • https://aras-p.info/blog/2019/01/16/time-trace-timeline-flame-chart-profiler-for-Clang/
• .ninja_log + buildbloat + webtreemap
• https://github.com/aras-p/ClangBuildAnalyzer/
• https://cmake.org/cmake/help/latest/manual/cmake.1.html#cmdoption-cmake-profiling-output
• https://alangrow.com/blog/profiling-every-command-in-a-makefile
• https://github.com/konturio/make-profiler
• https://github.com/nico/ninjatracing
• https://danielchasehooper.com/posts/syscall-build-snooping/
• https://github.com/jeremy-rifkin/cpp-dependency-analyzer - does not require clang
  • Mostly folded into https://github.com/jeremy-rifkin/build-blame which is built on top of clang -ftime-trace

[Notgnoshi]

What's your high level (target / library) dependency graph?

https://cmake.org/cmake/help/latest/manual/cmake.1.html#cmdoption-cmake-graphviz

[Notgnoshi]

What other questions might I ask during a build systems interrogation session?

• What are the top 10 worst source file offenders - use compile-time-perf
• Sort headers by number of source files that include them - ninja deps
• Sort headers (and sources) by how frequently they change - can identify bug hotspots and poor APIs
• What's the critical path in the build graph? Longest dependency chain? Can you decide this just with depth, or do you also need timing stats?
• Are there parts of the build where we're not parallelizing effectively? (cargo build timings was really good at this, can we do something similar for C++?)
• For any header, what source files depend on it?
• Are precompiled headers helping? Or are they masking poor dependency hygiene?
• What headers are good candidates for precompilation?
• If I change a source file, what will rebuild?
• If I change a header, what will rebuild?
• What all happens during a CMake reconfigure?
• When and where does code generation happen?
• Where can I find the generated code?
• What's the maximum include depth? Deeply nested includes are spicy
• Sort source files by the number of headers they transitively include
• What components have the highest fan-in / fan-out?
• How can I find circular dependencies in headers?
• Which C++ stdlib headers are expensive to include? Can we reduce the number of source files that include them? (or precompile?)
• How sensitive is the project to platform-specific code? Architecture specific code?
• How to identify forward declaration candidates?
• How to identify unused headers?
• How to identify headers that depend on symbols that aren't included? (depend on other headers being included before them)
• How to identify headers missing an include guard?
• Do headers compile standalone? (g++ -fsyntax-only header.h)
• Are there private headers being improperly included by other components? How to tell?
• Are there public headers that should be private? How to tell?
• Are there template instantiations repeated in multiple object files? How to tell?
• What headers define templated / inlined functions used in only one source file?
• Are there PUBLIC cmake dependencies that should be PRIVATE or INTERFACE? How to tell? What's the impact?
• What headers are precompiled? Does this make sense?
• What's the pch compile time?
• Where are our conditional compilation switches? Do we test them? Are they dead?
• How many object files are in each of our libraries?
• How do we decide whether to use an object library or a static library?
• Which object files contain the most symbols?
• Are there circular or other awkward library dependencies?
• How to identify circular symbol dependencies?
• Which header in a TU provides a given symbol? How did it get included?
• What includes can be moved from the header to the source file of a TU?
• How can I find the concrete instantiations of a templated type?