Adapt parser to monorepo

[randyspauldingamd]

Proposed changes

Addressing issues found trying to run the dependency parser on MIOpen:

• Ninja is recording the full path, e.g.: [json]

  "file_to_executables": {
     "/home/rspauldi/repos/rocm-libraries/projects/miopen/include/miopen/miopen.h": [

• Running git in monorepo reports the full relative path, e.g.:

    "projects/miopen/include/miopen/miopen.h"

Of course, git diff also returns all files modified in every other project's commits. These are filtered out as early as possible.

This solution searches for rocm-libraries in the parsing step, and if found extracts the project name and stores it in enhanced_dependency_mapping.json. Leading folders are truncated from each file path, up to and including the project name. This allows _is_project_file to remain unchanged.

The selection step then retrieves the project name from the json if it is defined, and truncates the project folder from the git diff output so the filenames exactly match the json entries.

Checklist

Please put an x into the boxes that apply. You can also fill these out after creating the PR. If you're not sure, please don't hesitate to ask.

• I have added tests relevant to the introduced functionality, and the unit tests are passing locally
• I have added the test to REGRESSION_TESTS list defined at the top of CMakeLists.txt in tests/CMakeLists.txt, IF the test takes more than 30 seconds to run.
• I have added inline documentation which enables the maintainers with understanding the motivation
• I have removed the stale documentation which is no longer relevant after this pull request
• (If this change is user-facing) I have added release notes which provide the end users with a brief summary of the improvement from this pull request
• I have run clang-format on all changed files
• Any dependent changes have been merged

Discussion

Successfully runs on rocm-libraries MIOpen PRs and produces a list of tests. I haven't verified the results yet.

This version is not applicable to CI since it operates on a per-executable level and MIOpen CI uses the single gtest binary. I'll be working towards that in future PRs over the next few weeks.

/home/rspauldi/repos/rocm-libraries/projects/miopen# git checkout miopen/sgundabo_enable_ck_bwd_wrw_navi
<run CMake with TEST_DISCRETE=ON>
# ninja tests

# root@rjs1:/home/rspauldi/repos/rocm-libraries/projects/miopen# python3 /dep/main.py parse build/build.ninja
Parsing ninja dependencies from: build/build.ninja
Parsing ninja build file...
Found 312 executables
Found 820 object-to-source mappings
Found 820 object files
Extracting detailed dependencies for all object files...
Processed 100/820 object files...
Processed 200/820 object files...
Processed 300/820 object files...
Processed 400/820 object files...
Processed 500/820 object files...
Processed 600/820 object files...
Processed 700/820 object files...
Processed 800/820 object files...
Completed dependency extraction for 820 object files
Building file-to-executable mapping...
Found rocm-libraries project: 'miopen'
Built mapping for 608 files
Files used by multiple executables: 216
Sample files with multiple dependencies:
  build/include/miopen/config.h: 306 executables
  build/include/miopen/export.h: 306 executables
  build/include/miopen/export_internals.h: 304 executables
  driver/InputFlags.hpp: 2 executables
  driver/driver.hpp: 2 executables

=== Enhanced Dependency Mapping Summary ===
Total executables: 312
Total files mapped: 608
Total object files processed: 820

File types:
  .cpp files: 310
  .hpp files: 292
  .h files: 6

Files used by multiple executables: 216

Top files with most dependencies:
  build/include/miopen/config.h: 306 executables
  build/include/miopen/export.h: 306 executables
  include/miopen/miopen.h: 304 executables
  src/include/miopen/config.hpp: 304 executables
  build/include/miopen/export_internals.h: 304 executables
  src/include/miopen/rank.hpp: 303 executables
  src/include/miopen/errors.hpp: 302 executables
  src/include/miopen/object.hpp: 302 executables
  src/include/miopen/returns.hpp: 302 executables
  src/include/miopen/sysinfo_utils.hpp: 302 executables
Exporting mapping to build/enhanced_file_executable_mapping.csv
Exporting complete mapping to build/enhanced_dependency_mapping.json

Results exported to:
  CSV: build/enhanced_file_executable_mapping.csv
  JSON: build/enhanced_dependency_mapping.json

root@rjs1:/home/rspauldi/repos/rocm-libraries/projects/miopen# python3 /dep/main.py select  build/enhanced_dependency_mapping.json  1b13d8b72d54e34bdc7ae70dd2b6e809dca8b10e  09e5965d55ebbfacfd1ed18e5092580c2ffae748
Identified 30 files modified in project 'miopen'
Exported 304 tests to run to tests_to_run.json

I don't know if clang-format applies to scripts. If so, could someone show me how to run it in CK?