Add possibility to run CKF Tracking on multiple threads

[samf25]

I made a number of changes to the algorithms in k4ActsTracking.

1. I improved the TruthTracking alg. Previously, it wasn't actually making use of its maps' runtime speed ups. I also used tbb's threading for the beefy for loop. This took the runtime (with BIB) from ~1000s to 0.5141s.
2. I used stable sort and memory reserving instead of the insertion method in the DuplicateRemoval to decrease runtime (I think this is $N^2 \rightarrow N\log N$).
3. I parallelized the CKFTrackingAlg's bottle neck loops (hit loop and seeding loop). This brings the runtime under control $\left(\frac{1}{n_{Threads}}\right)$ without increasing the memory demands (see plot).

[samf25]

The steering files for these new, parallel algs can be found here.
These steering files also use the thread-safe version of k4GaudiPandora

[madbaron]

Hi @samf25,

we've merged the PR with the previous set of updates into the key4hep/main.
When you have a moment, could you take a look at resolving the conflicts here?

We'll then merge this into our main, and start the PR to propagate the changes upstream.
Once we are done with this, I think we'll be able to close our fork, so any additional updates should target https://github.com/key4hep/k4ActsTracking directly.

[samf25]

@madbaron Merge conflicts resolved. I believe I caught all the places where @tmadlener's changes to the SourceLink TrackerHit structure (Thank you for those by the way! It was bothering me the way I was doing it before) would affect my changes.

[samf25]

There was a bug in the TrackTruthAlg where, if no prior entry existed in the map, it dereferenced the end of the map. Now it updates correctly.

[tmadlener]

Unless you have made any changes to any of the files in the data directory these can be removed again. Since key4hep#25 these are all downloaded via cmake in order to not pollute the git repo too much.

Otherwise this looks good to me. There are a few minor things that could be done. I have also added a few questions for clarification.

[tmadlener]

It looks like there is only one use of this in track_equal. Since the workaround is not necessary we could remove it entirely and simply use a std::find there, I think.

[tmadlener]

I just had another look, a few minor things below. The main thing that is blocking me from merging at the moment is the presence of all the data files. Did you change any of those?

[tmadlener]

Out of curiosity, does this impose a penalty timewise on the sequential execution compared to not using tbb? I suppose there would at least be some slowdown due to all the (needless) synchronization primitives that now also happen on a single thread, but I am not sure if this would be visible in our case since we do so much other stuff.

[samf25]

As for the data dir, I haven't messed with it. I remember you saying that you wanted to take it out entirely and just move it to k4geo. Is this what you want to do?

[tmadlener]

The files are already automatically downloaded:

k4ActsTracking/data/CMakeLists.txt

Lines 21 to 36 in fd2f114

	set(DATA_BASE_URL "https://key4hep.web.cern.ch/key4hep/k4ActsTracking/MuColl/data")
	set(FILE_LIST_PATH "${CMAKE_CURRENT_SOURCE_DIR}/file_list.txt")
	set(OUTPUT_DIR "${CMAKE_CURRENT_SOURCE_DIR}")
	message(STATUS "Running download script for data files...")
	execute_process(
	COMMAND bash "${CMAKE_CURRENT_SOURCE_DIR}/download_files.sh" ${FILE_LIST_PATH} ${DATA_BASE_URL} ${OUTPUT_DIR}
	RESULT_VARIABLE download_result
	ERROR_VARIABLE download_error
	)
	if(download_result EQUAL 0)
	message(STATUS "Data files download completed successfully")
	else()
	message(FATAL_ERROR "Failed to download data files:\n${download_error}")
	endif()

this has been introduced in the upstream and then merged back here.

[samf25]

Ah great, so I should just clear out all the files in the data dir? Or remove the whole dir itself?

[tmadlener]

Clear out the full data dir, but leave the CMakeLists.txt, file_list.txt and download_files.sh in there. Effectively all .json and .root files can go as they will be fetched during cmake.