Multi server multi gpu

[henrykironde]

Description

Related Issue(s)

AI-Assisted Development

• I used AI tools (e.g., GitHub Copilot, ChatGPT, etc.) in developing this PR
• I understand all the code I'm submitting
• I have reviewed and validated all AI-generated code

AI tools used (if applicable):

[codecov]

Codecov Report

❌ Patch coverage is 82.50000% with 35 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.53%. Comparing base (408e150) to head (b9c119c).
⚠️ Report is 5 commits behind head on main.

Files with missing lines	Patch %	Lines
src/deepforest/distributed.py	70.83%	14 Missing ⚠️
src/deepforest/main.py	80.64%	12 Missing ⚠️
src/deepforest/predict.py	76.19%	5 Missing ⚠️
src/deepforest/scripts/evaluate.py	71.42%	2 Missing ⚠️
src/deepforest/datasets/prediction.py	96.77%	1 Missing ⚠️
src/deepforest/scripts/train.py	90.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1367      +/-   ##
==========================================
- Coverage   86.87%   85.53%   -1.35%     
==========================================
  Files          24       25       +1     
  Lines        3064     3326     +262     
==========================================
+ Hits         2662     2845     +183     
- Misses        402      481      +79     

Flag	Coverage Δ
unittests	85.53% <82.50%> (-1.35%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[bw4sz]

Can you confirm what SLURM script you used to check this so i can match that?

[bw4sz]

I want to approve this, @jveitchmichaelis any objections? I have spoken to comet and they agree that the lack of multi-node GPU utilization graph is probably on their end and not anything wrong here.

[bw4sz]

remove references to torchrun, we can use srun alone.

[bw4sz]

To do, is @henrykironde comparing this with #1304 and decide if both are needing, but we want to get this done because it's broad and could make rebasing harder.