add point model

[jveitchmichaelis]

Point prediction model

This PR adds a model that can be used for keypoint/point detection. The (segmentation) model is trained to output a 1/4 size density map with sharp peaks corresponding to training data (i.e. trees). A local maximum detector is run over the map and these peaks are returned to the user, with scores integrated from a small radius. There is no technical limit on the number of predictions per forward pass, and the model seems to perform well in tiled scenarios.

The architecture is based on Treeformer, using the supervised branch, which in turn is a modification of DM-Count and some other past work. I've trained checkpoints on the authors' KCL dataset which replicates their results, and a pretrained checkpoint on NEON Lidar which seems to transfer quite well to other unseen datasets, like some of our test imagery in the everglades.

Default checkpoint is here: https://huggingface.co/weecology/deepforest-keypoint/tree/main and can be loaded with deepforest(config='keypoint').

This branch also includes tested inference code (I've checked tiling) and refactors some of the prediction post-processing to handle the different geometries a little more cleanly. The upshot is main is largely unchanged.

I'm not a huge fan of adding scikit-image as a dependency, but their peak detection algorithm (peak_local_max) is very good and I've not found a suitable alternative. There is a discussion on GH about this, but no conclusion.

This branch does not have training support (forward doesn't return a loss), which is in a separate branch and will be split out separately for ease of review.

This example is from Hidden, resampled to 5 cm/px, but probably needs a tweak to the NMS (peak separation) threshold.

Replication

Sample predictions trained and tested on the paper's KCL dataset (orange open circles are gt, annotated Google Earth imagery at 0.2 m/px):

Note: this dataset is probably overfit. The paper reports 500 epochs, but the dataset is only 400 images. Nevertheless, the implementation here beats the paper benchmark in supervised mode and also the unsupervised benchmark (likely due to backbone pretraining) based on MAE (paper: 16.7/18.5, ours: 18.24 using density sum / <15 with peak extraction). We get a peak F1 of around 0.68. Convergence is seen after around 200 epochs, but we train for 500 to match the original hyperparameters.

Predictions after a single epoch on NEON LIDAR (purple are preds). The model used for the prediction above was trained for 10 epochs and will be updated when we have a final version as it still had training headroom.

Model differences

• TreeFormer is, as far as I can tell, not invariant to input image shape. The vanilla implementation trains on 256 crops and then predicts over 256 tiles. If you change the input shape at test time, the model will under or over-predict tree density. This implementation changes the output to regress density instead of absolute count, and learns a scale factor that is applied to the input image. This does only affect scale factor, but it's nice to be consistent here.
• In the original paper the model learns spatial structure and count separately. In this version, we couple the two together by forcing the sum of the density map to match the object count for consistency. Typically during training, the model will quickly learn how to place tree "points" and then will slowly learn to correct the predict count output.

Licensing

TreeFormer does not list a license, and the ancestral code from DM-Count (which is mostly copied verbatim in TreeFormer) is MIT-licensed. Here, the backbone model is taken directly from transformers and the various heads are re-implemented in PyTorch with some optimization from the original.

Related Issue(s)

#809

AI-Assisted Development

• I used AI tools (e.g., GitHub Copilot, ChatGPT, etc.) in developing this PR
• I understand all the code I'm submitting
• I have reviewed and validated all AI-generated code

AI tools used (if applicable):

Claude (Opus/Sonnet 4.6) and Codex 5.3/5.4

[codecov]

Codecov Report

❌ Patch coverage is 93.23308% with 27 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.61%. Comparing base (38cc698) to head (e98fede).
⚠️ Report is 6 commits behind head on main.

Files with missing lines	Patch %	Lines
src/deepforest/models/treeformer.py	93.28%	9 Missing ⚠️
src/deepforest/predict.py	85.00%	9 Missing ⚠️
src/deepforest/models/treeformer_decoder.py	95.32%	5 Missing ⚠️
src/deepforest/metrics.py	87.50%	2 Missing ⚠️
src/deepforest/utilities.py	94.11%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1381      +/-   ##
==========================================
+ Coverage   86.92%   87.61%   +0.69%     
==========================================
  Files          24       26       +2     
  Lines        3204     3537     +333     
==========================================
+ Hits         2785     3099     +314     
- Misses        419      438      +19     

Flag	Coverage Δ
unittests	87.61% <93.23%> (+0.69%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[jveitchmichaelis]

Adding a doc page now, but this is good to review.

The coverage gaps are training related - I'll add a sanity test case that'll catch those for now.

Hmm. Maybe we want to call the checkpoint deepforest-keypoint-tree? Then we open the door to other classes more easily.

[jveitchmichaelis]

@ethanwhite I haven't tapped you for a review, but probably good to have everyone's eyes on this.

[bw4sz]

waiting on the metrics to review.

[jveitchmichaelis]

Metrics added, I really thought we'd merged that already but I think just the dataset...

@bw4sz ready for you, and then I'll squash.

To avoid confusion, I've renamed all everything to point instead of keypoint as the names are interchangeable in the literature, but keypoint is mostly used for pose/skeleton estimation.

Prediction results tested with:

uv run deepforest --config-name point predict /tmp/odm_orthophoto_005m.tif --mode tile

And Treeformer's KCL paper results I've tested as (I didn't push the checkpoint, but might as well):

uv run deepforest evaluate data/treeformer/test.csv \
  --root-dir data/treeformer/images \
  architecture=treeformer \
  "model.name=scratch/london_025_hf" \
  score_thresh=0.15 \
  "point.score_integration_radius=2"

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃      Validate metric      ┃       DataLoader 0        ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│      point_precision      │     0.703460156917572     │
│       point_recall        │    0.6459216475486755     │
│         val_loss          │    17.438983917236328     │
│          val_mae          │    16.968944549560547     │
└───────────────────────────┴───────────────────────────┘

[jveitchmichaelis]

Once #1384 is in, I'll refactor the test here.

[ethanwhite]

Once #1384 is in, I'll refactor the test here.

Just merged #1384

[bw4sz]

Looks great. We should probably call this 'point' and not 'keypoint' anywhere, since keypoint makes me think of deeplabcut style things and I think our target user will be happier with point. Next step is to see what the score is on the MillionTrees eval. Great work!