Point detection workflow

[bw4sz]

As part of DeepForest 2.0 we need a model for point prediction.

A couple examples we can work from

https://github.com/xingyizhou/CenterNet.
https://github.com/FateScript/CenterNet-better
https://arxiv.org/abs/1904.07850

We have the data from

https://doi.org/10.1016/j.jag.2024.103848

Roadmap

• Add small point dataset example to get_data. We should use the existing box dataset (OSBS_029) and just get the centroids
• Add a point dataloader to datasets.py similar to

  DeepForest/deepforest/dataset.py

  Line 204 in 7e0aa43

  class BoundingBoxDataset(Dataset):

• Add a point model to models.
• Example training script in examples/
• Confirm visualization
• Confirm can be extended to multi-class

@Mu-Magdy might be interested, since its isolated from the code base, doesn't have a time frame and fits objectives.

Bonus

• Train a tree point detector on the MillionTrees point datasets and/or the center points of the boxes/polygons datasets.

[bw4sz]

Also check https://github.com/airctic/icevision/tree/master/icevision/models/torchvision/keypoint_rcnn

[bw4sz]

and https://link.springer.com/chapter/10.1007/978-3-031-92387-6_12

[bw4sz]

The question is whether we use segment anything to just convert these points into box structures to train.

[musaqlain]

hi @bw4sz

I've been researching the point detection workflow outlined in this issue over the past few days, and I'd like to contribute to implementing this feature. After thoroughly analyzing all the options discussed here and conducting additional research, I believe a custom CenterNet implementation is the best path forward because: (1) it is purpose-built for point detection unlike adapting box-based models, (2) it is fast (140+ FPS) with no NMS post-processing needed, and (3) it can integrate into DeepForest's existing BaseModel wrapper pattern.
I spent significant time studying the DeepForest codebase to understand the existing patterns , particularly how RetinaNet implements the BaseModel interface and how BoxDataset handles annotations. I also reviewed PR #590 (multi-geometry support) which provides the geopandas foundation for handling point annotations.
Here's my planned roadmap, and I need your suggestions/guidelines along the way:

1. Create PointDataset class that generates Gaussian heatmap targets from point annotations
2. Implement CenterNetHub model with ResNet backbone + heatmap/offset heads
3. Add focal loss for heatmap + L1 loss for offset regression
4. Implement peak extraction via 3x3 max pooling (built-in NMS)
5. Add point visualization and evaluation metrics (distance-based instead of IoU)
6. Create training script in examples/train_centernet.py
7. Add comprehensive unit tests

I read the original 'Objects as Points' paper, it includes size regression (for height/width), but I think the point coordinates might be sufficient, right?

[ethanwhite]

@jveitchmichaelis has been working on this so pinging him to engage with @musaqlain

[jveitchmichaelis]

Hi @musaqlain thanks for offering to work on this. We are working internally on some of the infrastructure to support non-box geometries, so I would prefer if you could wait until we have a dataset + training loop pushed. This is likely to touch a large part of the codebase. However, once that's done we'd would welcome other architecture work if you're interested.

#1311 adds a dataset that includes both keypoint and heatmap output. You could also look at #1182 which is an experimental integration of a DeformableDETR architecture with keypoint (xy) output, but I'm breaking that up into separate PRs for ease of review.

I'm hoping to add a basic trained model in the next couple of weeks to validate the pipeline.

Please use the recently updated CLI tools for training rather than a separate script (e.g. call deepforest train + config options). A script is also OK, but I would like to encourage users to use the train tool as it sets up a lot of the logging.

[musaqlain]

hi @jveitchmichaelis

thankyou for your reply and for pointing me to those PRs. to fully understand and align myself with Deepforest 2.2 milestone, i spent the full day studying those 2 PRs (#1311 and #1182) and here is my progress:

related to #1311 (keypoint dataset pipeline):

• I understand how KeypointDataset generates Gaussian heatmaps (via output="density") / keypoints (via output="centroid") for detection tasks, and it will cover the dataloading requirements for my CenterNET implementation.
• I also ran test_datasets_training_keypoint.py locally to observe how Kornia handles affine transformations on point geometries.

related to #1182 (evaluation and matching):

• after exploring this PR (specifically keypoint_distance.py and evaluate.py) and reading other articles I now understand it completely. since IoU is invalid for point detection tasks, you have implemented Hungarian matching algo.. using Euclidean pixel distance**, bounded by the pixel_distance_threshold.**
• I also explored models/keypoint.py to understand the wrapper pattern and main.py's training_step to correctly interface (or returning the required loss_dict) and to consider specific loss components.

since, it would take some time before you give green signal for architectural work, I will start building a standalone CenterNet wrapper in my private repo/ or locally that will adheres to this exact API contract.

I am highly interested in working on this specific issue, and I would like you to please share any insights/ understanding/ workflow related to this implementation. 🙌

--
last but not least, I see DETR keypoint architecture in #1182 is currently marked as experimental, I want to know if the CenterNet implementation is also intended to remain experimental for now, or is there any plan to merge it as fully supported model by Deepforest package? 👀
thank for your attention