Skip to content

Conversation

@proost
Copy link
Contributor

@proost proost commented Jan 7, 2026

Currently I'm writing code for go t-Digest.

While I refer C++ implementation, i found that no guard for +inf, -inf.

Additional Question: There is no validation logic in the t-Digest about overflow or underflow from calculation. Is intended behavior?

@AlexanderSaydakov
Copy link
Contributor

Regarding overflow and underflow, I believe I followed the reference implementation as close as practical. If you see potential problems I would suggest asking @tdunning first.

@tdunning
Copy link

tdunning commented Jan 8, 2026

@proost Let's talk. Feel free to ping me directly on my apache email address. I would love to work with you on a Go implementation and there are important simplifications in Go relative to the Java implementation because you can easily have an array of structs.

Regarding your question, a guard for ±inf is more subtle than it looks. If you insert a single infinite value, it will have a centroid with one data point in it. That's fine. If you insert another on the same side, you will start to get centroids with infinite value which is actually numerically correct. You will also have effects on the interpolation code, but that is also likely fairly reasonable.

How do you think an empirical distribution with infinite values should be handled?

What do you think the mean of such a distribution should be?

@proost
Copy link
Contributor Author

proost commented Jan 8, 2026

@tdunning
Thank you for the thoughtful response. I now understand that whether to allow infinities really depends on what semantics we want the empirical distribution to have.

In my use case, I was mainly trying to protect the t-digest from values that come from numeric overflow or underflow, not from intentional ±∞ in the data. Your explanation helped clarify the distinction for me.

@proost
Copy link
Contributor Author

proost commented Jan 8, 2026

@AlexanderSaydakov

There is one thing I wanted to clarify regarding this PR.

In the current implementation, ±∞ values can flow into the t-digest and be treated as data points. With this PR, ±∞ values passed to update() are silently ignored, and ±∞ passed to query methods such as get_rank() result in an exception. That means this change alters the behavior for users who might already be passing infinity values (intentionally or not), which effectively makes it a breaking change.

If supporting ±∞ as valid data was intentional in the original design, then I agree that changing this behavior is not appropriate. However, if it wasn’t an intentional design choice and infinity handling was simply unspecified, then I think explicitly rejecting/ignoring infinities and documenting the behavior would improve robustness and make the policy clearer for users.

I wanted to check your view on this before moving forward, since it affects compatibility and library semantics.

@tdunning
Copy link

tdunning commented Jan 8, 2026 via email

@proost
Copy link
Contributor Author

proost commented Jan 8, 2026

@tdunning

Thanks for the detailed explanation and examples.

Since this PR goes against core model of data structure, I’m going to close it. Thanks
for taking the time to explain the reasoning — it helped clarify the right direction here!

@proost proost closed this Jan 8, 2026
@proost proost deleted the fix-tdigest-inf-params branch January 8, 2026 10:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants