Add tokenutil alias backend support

[JustAGhosT]

Summary

• map Sluice policy aliases to cl100k_base token counting
• add optional SentencePiece backend stub for Gemini-family models
• tighten alias coverage tests

Validation

• pytest tests/ -v
• mypy src/tokenutil

[blocksorg]

Mention Blocks like a regular teammate with your question or request:

@blocks review this pull request
@blocks make the following changes ...
@blocks create an issue from what was mentioned in the following comment ...
@blocks explain the following code ...
@blocks are there any security or performance concerns?

Run @blocks /help for more information.

Workspace settings | Disable this message

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 51bd0cd5-a501-462b-9a53-9edb20e0e463

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch phase-4-tokenutil

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 35a41e4223

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Convert model-load errors into fallbackable ImportError

If TOKENUTIL_SENTENCEPIECE_MODEL is set to an invalid/missing file, SentencePieceProcessor(model_file=...) raises a non-ImportError exception (commonly OSError), which is not caught here. Because count_tokens_for_text only falls back on ImportError, Gemini token counting will raise instead of using the heuristic path, turning a recoverable misconfiguration into a runtime failure.

Useful? React with 👍 / 👎.