Skip to content

Fix range in Oniguruma regex for Hack grammar (#159)#162

Closed
slevithan wants to merge 1 commit intoslackhq:masterfrom
slevithan:regexfix
Closed

Fix range in Oniguruma regex for Hack grammar (#159)#162
slevithan wants to merge 1 commit intoslackhq:masterfrom
slevithan:regexfix

Conversation

@slevithan
Copy link
Contributor

Fixes #159. See extended details there, but essentially, \xHH for values above 7F doesn't the work the same in Oniguruma (the regex engine used by TextMate grammars) as in other regex engines. As a result, a standalone \xff is an invalid UTF-8 encoded byte value, rather than a valid code point value as it would be if using \x{ff}. So although this regex isn't throwing in Oniguruma due to loose error handling for encoded bytes that are never used in a valid UTF-8 encoded byte sequence (bytes F5 through FF), this is in fact an invalid Oniguruma pattern.

This error is currently leading to edge case bugs in Oniguruma (code points above FF are not matched by this negated range) and preventing the Hack grammar from working with Shiki's JS engine (which has stricter error handling for this invalid Oniguruma pattern).

@salesforce-cla
Copy link

Thanks for the contribution! Before we can merge this, we need @slevithan to sign the Salesforce Inc. Contributor License Agreement.

@slevithan
Copy link
Contributor Author

slevithan commented Dec 16, 2024

Heads up that there was a server error when I signed the CLA at the link above, and now when I return to the Salesforce contributor license agreement page and authenticate using GitHub, it tells me "You already signed the CLA on 2024-12-16".

@slevithan
Copy link
Contributor Author

Created a new PR #163 to correctly trigger the cla:signed tag.

@slevithan slevithan closed this Dec 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Hack grammar contains invalid Oniguruma token \xff

1 participant