Zero out category score for showstopper rules (e.g. no-a11y-statement)#79
Open
marcusosterberg wants to merge 1 commit into
Open
Zero out category score for showstopper rules (e.g. no-a11y-statement)#79marcusosterberg wants to merge 1 commit into
marcusosterberg wants to merge 1 commit into
Conversation
Some rules represent failures so fundamental that the affected category
should be considered failed outright, rather than just deducted from.
Missing an accessibility statement when it is legally required is one
such case: under the current scoring (-25 for severity 'critical'),
the site still gets 75/100, which doesn't reflect that a mandatory
artifact is entirely absent.
This commit introduces a `showstopperRules` set in Score. Any issue
whose rule is in the set, with a non-resolved severity, forces its
category score to 0 instead of going through the normal severity
deduction. Currently only 'no-a11y-statement' is in the set.
Other plugins using severity 'critical' (notably the lighthouse
converter's crash fallback) are unaffected, since they use different
rule names. Sites that have a statement but flag it as partially
compliant ('compatible-word-partly', severity 'error') also continue to
score as before.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a mechanism for rules that represent fundamental, "you failed the
exam" failures to zero out their category score, instead of going
through the normal severity deduction.
Motivation
The current scoring deducts 25 points for
severity: 'critical', whichmeans a site missing a legally required accessibility statement scores
75/100 (rating 3.75/5.0). That doesn't reflect the severity of the
underlying problem — a legally mandated artifact is entirely missing.
We can't simply make all
criticalissues zero out the category,because the lighthouse converter also uses
severity: 'critical'inits try/catch fallback when an audit crashes. That's a technical
failure of one audit, not a wholesale failure of the category. So the
trigger must be a rule whitelist, not the severity field.
Change
Scorenow holds ashowstopperRulesset. DuringcalculateScore,any issue whose rule is in the set, with a non-resolved severity,
forces its category score to 0. All other scoring behaviour is
unchanged.
Currently the set contains only
'no-a11y-statement'. If we lateridentify other fundamental failures (e.g. missing HTTPS, depending on
policy), they can be added here.
Impact on other plugins
None. Verified by examining all uses of
severity: 'critical'acrossthe plugin family:
lighthouse-converter.jscrash fallback usescriticalbut withaudit-specific rule names — not in the showstopper set.
plugin-accessibility-statementusescriticalonly forno-a11y-statement, which is the rule we explicitly want to flag.Verification
https://finsamornskoldsvik.se(no statement):score.a11ygoesfrom 75 → 0,
rating_a11yis now lowest possible.https://eskilstuna.se("delvis förenlig" statement): unchanged,score.a11y: 90.Related
Fixes <link to issue in Webperf-se/webperf_core>