Skip to content

Commit 632179e

Browse files
sidmohan0claude
andcommitted
style(ci): fix flake8 formatting issues in fair_benchmark.py
- Add missing whitespace around arithmetic operators - Remove trailing whitespace - Clean up blank lines with whitespace Resolves pre-commit CI failures in GitHub Actions. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
1 parent f8c3ef5 commit 632179e

File tree

2 files changed

+125
-88
lines changed

2 files changed

+125
-88
lines changed

scripts/benchmark_analysis_report.md

Lines changed: 30 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -7,59 +7,69 @@
77
## Methodology Validation
88

99
### Fair Benchmark Approach
10+
1011
- **Clean Environment**: Used minimal dependencies (only spaCy + Pydantic) to eliminate interference
1112
- **Identical Test Data**: Both engines processed the exact same 13.3KB text sample
1213
- **Multiple Runs**: 5 measured runs per engine (excluding warmup) to ensure statistical reliability
1314
- **Real-world Text**: Test data included actual PII patterns users would encounter
1415
- **Proper Warmup**: Each engine ran once before measurement to eliminate cold-start effects
1516

1617
### Test Data Characteristics
18+
1719
- **Size**: 13.3KB (10x multiplier of 1.33KB base text)
1820
- **Content**: Realistic business document with emails, phones, SSNs, credit cards, names, organizations, dates, etc.
1921
- **PII Density**: High concentration of various entity types for comprehensive testing
2022

2123
## Raw Performance Numbers
2224

2325
### Fair Benchmark Results (3 Runs)
26+
2427
| Run | Regex Time | SpaCy Time | Speedup Ratio |
25-
|-----|------------|------------|---------------|
26-
| 1 | 2.42 ms | 458.76 ms | 189.6x |
27-
| 2 | ~2.4 ms | ~460 ms | 193.0x |
28-
| 3 | ~2.4 ms | ~474 ms | 197.9x |
28+
| --- | ---------- | ---------- | ------------- |
29+
| 1 | 2.42 ms | 458.76 ms | 189.6x |
30+
| 2 | ~2.4 ms | ~460 ms | 193.0x |
31+
| 3 | ~2.4 ms | ~474 ms | 197.9x |
2932

3033
**Average Speedup**: **193.5x faster**
3134

3235
### Throughput Analysis
36+
3337
- **Regex Engine**: 5,502 KB/s
3438
- **SpaCy Engine**: 29 KB/s
3539
- **Performance Gap**: 190x throughput advantage for regex
3640

3741
### Existing Benchmark Comparison
42+
3843
The existing pytest benchmarks showed similar patterns:
44+
3945
- **Regex**: 4.05 ms mean time
40-
- **SpaCy**: 394.42 ms mean time
46+
- **SpaCy**: 394.42 ms mean time
4147
- **Speedup**: 97.4x faster (on ~10KB text)
4248

4349
The fair benchmark shows higher speedup ratios, likely due to:
50+
4451
1. Cleaner test environment (fewer dependencies)
4552
2. Different text composition
4653
3. More focused measurement approach
4754

4855
## Entity Detection Analysis
4956

5057
### Regex Engine Results
58+
5159
- **Total Entities Found**: 190 entities
5260
- **Entity Types**: EMAIL (50), PHONE (70), SSN (20), CREDIT_CARD (20), IP_ADDRESS (30)
5361
- **Precision**: High precision for structured PII (emails, phones, SSNs)
5462
- **Approach**: Pattern-based matching for well-defined formats
5563

56-
### SpaCy Engine Results
64+
### SpaCy Engine Results
65+
5766
- **Total Entities Found**: 550 entities
5867
- **Entity Types**: PERSON (80), ORG (70), GPE (90), CARDINAL (110), DATE (70), TIME (40), MONEY (50), PERCENT (30), FAC (10)
5968
- **Precision**: Mixed precision due to NLP interpretation
6069
- **Approach**: Natural language understanding for contextual entities
6170

6271
### Detection Comparison
72+
6373
- **Regex**: Fewer entities but higher precision for structured PII
6474
- **SpaCy**: More entities but includes contextual/semantic matches
6575
- **Complementary**: Each engine excels at different types of PII detection
@@ -68,18 +78,21 @@ The fair benchmark shows higher speedup ratios, likely due to:
6878
## Technical Findings
6979

7080
### Performance Characteristics
81+
7182
1. **Regex Consistency**: Very stable performance (±0.08ms standard deviation)
7283
2. **SpaCy Variability**: Higher variability (±23.38ms standard deviation) due to model complexity
7384
3. **Memory Usage**: Regex uses minimal memory; spaCy loads large language models
7485
4. **Scalability**: Regex performance scales linearly; spaCy has model overhead
7586

7687
### Accuracy Assessment
88+
7789
1. **Structured PII**: Regex is more accurate for emails, phones, SSNs, credit cards
7890
2. **Contextual PII**: SpaCy better detects people, organizations, locations in natural text
7991
3. **False Positives**: SpaCy prone to over-detection; regex more conservative
8092
4. **Entity Coverage**: Different engines detect non-overlapping entity types
8193

8294
### System Requirements
95+
8396
1. **Regex**: No external models, minimal resource requirements
8497
2. **SpaCy**: Requires 15-50MB language models, more CPU/memory intensive
8598
3. **Startup Time**: Regex instant; spaCy has model loading overhead
@@ -88,22 +101,26 @@ The fair benchmark shows higher speedup ratios, likely due to:
88101
## Marketing Recommendations
89102

90103
### Validated Claims
104+
91105
**"190x faster than spaCy"** - Defensible and accurate based on comprehensive testing
92106
**"High-performance regex engine"** - 5,500+ KB/s throughput validates this claim
93107
**"Intelligent engine selection"** - Auto mode combines both approaches effectively
94-
**"Production-ready performance"** - Consistent sub-3ms response times
108+
**"Production-ready performance"** - Consistent sub-3ms response times
95109

96110
### Updated Marketing Copy
111+
97112
**Before**: "123x faster than spaCy"
98113
**After**: "190x faster than spaCy-based PII detection"
99114

100115
### Positioning Strengths
116+
101117
1. **Speed Advantage**: Clear and measurable performance benefit
102118
2. **Resource Efficiency**: Lower memory and CPU requirements
103119
3. **Precision**: Higher accuracy for structured PII types
104120
4. **Scalability**: Better performance at enterprise scale
105121

106122
### Competitive Advantages
123+
107124
1. **No Model Dependencies**: Works without downloading large ML models
108125
2. **Instant Startup**: No model loading time
109126
3. **Predictable Performance**: Consistent response times
@@ -112,18 +129,21 @@ The fair benchmark shows higher speedup ratios, likely due to:
112129
## Technical Recommendations
113130

114131
### Current Benchmarks Assessment
132+
115133
1. **Accuracy**: Existing pytest benchmarks are adequate for CI/CD
116134
2. **Coverage**: Good coverage of different engines and scenarios
117135
3. **Performance Targets**: Current thresholds appropriate (100x+ faster requirement)
118136
4. **Monitoring**: Benchmark automation provides good regression detection
119137

120138
### Suggested Improvements
139+
121140
1. **Consistency**: Use the fair benchmark approach for marketing measurements
122141
2. **Documentation**: Document the methodology for external validation
123142
3. **Baselines**: Establish the 190x number as the new baseline for CI monitoring
124143
4. **Test Scenarios**: Add more diverse text types to benchmark suite
125144

126145
### Performance Targets for CI/CD
146+
127147
1. **Regression Threshold**: No more than 10% performance degradation
128148
2. **Minimum Speedup**: Maintain 150x+ advantage over spaCy
129149
3. **Throughput Target**: Keep regex above 5,000 KB/s
@@ -132,12 +152,14 @@ The fair benchmark shows higher speedup ratios, likely due to:
132152
## Limitations and Caveats
133153

134154
### Test Scope
155+
135156
1. **Text Size**: Tested on 13.3KB samples; larger texts may show different ratios
136157
2. **Content Type**: Business document format; other domains may vary
137158
3. **Hardware**: MacBook M-series results; Intel/cloud performance may differ
138159
4. **spaCy Model**: Used small model (en_core_web_sm); large models would be slower
139160

140161
### Comparison Fairness
162+
141163
1. **Entity Types**: Engines detect different PII types, making direct comparison challenging
142164
2. **Accuracy vs Speed**: Different precision/recall tradeoffs between engines
143165
3. **Use Cases**: Each engine optimized for different scenarios
@@ -153,4 +175,4 @@ The performance advantage translates to real business value through reduced infr
153175

154176
**Report Generated**: May 25, 2025
155177
**Test Environment**: macOS, Python 3.12, Clean benchmark environment
156-
**Validation**: Multiple runs with consistent results (±2% variance)
178+
**Validation**: Multiple runs with consistent results (±2% variance)

0 commit comments

Comments
 (0)