feat: scan runner pipeline (Plans 1-5) + performance optimizations#9
Merged
feat: scan runner pipeline (Plans 1-5) + performance optimizations#9
Conversation
Task-graph-based security scan orchestration engine with auto-detect profiles, reactive edges, Claude-assisted steering, semantic finding dedup, and cross-tool corroboration scoring. Covers: data model, DAG executor, MCP client, profiles, finding pipeline, surface integration (CLI/web/Claude skill), database schema, and testing strategy. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds: target rate limiting, task isolation/sandboxing, task coalescing, dependency-aware pre-fetching, orphaned resource cleanup, graceful degradation matrix, observability metrics, scan rollback, scan quotas, CVSS-calibrated severity, finding context enrichment, multi-pass dedup, compressed output caching, and preferred output format selection. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
15-task implementation plan covering: all Pydantic models, CancellationToken, CWE hierarchy + static data, shared subprocess/progress/retry/resource_pool modules, SqliteScanStore, Finding model update, and RecipeRunner refactor. Plan 1 of 5 for the scan-runner feature. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ckage structure Adds the scanner orchestration engine package under opentools/scanner with all enum types (ScanStatus, ScanMode, TargetType, TaskType, TaskStatus, ExecutionTier, TaskIsolation, EvidenceQuality, LocationPrecision) and core Pydantic models (TargetRateLimit, NotificationChannel, ScanNotification, RetryPolicy, ScanConfig, ScanMetrics, ReactiveEdge, ScanTask, Scan). Also scaffolds empty sub-packages for data, executor, parsing, and shared. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements CancellationToken with idempotent cancel(), reason tracking, and async wait_for_cancellation(). Includes 5 TDD tests covering initial state, cancel, idempotency, delayed wakeup, and immediate return. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…aps, title normalization Add 6 JSON data files to scanner/data/: cwe_hierarchy.json (parent/child relationships for 35+ CWEs), cwe_aliases.json (60+ lowercase aliases to canonical CWE IDs), cwe_owasp_map.json (CWE→OWASP Top 10 2021), severity_maps.json (per-tool severity label normalization for semgrep/nuclei/trivy/codebadger/ gitleaks/nikto/nmap/sqlmap), title_normalization.json (34 regex patterns to canonical finding titles), and parser_confidence.json (base confidence scores per tool). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements run_streaming() with 4096-byte stdout chunking, stderr capture, asyncio.wait()-based timeout, CancellationToken integration, and FileNotFoundError guard; backed by 7 TDD tests (all passing). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add execute_with_retry for async functions using RetryPolicy (max_retries, backoff_seconds, retry_on). Non-matching errors propagate immediately; retryable errors use backoff_seconds * 2^attempt delay. Includes _is_retryable helper with case-insensitive pattern matching against type name and str(error). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add runtime-checkable ScanStoreProtocol and aiosqlite-backed SqliteScanStore with JSON blob persistence for Scan and ScanTask models; also adds scan_id field to the core Finding model with two covering tests. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements Task 14: priority-aware concurrency pool with per-group slot limits, heapq-backed waiter queue (lowest priority number = highest priority), FIFO tiebreaking within equal priorities, and cancellation safety. Also migrates RecipeRunner shell steps to use shared subprocess. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The original implementation placed the cancellation-wait task inside asyncio.wait(ALL_COMPLETED), causing the wait to block for the full timeout when a CancellationToken was provided but never triggered. Replace with a background watchdog coroutine that kills the process when cancellation fires, which unblocks the I/O reader tasks naturally. asyncio.wait now only tracks the two I/O tasks. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…scovery, execute Implements McpConnection (lazy connect, stdio/HTTP transports, tool discovery, call_tool, clean disconnect) and McpExecutor (server registry, execute with cancellation guard and error capture, close_all). Tool discovery and invocation are stubs pending JSON-RPC wiring in a later plan. 14 tests covering connection lifecycle, protocol compliance, execute resilience, cancellation, and close_all. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Accumulates tool output in memory (default 10 MB), spills to a temp file on overflow, and raises OverflowError when a per-run disk cap (default 500 MB) is exceeded. Used as an on_output callback by the scan engine. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…dges, partial failure Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds DetectedTarget and SourceMetadata Pydantic models plus TargetDetector with pattern-based resolution (URL, CIDR/IP, Docker image, file extension, source directory) and lightweight source metadata extraction. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds TargetValidator with per-type async validation (source directory, URL via aiohttp HEAD, binary magic bytes, APK ZIP/manifest, Docker inspect, network ping) plus ValidationResult model and __all__ export list. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ScanProfile, ProfilePhase, ProfileTool, ReactiveEdgeTemplate with Pydantic v2 validation; load_builtin_profile, load_profile_yaml, list_builtin_profiles, and DEFAULT_PROFILES mapping per target type. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Eight YAML profiles covering all target types: source-quick, source-full, web-quick, web-full, binary-triage, network-recon, container-audit, apk-analysis. Includes reactive edge templates for binary packing detection, high-severity deep-dive, web framework rulesets, and open-port vuln scanning. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
OpenPortsToVulnScan, WebFrameworkToRuleset, PackingDetectedToUnpack, HighSeverityToDeepDive, plus get_builtin_evaluators() registry. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add SteeringAction enum and GraphSnapshot model to models.py. Implement SteeringDecision, SteeringInterface protocol (runtime_checkable), and SteeringThrottle with four frequency modes (every_task, phase_boundary, findings_only, manual). Scan completion always triggers regardless of mode. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Resolves profile inheritance, evaluates tool conditions against target metadata, builds phase-ordered task DAG with proper dependencies, resolves command templates, and instantiates ReactiveEdgeTemplate into concrete ReactiveEdge instances. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Implements plan(), execute(), pause(), resume(), cancel() as the unified public interface for scan orchestration; plan() creates a Scan record and calls ScanPlanner to produce the task DAG, lifecycle methods delegate to the active engine or cancellation token. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Extend ScanStoreProtocol and SqliteScanStore with methods for raw findings, deduplicated findings, progress events, suppression rules, FP memory, output cache, and tool effectiveness stats. Adds 8 new SQLite tables with appropriate indexes. 18 new tests all passing, 7 existing store tests unaffected. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements ScanPipeline that assembles ParserRouter -> NormalizationEngine -> DedupEngine -> CorroborationScorer -> SuppressionEngine -> FindingLifecycle -> Store. Extends ScanEngine with optional pipeline param; completed task outputs are queued and processed asynchronously via _process_pipeline_results. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds opentools scan subcommand group with plan, profiles, run, status, history, findings, and cancel commands. Registers scan_app in the main CLI entry point. Uses asyncio.run() to bridge async ScanAPI from sync Typer handlers. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add FastAPI router at /api/v1/scans with full endpoint set:
- GET/POST /api/v1/scans (list, create)
- GET /api/v1/scans/{id} (detail), /tasks, /findings
- POST /api/v1/scans/{id}/pause|resume|cancel (control)
- GET /api/v1/scans/{id}/stream (SSE event stream)
- GET /api/v1/scans/profiles
Register router in main.py. Test: 17 structural/model tests.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add migration adding 16 scan-related tables: scan, scan_task, raw_finding, dedup_finding, finding_correlation, remediation_group, suppression_rule, fp_memory, finding_annotation, scan_event, steering_log_entry, scan_attestation, output_cache, tool_effectiveness, scan_batch, scan_metrics. Follows existing 001-005 pattern with idempotent upgrade() and full downgrade(). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Wire ScanAPI.execute() from stub to real implementation: - Creates AdaptiveResourcePool, EventBus, CancellationToken - Registers ShellExecutor (Docker/MCP require caller-supplied context) - Builds ScanPipeline when store is provided - Constructs ScanEngine with pipeline, runs DAG, returns final Scan - Tracks active scans for pause/resume/cancel Also add e2e integration tests: mock executor DAG execution, pipeline finding persistence, multi-task dependency ordering, and ScanAPI.execute end-to-end (5 tests, 497 total). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Profile parsing consumed 73% of engine runtime. The same YAML file was re-parsed on every ScanPlanner.plan() call. Since profiles are static at runtime, cache the parsed ScanProfile in a module-level dict.
Replaces the N+1 pattern where get_summary() (7 SQL queries) was called per-engagement in the sidebar refresh loop. Single LEFT JOIN query returns engagement_id + critical/high counts for all engagements at once.
_apply_refresh now only calls update_from_state() on the visible tab instead of all 4 tabs. Tab switches trigger an immediate refresh of the newly visible tab. Sidebar uses get_sidebar_summaries() batch query instead of N calls to get_summary() (7 SQL statements each). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
resolve_alias() had a linear scan fallback over all alias keys for case-insensitive matching. Pre-building a lowercase-keyed dict in __init__ makes all lookups O(1). Profiler showed 4,000 calls during pipeline normalization.
refresh_selected() now accepts a 'needs' set specifying which data categories to fetch. The Findings tab only queries findings + summary. Docker container status HTTP call is skipped unless the Containers tab is active. Eliminates 3 of 4 SQLite queries and the Docker API call on most refresh ticks. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Each tab and the sidebar now compute a lightweight tuple snapshot of their data before rebuilding. If the snapshot matches the previous tick, the table.clear() + rebuild is skipped entirely — no Rich markup parsing, no Textual layout reflow, no Pydantic model_construct calls. Only actual data changes trigger a visual update. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…thrashing Every scan endpoint was opening a new SQLite connection, setting WAL mode, running PRAGMAs, doing one query, then closing. Under concurrent SSE connections this caused 100 open/close cycles per second. Now uses a module-level lazy singleton with double-checked locking.
SSE event stream was polling SQLite every 1 second even when idle. Now starts at 0.5s and backs off to 5s max when no events arrive. Resets to aggressive polling on activity. Reduces idle queries ~80%.
…waits Pipeline was saving findings one-by-one in serial await loops (200+ individual round-trips per scan). Now batches raw and dedup finding saves into single transactions. Falls back to serial for stores that don't implement batch methods.
Pipeline stages were copying every finding via Pydantic model_copy() at each stage (1000+ copies per scan). Since the pipeline owns these objects with no shared references, direct attribute mutation is safe and eliminates all copy overhead.
Strict dedup pass was building four separate defaultdict indexes, each hashing and allocating per-finding. Now uses one dict with priority-ordered composite keys. 4x fewer hash computations and list allocations.
Reactive edge attachment was scanning all tasks per tool definition. Build a tool→tasks dict once, use O(1) lookups for edge attachment. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Schedule loop was scanning in_flight dict to find task_id from completed future. Maintain reverse mapping for O(1) lookup on completion. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… tuning Unifies chain store backends behind ChainStoreProtocol with async SQLite/Postgres implementations. Adds web scan service, IOC finder enhancements, profiling tooling, and broad performance improvements across scanner pipeline, dashboard, and subscription layers. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ChainStoreProtocolwith async SQLite and PostgreSQL backends, replacing the prior sync implementations across CLI and webTest plan
pytest packages/cli/tests/— all CLI tests passpytest packages/web/backend/— all web backend tests passopentools scan run) and verify findings pipeline🤖 Generated with Claude Code