monocle major overhaul #89
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…support ## Breaking Changes - Library API refactoring: All public functions now accessed through lens structs - Removed legacy helper functions (ip_lookup, time_string_to_time, etc.) - RPKI commons and validator submodules are now private - Users should use lens methods directly (e.g., IpLens::lookup(), TimeLens::parse_time_string()) ## New Features - Added collector argument to rpki aspas command for RPKIviews source ## Improvements - Clean JSON output: When --json flag is set, commands output only valid JSON to stdout - Progress messages, status updates redirected to stderr - Empty results return [] instead of text messages - Affected: rpki, as2rel, broker, whois, search, parse, radar commands - Added --json flag support to whois command ## Code Improvements - Lens-based architecture: Each lens module exports Lens struct, Args structs, and output types - Internal implementation details (helper functions, API calls) are private - CLI commands use lens methods directly for cleaner separation of concerns - Updated documentation in lib.rs and lens/mod.rs
- Add WEB_API_DESIGN.md with comprehensive REST and WebSocket API design - Base Lens trait for all lenses with introspection - WebLens and StreamLens traits for web endpoints - Queue system for resource-intensive operations - Connection management with cancellation support - UI considerations (pagination, batch ops, export formats) - Implementation phases and roadmap - Add DEVELOPMENT.md with contribution guidelines - Step-by-step guide for adding new lenses - Guide for adding web/WebSocket endpoints - Issue location reference for bug fixes - Testing guidelines and code style - Update ARCHITECTURE.md and lens README.md with references to new docs
- Add CountryOutputFormat enum (Table, Json, Simple, Markdown) - Add CountryLookupArgs struct with query, all, and format fields - Add builder methods: new(), all_countries(), with_format() - Add validate() method for argument validation - Add search() method that uses CountryLookupArgs - Add format_results() method for consistent output formatting - Update CLI command to use new Args pattern - Add comprehensive tests for new functionality This change ensures CountryLens follows the same patterns as other lenses (TimeLens, IpLens, etc.) as documented in DEVELOPMENT.md.
Breaking changes: - Remove standalone 'broker' command - Remove 'radar' command (Cloudflare Radar API) - Add '--broker-files' flag to 'search' command to list matching MRT files without searching The '--broker-files' flag provides the same functionality as the old broker command but integrated into the search workflow. It outputs URLs (one per line) or full item details when used with '--json'.
- Add duckdb 1.4.3 dependency to Cargo.toml (keeping rusqlite for SQLite exports) - Create DuckDbConn wrapper in database/core/duckdb_conn.rs - Connection management with file and in-memory support - INET extension loading for IP/prefix operations - Memory limit configuration - Transaction support - Create DuckDbSchemaManager in database/core/duckdb_schema.rs - Schema definitions for all tables (as2org, as2rel, rpki_cache, pfx2as, elems) - INET type support for prefix/IP columns - Array type support for provider_asns and origin_asns - Schema versioning and migration support - Update database/core/mod.rs to export new DuckDB types - Update database/mod.rs with documentation and exports All tests pass (94 passed, 1 ignored)
- Create DuckDbAs2orgRepository with denormalized schema - Single table instead of 3 tables with views - Case-insensitive search using ILIKE - Batch lookup support for org names - Load from bgpkit-commons integration - Create DuckDbAs2relRepository - Same structure as SQLite version, adapted for DuckDB - TIMESTAMP handling for metadata - Aggregated relationships with AS2Org name lookup - Create DuckDbMonocleDatabase wrapper - Unified interface for all monocle data tables - Schema initialization and migration support - Cross-table query support - Memory limit configuration - Update monocle/mod.rs to export both SQLite and DuckDB types - Retain SQLite backend for backward compatibility All tests pass (112 passed, 1 ignored)
Implement caching layers for RPKI and Pfx2as data with DuckDB: RPKI Cache (rpki_cache.rs): - RoaRecord and AspaRecord types for cached data - RpkiCacheMeta for tracking cache freshness - Store and query ROAs by origin ASN, prefix, or covering prefixes - Store and query ASPAs by customer ASN or provider ASN (using list_contains) - Cache freshness checking with configurable TTL - Support for historical data caching (never expires) - Clear and metadata management Pfx2as Cache (pfx2as_cache.rs): - Pfx2asRecord with MOAS support (multiple origin ASNs per prefix) - Pfx2asCacheMeta for tracking cache freshness - Exact prefix lookup - Longest prefix match using INET >>= operator - Find covering prefixes (super-prefixes) - Find covered prefixes (sub-prefixes) - Lookup by origin ASN using list_contains Technical notes: - Manual array parsing for Vec<u32> (DuckDB doesn't implement FromSql for it) - Manual timestamp parsing for DateTime<Utc> - Use split_part for prefix length extraction (masklen not available) - INET containment operators (>>= and <<=) for prefix matching Update DuckDbMonocleDatabase to include cache repositories. All tests pass (127 passed, 1 ignored)
…alidation Implemented Phase 4 of the DuckDB migration plan: 1. Created duckdb_query.rs with query helpers: - PrefixQueryBuilder for building prefix containment queries - build_prefix_containment_clause() for <<= and >>= operators - order_by_prefix_length() for sorting by prefix specificity - RpkiValidationQuery for RPKI validation via SQL JOINs - Pfx2asQuery for pfx2as lookups 2. Created sql_validator.rs for SQL-based RPKI validation: - SqlRpkiValidator for ROA validation using cached data - SqlAspaValidator for ASPA validation using cached data - RpkiStatus/AspaStatus enums for validation results - Bulk validation support with optional detailed results - Cache statistics and availability checks 3. Created query_builder.rs for search query construction: - SearchQueryBuilder with fluent API for building queries - Prefix filtering with include_sub/include_super support - Support for all search filters (origin_asn, peer_asn, etc.) - RPKI annotation queries (adds rpki_status column) - Pfx2as annotation queries (adds pfx2as_origins column) - SearchFilterSpec for converting filter specs to queries 4. Updated module exports: - core/mod.rs: Export query helpers - database/mod.rs: Export query types - lens/rpki/mod.rs: Export SQL validators - lens/search/mod.rs: Export query builder All 175 tests pass.
… migration Completed Phase 5 of the DuckDB migration plan: 1. Updated database/README.md with comprehensive DuckDB documentation: - Dual-backend architecture explanation (DuckDB + SQLite) - Module overview with new DuckDB types and query helpers - Usage examples for connections, queries, caches - Schema definitions for all DuckDB tables - Prefix containment operators reference - Cache TTL configuration - DuckDB-specific notes (INET extension, no INET indexes, etc.) - Testing guide 2. Updated CHANGELOG.md with new features: - DuckDB Internal Database as primary backend - RPKI Cache Repository (ROAs and ASPAs) - Pfx2as Cache Repository - SQL-based RPKI Validation - Query Helpers (PrefixQueryBuilder, SearchQueryBuilder, etc.) - Added duckdb dependency note 3. Updated README.md architecture section: - Noted dual-backend approach (DuckDB + SQLite) - Highlighted native INET type support for prefix queries - Explained SQLite retention for backward compatibility Test results: - 47 DuckDB-specific tests pass - 17 cache tests pass - 27 query builder tests pass - 13 SQL validator tests pass - Build completes cleanly with no warnings Note: Some network-dependent tests (IP lookup, broker queries) may timeout in restricted network environments - these are unrelated to the DuckDB migration.
Added new 'config' command that displays: - Config file path (~/.monocle/monocle.toml) - Data directory path - SQLite database status, size, and record counts (AS2Org, AS2Rel) - DuckDB database status, size, and schema version Options: - --verbose: Lists all files in data directory with sizes and modification times - --json: Machine-readable JSON output This helps users understand where monocle stores its data and what databases are being used (SQLite for legacy/export, DuckDB for primary). Example output: monocle config monocle config --verbose monocle config --json Updated CHANGELOG with the new feature.
Add global --output / -o option with formats: - table (default): Pretty table with rounded borders - markdown: Markdown table format - json: Compact JSON (single line) - json-pretty: Pretty-printed JSON with indentation - json-line: JSON Lines format (one JSON object per line) - psv: Pipe-separated values with header row Changes: - Move all informational/debug messages to stderr for clean piping - Remove per-command format flags (--pretty, --psv) in favor of global option - Add --show-full-name flag to whois and as2rel for disabling name truncation - Add lens/utils.rs with OutputFormat enum and truncate_name helper - Change default output format from markdown to table Bug fixes: - Fix AS2Rel JSON deserialization (remove incorrect serde rename) - Fix AS2Rel duplicate rows for same ASN pair - Fix AS2Rel percentage calculation (connected = peer + as1_upstream + as2_upstream) - Optimize AS2Rel queries with SQL JOINs instead of two-step aggregation Breaking changes: - Removed --pretty flag from whois, as2rel, search, parse commands - Removed --psv / -P flag from whois command - Default output format changed from markdown to table
Remove the -o short form and use long-only --format. Update CLI code, README, and CHANGELOG to reflect the new flag and adjust examples. Add new subcommands: ip, pfx2as, as2rel, and config.
- Add ParseProgress enum with Started, Update, Completed variants - Add SearchProgress enum with QueryingBroker, FilesFound, FileStarted, FileCompleted, ProgressUpdate, Completed variants - Add parse_with_progress() and parse_with_handler() to ParseLens - Add search_with_progress() and search_and_collect() to SearchLens - Progress callbacks are thread-safe (Arc<dyn Fn + Send + Sync>) - Progress types are JSON-serializable for GUI communication - Remove server feature and WEB_API_DESIGN.md (keeping focus on library) - Update ARCHITECTURE.md and CHANGELOG.md This enables building responsive GUI applications that can display real-time progress for long-running parse and search operations.
- Remove duplicate 'list' subcommand (use 'roas' instead) - Rename 'check' to 'validate' with auto-detect prefix/ASN arguments - Update 'roas' to accept multiple resources with union semantics - Add data source display via eprintln for all RPKI commands - Fix markdown table formatting (remove line wrapping in ASPAs) - Current data always uses Cloudflare endpoint, historical uses specified source
The summary subcommand relied on Cloudflare GraphQL API data that is no longer available.
Phase 1 - SQLite Schema and Caching: - Add RpkiRepository with tables for ROAs, ASPAs, and metadata - Store IP prefixes as 16-byte start/end byte arrays (IPv4 mapped to IPv6) - Implement 24-hour cache TTL with automatic refresh - Add --refresh/-r flag to force cache updates Phase 2 - RPKI Validation (RFC 6811): - Implement local validation using SQLite queries - Valid: Covering ROA with matching ASN and prefix_len <= max_length - Invalid: Covering ROA exists but ASN mismatch or length violation - NotFound: No covering ROA for the prefix - Return detailed validation results with reason explanations Schema changes: - Bump SCHEMA_VERSION to 2 - Add rpki_roa table with prefix range indexes - Add rpki_aspa table for customer-provider relationships - Add rpki_meta table for cache metadata
- Delete validator.rs (GraphQL API functions no longer used) - Remove RpkiValidationArgs, RpkiSummaryArgs, RpkiListArgs - Remove RpkiLens::validate() and RpkiLens::list_roas() methods - Remove unused formatting methods from RpkiLens - Clean up exports and tests All validation now uses local SQLite-cached data via RpkiRepository.
- Add new 'monocle database' command with subcommands: - status: Show database status (default when no subcommand) - refresh <source>: Refresh specific data source - refresh --all: Refresh all data sources - backup <dest>: Backup database to destination - clear <source>: Clear specific data source - sources: List available data sources with status - RPKI is now a proper database source (not file cache): - Stored in SQLite using RpkiRepository - Shows separate ROA and ASPA counts - Tracks last update time - Add timing information to all refresh operations: - Shows duration for single source refresh - Shows per-source timing and total time for --all - Batch insert performance optimizations: - PRAGMA synchronous = OFF during batch inserts - PRAGMA journal_mode = MEMORY for faster writes - PRAGMA cache_size = -64000 (64MB cache) - Settings restored to safe defaults after batch - Shared database info functions for config and database commands: - get_sqlite_info(), get_cache_info(), get_cache_settings() - get_data_source_info() for detailed source status - DataSource, DataSourceInfo, DataSourceStatus types - Update config command to show RPKI ROA/ASPA counts separately - Make rpki::commons module public for database refresh access
Add a new 'monocle server' command that starts a WebSocket server providing JSON-RPC style access to monocle functionality. Features: - WebSocket endpoint at ws://<address>:<port>/ws - Health check endpoint at http://<address>:<port>/health - Streaming support with progress reporting for parse/search operations - Operation cancellation via op_id - DB-first policy: queries read from local SQLite cache Available methods: - system.info, system.methods (introspection) - time.parse, ip.lookup, ip.public - rpki.validate, rpki.roas, rpki.aspas - as2org.search, as2org.bootstrap - as2rel.search, as2rel.relationship, as2rel.update - pfx2as.lookup, country.lookup - parse.start, parse.cancel (streaming) - search.start, search.cancel (streaming) - database.status, database.refresh Server dependencies (axum, tokio, futures, etc.) are merged into the 'cli' feature flag for unified builds. Includes example clients in JavaScript and Rust.
Add SQLite-based Pfx2as repository with efficient prefix query support: Storage: - IP prefixes stored as 16-byte start/end address pairs (BLOB columns) - IPv4 addresses converted to IPv6-mapped format for uniform storage - Indexed on prefix range, ASN, prefix length, and prefix string Query modes: - exact: Find prefixes that exactly match the query - longest: Longest prefix match (most specific covering prefix) - covering: Find all supernets (prefixes that cover the query) - covered: Find all subnets (prefixes covered by the query) Features: - Pfx2asRepository with same pattern as RpkiRepository - DEFAULT_PFX2AS_CACHE_TTL of 24 hours - store_from_legacy() for backward compatibility with file cache format - Metadata tracking (source URL, prefix count, record count, updated_at) WebSocket API updates: - pfx2as.lookup now supports 'covering' and 'covered' modes - database.status includes pfx2as record count - database.refresh supports 'pfx2as' source to populate SQLite cache Backward compatibility: - Falls back to file-based cache if SQLite is empty - File cache still supported for existing installations Includes comprehensive tests for all query modes, IPv6 prefixes, metadata, and store/retrieve operations.
Remove backward compatibility for file-based pfx2as cache since it was never deployed. Now all pfx2as data uses SQLite with blob-based prefix storage. Changes: - Remove Pfx2asFileCache, Pfx2asRecord, Pfx2asCacheData, Pfx2asCacheMeta - Remove DEFAULT_PFX2AS_TTL from file_cache.rs - Remove ipnet-trie dependency (no longer needed) - Remove Pfx2asLens trie-based implementation (SQLite handles lookups) - Update CLI pfx2as command to use Pfx2asRepository directly - Add --covering and --covered flags to CLI for new query modes - Update database refresh/clear commands to use SQLite for pfx2as - Simplify lens/pfx2as/mod.rs to types only (Pfx2asEntry, args, modes) The Pfx2asRepository in database module now provides: - lookup_exact(): Exact prefix match - lookup_longest(): Longest prefix match - lookup_covering(): Find all supernets - lookup_covered(): Find all subnets
Major refactoring changes: - Replace as2org module with new asinfo module for AS information - Rename 'whois' command to 'inspect' for unified AS/prefix lookup - Consolidate 'database' command into 'config' command (db-refresh, db-backup, db-sources) - Remove standalone 'pfx2as' command (functionality merged into inspect) - Update pfx2as database to use SQLite blob storage - Add RPKI validation field to pfx2as records Clippy fixes included: - Replace expect() with match for proper error handling - Remove unnecessary casts (as i64, as u64) - Use is_some_and() instead of map_or(false, ...) - Use is_multiple_of() instead of manual modulo check - Collapse nested if statements - Use array literal instead of vec! for fixed-size collections - Add #[allow] attributes for intentional patterns Architecture updates: - Updated ARCHITECTURE.md with new module structure - Added REFACTOR_TODO.md for tracking remaining work - Updated server handlers for new data sources
…mats - Add SIGPIPE signal handler to prevent panics when output is piped to commands like 'head' (Unix only, uses libc) - Disable PSV format for inspect command with helpful error message - Config command now uses table format for markdown/psv (only JSON supported) The broken pipe issue occurred because Rust's default behavior is to panic on SIGPIPE. By resetting to SIG_DFL, the process terminates cleanly instead.
These documents were used during the refactoring process and are no longer needed: - REFACTOR_TODO.md - WEBSOCKET_TODOS.md - src/server/REFACTOR_PLAN.md
This commit reorganizes monocle's feature flags to allow users to use it
as a library with minimal dependencies based on their needs.
Feature Tiers:
- database: SQLite operations only (rusqlite, oneio, ipnet, chrono)
- lens-core: Standalone lenses like TimeLens (adds chrono-humanize, dateparser)
- lens-bgpkit: BGP-related lenses (adds bgpkit-*, rayon, tabled)
- lens-full: All lenses including InspectLens
- display: Table formatting with tabled (included in lens-bgpkit)
- cli: Full CLI binary with server support (default)
Changes:
- Reorganized Cargo.toml with layered feature dependencies
- Feature-gated module exports in lib.rs and lens/mod.rs
- Made tabled derives conditional with #[cfg_attr(feature = "display", ...)]
- Added MsgStore behind lens-bgpkit feature (requires bgpkit_parser)
New Examples (organized by feature tier):
- standalone/: time_parsing.rs, output_formats.rs (lens-core)
- database/: database_basics.rs, as2rel_queries.rs (database)
- bgpkit/: country_lookup.rs, rpki_validation.rs, mrt_parsing.rs, bgp_search.rs (lens-bgpkit)
- full/: inspect_unified.rs, progress_callbacks.rs (lens-full)
Usage examples:
# Minimal database access
monocle = { version = "0.9", default-features = false, features = ["database"] }
# BGP operations without CLI overhead
monocle = { version = "0.9", default-features = false, features = ["lens-bgpkit"] }
# Full functionality
monocle = { version = "0.9", default-features = false, features = ["lens-full"] }
Documentation updates: - Move WEBSOCKET_DESIGN.md to src/server/README.md - Update all documentation references to new location - Remove references to obsolete REFACTOR_PLAN.md Code cleanup: - Remove file_cache module (no longer needed) - RPKI data now stored in SQLite with blob-based prefix storage - Pfx2as data now stored in SQLite with blob-based prefix storage - Historical RPKI data loaded directly via bgpkit-commons - Remove CacheInfo, get_cache_info from config (no file cache to report) - Remove CacheInfoResponse from WebSocket database handler - Remove RpkiFileCache export from lib.rs
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This release introduces a major architectural overhaul of monocle, transforming it from a CLI-only tool into a library-first toolkit with WebSocket API support. The changes improve modularity, enable programmatic access, and consolidate all data storage into SQLite for better performance and simpler deployment.
Key Changes
1. Lens-Based Architecture
All functionality is now accessed through "lens" structs that encapsulate business logic:
Available lenses:
TimeLens,CountryLens,IpLens,ParseLens,SearchLens,RpkiLens,As2relLens,InspectLens2. WebSocket API Server (
monocle server)New WebSocket server enables programmatic access to all monocle functionality:
op_idThis enables building web UIs, integration with other services, and automation workflows.
3. Unified SQLite Storage
All persistent data now uses SQLite with optimized schema:
Benefits:
4. New Commands
monocle inspect: Unified AS/prefix information lookup combining multiple data sources. Replaceswhoisandpfx2ascommands with auto-detection of query type.monocle server: WebSocket API server for programmatic access.monocle config: Consolidated configuration and database management with subcommands for refresh, backup, and status.monocle as2rel: AS-level relationship queries with percentage-based analysis.5. Feature Flag Reorganization
Library users can now select minimal feature sets:
databaselens-corelens-bgpkitlens-fullcli(default)This allows embedding monocle in other applications with minimal dependency overhead.
6. Unified Output Format
All commands now support consistent output formatting via
--format:table(default): Pretty tables with bordersjson,json-pretty,json-line: JSON variantsmarkdown,psv: Other formatsInformational messages go to stderr, enabling clean piping of data output.
Breaking Changes
brokercommand (usesearch --broker-files)radarcommand (use Cloudflare Radar API directly)rpki listandrpki summarycommands (userpki roas)rpki checktorpki validatewhoistoinspectBenefits
For CLI users: Cleaner command structure, unified output formats, faster queries from local cache
For library users: Modular feature flags, typed APIs through lenses, progress callbacks for long-running operations
For automation: WebSocket API enables building web interfaces, integration with monitoring systems, and scripted workflows
For deployment: Single SQLite database file, automatic cache management, configurable TTLs
Testing
Documentation
ARCHITECTURE.md: Updated with lens-based architectureDEVELOPMENT.md: Guide for adding new lensessrc/server/README.md: WebSocket API protocol specificationexamples/: Organized by feature tier with comprehensive examples