fix: prevent learn tool from hanging + background indexing + memory optimizations

[jaggernaut007]

Summary

Fixes the learn tool hanging on large codebases with many folders/subfolders, reduces peak memory consumption during indexing, adds configurable timeout protection, and implements background indexing to prevent MCP transport timeouts.

Closes #1

Changes (3 commits)

Commit 1: fix: prevent learn tool from hanging on large codebases

• .gitignore support — discover_files() rewritten with os.walk() + pathspec for directory pruning; respects root and nested .gitignore files; followlinks=False prevents symlink loops
• Safety limits — max_files=200,000 circuit breaker (addresses SECURITY_REVIEW HIGH-003)
• Upsert-based indexing — get_or_create_collection() + collection.upsert() instead of destructive delete-recreate; stale chunks cleaned after embedding
• Resumable checkpointing — .codegrok/checkpoint.json saved every 1000 chunks via atomic os.replace(); deleted on success
• Progress reporting — new discovery_progress event + ETA in embedding progress messages

Commit 2: perf: reduce memory consumption and add configurable timeout

• Inline symbol→chunk conversion — symbols converted to chunks per file batch then freed
• File batch processing — parse 500 files at a time (FILE_BATCH_SIZE) with gc.collect() between batches
• Worker cap — parallel parse workers capped at 4 (MAX_PARSE_WORKERS, was up to 32)
• Post-embedding cleanup — explicit del chunks + gc.collect() after embedding
• Configurable timeout — default 600s; CODEGROK_TIMEOUT env var or timeout_seconds param

Commit 3: fix: background indexing to prevent MCP transport timeout

• Background indexing — learn returns immediately; indexing runs in threading.Thread(daemon=True); client polls get_stats() for progress
• IndexingStatus — Thread-safe dataclass in state.py with threading.Lock for progress tracking
• Stateful responses — learn returns indexing_started (new), indexing_in_progress (already running), complete (done), or raises ToolError (failed)
• get_stats() enhanced — includes indexing field with active, progress, message, error during indexing
• load_only unchanged — remains synchronous since it's fast (just loads existing index)

Why background indexing? MCP clients (Claude Desktop, etc.) have ~60-120s transport-level timeouts that kill long-running tool calls. Internal timeouts can't help because the transport drops first. Background threading solves this by returning immediately.

Polling workflow:

1. learn(path="/project")   → {"status": "indexing_started", "progress": 0}
2. get_stats()              → {"indexing": {"active": true, "progress": 42, "message": "Embedding..."}}
3. get_stats()              → {"loaded": true, "stats": {...}}  (indexing complete)

New Dependency

• pathspec>=0.11.0 — pure Python .gitignore pattern matching

Security

• Snyk Code Scan: 0 issues on all modified files (3 scans across all commits)
• Addresses HIGH-003 (Unbounded Resource Consumption / DoS) via max_files limit
• Addresses LOW-009 (Symlink Following) via followlinks=False

Test plan

• 10 unit tests for discover_files + memory constants (tests/unit/test_discover_files.py)
• 9 unit tests for timeout configuration (tests/unit/test_timeout.py)
• 33 unit tests for background indexing (tests/unit/test_background_indexing.py)
• 11 integration tests for upsert, checkpointing, memory optimizations (tests/integration/test_source_retriever.py)
• 191/191 tests pass in full suite (30s)
• black --check passes on all modified files
• Snyk SAST scan: 0 issues

Configuration

{
  "mcpServers": {
    "codegrok": {
      "command": "codegrok-mcp",
      "env": {
        "CODEGROK_TIMEOUT": "1200"
      }
    }
  }
}

🤖 Generated with Claude Code