fix: Add test harness, CLI options, risk tests, Docker sidecars, legacy API bridge, and real-time UI

[DevOpsMadDog]

fix: Add test harness, CLI options, risk tests, Docker sidecars, legacy API bridge, and real-time UI

Summary

This PR adds missing methods to the test harness classes, CLI options, comprehensive test coverage for risk modules, a complete Docker sidecar infrastructure for demos/testing, a legacy API bridge router to expose archive/enterprise_legacy APIs through the main app, and real-time API data fetching for the dashboard and pentagi UIs.

Test Harness & CLI Fixes

1. ServerManager.upload_files() - New method to upload SAST, SBOM, CVE, design, CNAPP, and context files to the API and trigger pipeline execution. This was missing and causing 14+ tests in test_critical_decision_policy.py to fail with AttributeError.

2. FlagConfigManager.create_config() - New convenience wrapper method that delegates to create_overlay_config or create_demo_config. Tests were calling this method but it didn't exist.

3. CLI --api-url option - Added --api-url option to scan and monitor commands. Tests were passing --api-url after the subcommand, but the option was only defined at the group level.

4. LLMProviderManager class - Added missing LLMProviderManager class to core/llm_providers.py. Multiple files were importing this class but it didn't exist.

5. CI configuration fixes - Updated .github/workflows/ci.yml to install requirements-test.txt and removed pytest-benchmark references from pytest.ini.

6. Flake8 cleanup - Fixed hundreds of flake8 errors across ~50 files.

Legacy API Bridge Router

Added apps/api/legacy_bridge_router.py to expose legacy APIs from archive/enterprise_legacy/src/api/v1/ through the main app without modifying the legacy code:

Bridged APIs (6 modules, 23 endpoints):

• business_context_enhanced: SBOM context upload (/api/v1/business-context/*)
• feeds: CVE/KEV feed status (/api/v1/feeds/*)
• processing_layer: Bayesian/Markov/Fusion testing (/api/v1/processing/*)
• production_readiness: Production readiness checks (/api/v1/production-readiness/*)
• sample_data_demo: Demo data generation (/api/v1/demo/*)
• system_mode: Demo/Enterprise mode toggle (/api/v1/system-mode/*)

NOT bridged (missing dependencies or already covered):

• business_context, cicd, decisions, scans: require bcrypt/sqlalchemy
• marketplace, oss_tools: hardcoded /app paths
• monitoring, evidence, policy, docs, system: duplicated by modern routers

Real-Time UI Data Fetching (No Demo Data)

Removed all hardcoded demo data from dashboard and pentagi UIs. Both apps now fetch data exclusively from real API endpoints:

Dashboard App (web/apps/dashboard/):

• app/lib/apiClient.ts - Extended with endpoints for MTTR, teams, issue trends, resolution trends, compliance trends, and recent findings
• app/hooks/useDashboardData.ts - Fetches all chart data from real API endpoints with 30-second polling
• app/page.tsx - Removed all hardcoded demo constants; shows loading spinner and error states when API unavailable

Pentagi App (web/apps/pentagi/):

• app/lib/apiClient.ts - API client for pentagi endpoints
• app/hooks/usePentagiData.ts - Fetches pentest requests, findings, and stats from real API
• app/page.tsx - Removed all hardcoded demo data; shows loading/error states

Configuration:

• NEXT_PUBLIC_FIXOPS_API_URL - API base URL (default: http://localhost:8000)
• NEXT_PUBLIC_FIXOPS_API_TOKEN - API token (default: demo-token)

Important: The UIs will NOT work without a running API server. There is no fallback to demo data - this was intentionally removed per user request.

Comprehensive Test Coverage

Added ~5000 lines of rigorous tests for risk modules to improve coverage:

• risk/scoring.py - 66 tests covering EPSS, KEV, version lag, exposure, and reachability scoring
• risk/secrets_detection.py - 30 tests covering secret patterns, file scanning, recommendations
• risk/threat_model.py - 35 tests covering CVSS parsing, reachability scores, threat model computation
• risk/enrichment.py - Tests for enrichment evidence and CVE enrichment
• risk/forecasting.py - Tests for Bayesian and Markov forecasting models
• risk/license_compliance.py - Tests for license detection and compatibility
• risk/runtime/iast.py - Tests for IAST analyzer, taint tracking, findings management
• risk/runtime/iast_advanced.py - Tests for advanced taint analysis, ML detection, anomaly detection
• risk/reachability/ - Tests for analyzer, cache, and code analysis modules

Docker Sidecar Infrastructure

Added comprehensive Docker sidecar setup for customer demos and testing:

• Dockerfile.sidecar - Lightweight Python container for running demos and API tests
• Dockerfile.risk-graph - Multi-stage build for Next.js Risk Graph UI
• scripts/demo_sidecar.py - Interactive demo script with rich animated CLI output
• scripts/feed_sidecar.py - Real-time CVE/KEV feed ingestion from CISA and NVD APIs
• scripts/micropentest_sidecar.py - Animated micro penetration testing with attack chain simulation
• tests/sidecar/test_smoke.py - Smoke test suite covering 40+ API endpoints
• docker-compose.yml - Updated with profile-based sidecars: demo, test, feeds, pentest, ui

Attack Chain Simulation with External CVE Sources

The attack-chain command in micropentest_sidecar.py supports:

• Custom SBOM files (--sbom): Accept CycloneDX or SPDX JSON files
• External CVE sources (--cve-source): demo, live (NVD/CISA KEV), or feeds
• LLM-based narratives (--use-llm): Generate intelligent attack narratives
• KEV filtering (--kev-only): Only include actively exploited CVEs

Updates Since Last Revision

• Removed ALL demo data from UIs - Dashboard and pentagi pages no longer have hardcoded demo constants
• Extended dashboard API client - Added endpoints for MTTR metrics, teams, issue trends, resolution trends, compliance trends, and recent findings
• Updated dashboard hook - Now fetches all chart data (MTTR, teams, trends, findings) from real API endpoints
• Added loading/error states - Both pages show loading spinners and error messages with retry buttons when API is unavailable
• No fallback behavior - UIs require a running API server; they will show errors instead of demo data if API is down

Review & Testing Checklist for Human

• Verify API endpoints exist - The dashboard now calls /api/v1/analytics/mttr, /api/v1/teams, /api/v1/analytics/issue-trends, /api/v1/analytics/resolution-trends, /api/v1/analytics/compliance-trends, /api/v1/analytics/findings/recent. Confirm these endpoints are implemented in the backend or the UI will show errors.
• Test UI with API server running - Start the API server and verify both dashboard and pentagi pages load real data. Check browser Network tab to confirm API calls are being made.
• Test UI without API server - Stop the API server and confirm the UIs show appropriate loading/error states (not blank screens or crashes).
• Verify legacy API bridge doesn't break existing routes - The sys.path manipulation in legacy_bridge_router.py could have side effects. Test that existing modern API endpoints still work correctly.

Recommended Test Plan

# Start the API server
docker run -d --name fixops -p 8000:8000 -e FIXOPS_API_TOKEN="demo-token" devopsaico/fixops:latest

# Test dashboard UI with real-time data
cd web/apps/dashboard
npm install
NEXT_PUBLIC_FIXOPS_API_URL=http://localhost:8000 NEXT_PUBLIC_FIXOPS_API_TOKEN=demo-token npm run dev
# Open http://localhost:3000 - should show real data or loading/error states

# Test pentagi UI with real-time data
cd web/apps/pentagi
npm install
NEXT_PUBLIC_FIXOPS_API_URL=http://localhost:8000 NEXT_PUBLIC_FIXOPS_API_TOKEN=demo-token npm run dev
# Open http://localhost:3001 - should show real data or loading/error states

# Test legacy API bridge
curl -H "X-API-Key: demo-token" http://localhost:8000/api/v1/feeds/status

# Test micropentest demo
python scripts/micropentest_sidecar.py demo

Notes

• CI may still be failing due to pre-existing issues: CodeQL security alerts (ignored per user request), coverage threshold not met (pre-existing)
• Breaking change: The UIs no longer work offline or without an API server. This was intentionally requested by the user ("nothing in ui should be demo").
• The UI hooks use a 30-second polling interval, configurable via the pollInterval parameter.
• The default API URL (http://localhost:8000) and token (demo-token) are for local development. Production deployments should set the environment variables appropriately.
• The legacy bridge uses sys.path manipulation which is not ideal but avoids modifying the legacy code.

Link to Devin run: https://app.devin.ai/sessions/85aef85fa473442fac2a1df0409ec3a6
Requested by: shiva kumaar (umshiva1@gmail.com) / @DevOpsMadDog

[devin-ai-integration]

Original prompt from shiva

Take a look at all the code in PR #187 and #186 , #191 and #192 , #190 ,fix its comments and do changes until all pre-merge is fixed , then trigger off deepwiki indexing in https://app.devin.ai/wiki/DevOpsMadDog/Fixops 



You only need to look in the following repo: DevOpsMadDog/Fixops

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment and CI monitoring

[cubic-dev-ai]

2 issues found across 3 files

Prompt for AI agents (all 2 issues)

Check if these issues are valid — if so, understand the root cause of each and fix them.


<file name="tests/harness/flag_config_manager.py">

<violation number="1" location="tests/harness/flag_config_manager.py:110">
P2: The `modules` parameter is silently ignored when `feature_flags` is None. If a caller expects to use demo feature flags with custom modules (e.g., `create_config(modules={&quot;guardrails&quot;: False})`), the custom modules will be discarded. Consider either documenting this limitation clearly, raising a warning/error when modules is provided without feature_flags, or passing modules to the demo config logic.</violation>
</file>

<file name="tests/harness/server_manager.py">

<violation number="1" location="tests/harness/server_manager.py:228">
P1: HTTP responses from file upload requests are ignored. If any upload fails (e.g., auth error, bad request), the method silently continues to trigger the pipeline, making test failures hard to debug. Consider checking response status with `response.raise_for_status()` or at least logging failures.</violation>
</file>

_{Reply to cubic to teach it or ask questions. Re-run a review with @cubic-dev-ai review this PR}

[cubic-dev-ai]

P2: The modules parameter is silently ignored when feature_flags is None. If a caller expects to use demo feature flags with custom modules (e.g., create_config(modules={"guardrails": False})), the custom modules will be discarded. Consider either documenting this limitation clearly, raising a warning/error when modules is provided without feature_flags, or passing modules to the demo config logic.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At tests/harness/flag_config_manager.py, line 110:

<comment>The `modules` parameter is silently ignored when `feature_flags` is None. If a caller expects to use demo feature flags with custom modules (e.g., `create_config(modules={&quot;guardrails&quot;: False})`), the custom modules will be discarded. Consider either documenting this limitation clearly, raising a warning/error when modules is provided without feature_flags, or passing modules to the demo config logic.</comment>

<file context>
@@ -85,6 +85,36 @@ def restore_env_vars(self) -&gt; None:
+        &quot;&quot;&quot;
+        if feature_flags is None:
+            # Use demo config defaults
+            return self.create_demo_config(dest=dest)
+
+        return self.create_overlay_config(
</file context>

✅ Addressed in 2ef2d0a

[cubic-dev-ai]

P1: HTTP responses from file upload requests are ignored. If any upload fails (e.g., auth error, bad request), the method silently continues to trigger the pipeline, making test failures hard to debug. Consider checking response status with response.raise_for_status() or at least logging failures.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At tests/harness/server_manager.py, line 228:

<comment>HTTP responses from file upload requests are ignored. If any upload fails (e.g., auth error, bad request), the method silently continues to trigger the pipeline, making test failures hard to debug. Consider checking response status with `response.raise_for_status()` or at least logging failures.</comment>

<file context>
@@ -182,6 +182,87 @@ def get_logs(self) -&gt; tuple[str, str]:
+                        else &quot;text/csv&quot;
+                    )
+                    files = {&quot;file&quot;: (Path(file_path).name, content, content_type)}
+                    requests.post(
+                        f&quot;{self.base_url}/inputs/{endpoint}&quot;,
+                        files=files,
</file context>

Suggested change

	requests.post(
	f"{self.base_url}/inputs/{endpoint}",
	files=files,
	headers=headers,
	timeout=30,
	)
	resp = requests.post(
	f"{self.base_url}/inputs/{endpoint}",
	files=files,
	headers=headers,
	timeout=30,
	)
	resp.raise_for_status()

✅ Addressed in 2ef2d0a

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

upload_files omits API token used by server

The new upload_files helper always sends X-API-Key from self.env, but the default ServerManager fixture constructs the instance with an empty env and start() generates a random FIXOPS_API_TOKEN without writing it back to self.env. As a result the helper sends an empty API key, so every call to /inputs/* and /pipeline/run will be rejected by the FastAPI auth guard (see apps/api/app.py lines 245-258) and the E2E tests expecting 200 responses will still fail. The helper needs to pull the token from the environment used to start the server or persist the generated token before sending requests.

Useful? React with 👍 / 👎.

To fix the vulnerability, verify_allowlisted_path should not resolve the input path (which may be user-controlled) in a context-independent manner. Instead, it should, for each root in the allowlist, attempt to construct the candidate path by joining the root to the user-supplied relative path, normalize this candidate path, and then check that this candidate resides within the root directory (relative_to or startswith). Only then should the path be resolved and used. This ensures that absolute paths, symlinks, and directory traversal (e.g. ../../..) cannot escape allowlist roots, even if the user input is maliciously crafted.
Thus, modify the body of verify_allowlisted_path (lines 98–142):

• For each allowlist root, attempt to join root with the user path (as a relative path string), normalize, resolve, and check containment.
• Only accept paths that are contained within an allowlist root after normalization.
• For legacy compatibility, fallback to current logic for backwards API.
  The only file to edit is core/paths.py.

---