SkillFlowGuard

A lightweight workflow-level security auditor for agent skill ecosystems.

SkillFlowGuard detects risks that emerge when individually reasonable skills are composed into a workflow, such as recommendation chains, artifact handoffs, permission escalation, and hidden natural-language coordination signals.

---

Features

• Cross-skill workflow risk detection
• Structured analysis from workflow.json and skill.json
• Lightweight SKILL.md recommendation extraction
• Optional LLM-assisted semantic extraction
• Anthropic and OpenAI-compatible LLM provider support
• Generic JSON adapter for importing external workflow descriptions
• Text, JSON, and HTML reports
• CI-friendly --fail-on exit codes
• Synthetic benchmark with precision / recall / F1
• Pytest tests and GitHub Actions CI

---

Installation

git clone https://github.com/Calvin1989/SkillFlowGuard.git
cd SkillFlowGuard
pip install -e .

For optional LLM support:

pip install -e ".[llm]"

---

Quick Start

Analyze a workflow:

skillflowguard analyze examples/suspicious_chain --extract-doc

JSON output:

skillflowguard analyze examples/suspicious_chain --extract-doc --format json

HTML report:

skillflowguard analyze examples/suspicious_chain --extract-doc --format html --output reports/suspicious.html

CI gate:

skillflowguard analyze examples/suspicious_chain --extract-doc --fail-on high

---

Import External Workflow JSON

Import a generic external workflow description:

skillflowguard import generic-json examples/generic_adapter_input.json --output imported/generic_chain

Analyze the imported workflow:

skillflowguard analyze imported/generic_chain

This adapter converts external workflow descriptions into SkillFlowGuard's internal format:

workflow.json
skills/<skill-name>/skill.json
skills/<skill-name>/SKILL.md

---

Example Output

Summary:
  Findings: 4
  Risk Score: 0.85
  Risk Level: HIGH
  Document Extraction: ON
  LLM Analysis: OFF

Detected Risks:
  [MEDIUM] code-review recommends report-exporter, which appears later in the workflow
  [HIGH] code-review writes [report.json], and report-exporter reads them later
  [HIGH] report-exporter requests network access after code-review used local-only permissions
  [CRITICAL] recommendation + artifact dependency + network access appear in one chain

---

Detection Rules

Rule	Level	Description
cross_skill_recommendation	Medium	A skill recommends another downstream skill.
workspace_anchor_dependency	High	A skill writes an artifact that a later skill reads.
permission_escalation	High	The workflow moves from local-only permissions to network access.
description_permission_mismatch	Medium	A skill claims local/offline behavior but requests network permission.
combined_high_risk_chain	Critical	Recommendation + artifact dependency + network access occur together.

---

Optional LLM Analysis

LLM mode extracts subtle semantic signals from SKILL.md, such as implicit skill pairing or artifact handoff language.

Default Anthropic provider:

skillflowguard analyze examples/subtle_chain --llm

OpenAI-compatible provider:

skillflowguard analyze examples/subtle_chain --llm \
  --llm-provider openai-compatible \
  --llm-base-url <provider-base-url> \
  --llm-model <model-name> \
  --llm-api-key-env <ENV_VAR_NAME>

This supports OpenAI-compatible endpoints such as DeepSeek, Qwen, OpenRouter, LM Studio, and vLLM.

--llm sends SKILL.md content to the configured provider. Do not use it on sensitive documents unless authorized.

---

Evaluation

SkillFlowGuard includes a synthetic benchmark under evaluation/.

python evaluation/run_eval.py --mode structured
python evaluation/run_eval.py --mode extract-doc
python evaluation/run_eval.py --mode llm-mock

Current rule-level results on 21 manually labeled synthetic workflow cases:

Mode	Precision	Recall	F1
structured	1.000	0.558	0.716
extract-doc	1.000	0.750	0.857
llm-mock	1.000	1.000	1.000

llm-mock is deterministic and does not call a real LLM API. The benchmark is synthetic and should not be interpreted as real-world detection performance.

---

Testing

pytest

Current suite:

45 passed

---

Project Structure

skillflowguard/
  adapters/
  loader.py
  doc_parser.py
  llm_doc_parser.py
  rules.py
  analyzer.py
  report.py
  cli.py

examples/
evaluation/
tests/
docs/

---

Roadmap

• v1.0: Stable CLI
• v1.1: Optional LLM-assisted semantic analysis
• v1.2: OpenAI-compatible provider support
• v1.3: Toy evaluation benchmark
• v1.4: Expanded synthetic benchmark
• v1.5: Generic JSON adapter and import command
• v1.6: Real framework adapter, such as LangGraph or AutoGen

---

License

MIT License.