Skip to content

osctrl-api: API extensions for a React admin frontend (round 2 of 3)#814

Open
alvarofraguas wants to merge 2 commits into
jmpsec:mainfrom
alvarofraguas:pr/round-2-api-extensions
Open

osctrl-api: API extensions for a React admin frontend (round 2 of 3)#814
alvarofraguas wants to merge 2 commits into
jmpsec:mainfrom
alvarofraguas:pr/round-2-api-extensions

Conversation

@alvarofraguas
Copy link
Copy Markdown

Summary

Round 2 of 3. Adds the API surface a React admin frontend needs to fully cover what the current osctrl-admin templates do. Every new endpoint is additive — no existing routes are removed or repurposed. The legacy admin keeps working unchanged.

End-to-end tested against a Kali docker deployment.

⚠️ Stacked on #813 (security hardening). Merge #813 first, then this. If #813 is merged before review starts here, this branch will be cleanly rebased on the new main HEAD with no conflicts.

New endpoints

Stats / dashboard

Method Path Purpose
GET /api/v1/stats Cross-env summary KPIs
GET /api/v1/stats/osquery-versions Fleet agent versions
GET /api/v1/stats/activity/{env} Env-scoped audit-log activity heatmap
GET /api/v1/stats/activity/node/{env}/{uuid} Per-node activity heatmap
GET /api/v1/stats/activity/node-batch/{env} Per-node heatmap, up to 100 uuids

Logs (live log viewer)

Method Path Purpose
GET /api/v1/logs/{type}/{env}/{uuid} Paginated, since-aware, optional ?q= search

Saved queries (full CRUD)

Method Path
GET /api/v1/saved-queries/{env}
POST /api/v1/saved-queries/{env}
PATCH /api/v1/saved-queries/{env}/{name}
DELETE /api/v1/saved-queries/{env}/{name}

User profile + permissions + token

Method Path
GET /api/v1/users/me
PATCH /api/v1/users/me
POST /api/v1/users/me/password
POST /api/v1/users/{username}/permissions
POST /api/v1/users/{username}/token/refresh
DELETE /api/v1/users/{username}/token

Environment CRUD + config PATCHes

Method Path
POST /api/v1/environments
PATCH /api/v1/environments/{env}
DELETE /api/v1/environments/{env}
GET /api/v1/environments/{env}/config
PATCH /api/v1/environments/{env}/config
PATCH /api/v1/environments/{env}/intervals
PATCH /api/v1/environments/{env}/expiration

Settings PATCH

Method Path
PATCH /api/v1/settings/{service}/{name}

Audit log filters + pagination

Method Path
GET /api/v1/audit-logs?service=&username=&type=&envUuid=&since=&until=&page=&pageSize=

Login envs (pre-auth env list for the SPA's login page env picker)

Method Path
GET /api/v1/login/environments (UUID + name only; everything else stays behind auth)

Sample libraries (operator starter packs)

Method Path
GET /api/v1/queries/samples
GET /api/v1/carves/samples
GET /api/v1/osquery/tables

Pagination, sort, search conventions

Every list endpoint accepts ?page=&page_size= (default 50, max 500) and returns the envelope:

{ "items": [...], "page": N, "page_size": N, "total_items": N, "total_pages": N }

Sortable fields use a per-resource SortableColumns allowlist enforced at the package layer (pkg/nodes, pkg/queries, pkg/carves). Unknown sort keys fall back to the resource's default order without 400ing.

Search is ?q= free-text against a per-resource field set (case-insensitive LIKE). Wildcards are escaped server-side.

New package: pkg/dbutil

Dialect-aware SQL bucket-expression helper (postgres / mysql / sqlite) used by the activity heatmap endpoints. Each category (status logs / result logs / distributed queries / carves) issues a single SQL GROUP BY rather than plucking every timestamp — at 50k+ nodes the table-page heatmap query is bounded by the index instead of the chatty-row count.

Package-layer additions

  • pkg/nodesGetByEnvPaged, NodeView projection, SortableColumns, platform-bucket helpers, GetOsqueryVersionCounts.
  • pkg/queriesGetByEnvTargetPaged, GetSaved* CRUD, SortableColumns, sample-template loader, GetNodeQueryBucketed.
  • pkg/carvesGetByEnvPaged, sample-template loader, GetNodeCarveBucketed.
  • pkg/environmentsCreate / Update / Delete, UpdateConfig / UpdateIntervals / UpdateExpiration helpers.
  • pkg/auditlogGetPaged with PageFilter; GetEnvActivityBucketed for the heatmap.
  • pkg/loggingGetNodeLogs with ?q= search filter, GetNode{Status,Result}Bucketed for the heatmap.
  • pkg/osqueryLoadTables (osquery schema for the SPA query editor).
  • pkg/typesNodeView, paginated response envelopes, EnvCreate / EnvUpdate / EnvConfig* request types, SettingPatchRequest, SavedQueryView, AdminUserView.

Test plan

  • go build ./... — clean
  • go vet ./... — clean
  • go test ./... — all packages pass
  • gofmt -l ./... — empty
  • End-to-end smoke against a Kali docker deployment (paginated lists, activity heatmap, env CRUD, saved-queries CRUD, profile flow)
  • New tests added: cmd/api/handlers/stats_test.go, pkg/nodes/nodes_test.go, pkg/queries/queries_test.go, pkg/queries/saved_test.go

What this enables

Round 3 will land the React admin SPA under a new frontend/ directory at the repo root. The SPA consumes only the endpoints in this PR — no admin-template surface is touched.

@alvarofraguas alvarofraguas force-pushed the pr/round-2-api-extensions branch from c7b8c25 to 75b36e1 Compare May 13, 2026 15:38
@javuto javuto self-assigned this May 13, 2026
@javuto javuto added ✨ enhancement New feature or request osctrl-api osctrl-api related changes ⭐️ frontend Frontend related issues labels May 13, 2026
@alvarofraguas alvarofraguas force-pushed the pr/round-2-api-extensions branch from 75b36e1 to 9daefb3 Compare May 14, 2026 17:17
…, shared rate-limit + audit-log infra

Server-side hardening for osctrl-api, plus shared infrastructure
(rate-limit package, audit-log helpers, trusted-proxies plumbing)
that osctrl-tls also consumes — its consumer-side changes ship in a
companion PR so the TLS-facing surface can be tested in isolation.

== Auth bedrock ==

cmd/api:
  - --auth=jwt is now the default. Refuse to start with --auth=none
    unless OSCTRL_INSECURE_NO_AUTH=1 is set. When opted in, a 60s
    warning ticker keeps the deployment from drifting into
    'auth-off forever'.
  - HttpOnly + Secure cookie session for SPA-style clients
    (osctrl_token). CLI clients with Authorization: Bearer continue
    to work unchanged.
  - Double-submit CSRF (osctrl_csrf cookie + X-CSRF-Token header) for
    mutating cookie-authenticated requests. CLI Bearer flows exempt.
  - JWT signing-algorithm pin (HMAC only) to defeat alg-confusion
    attacks (alg:none / RS256-with-HS256-verify).
  - JWT secret minimum 32 bytes (HS256 needs HMAC key ≥ hash output).
    Startup fails fast with the openssl one-liner if too short.
  - Strict 'forwarded headers' trust via --trusted-proxies. Empty
    default means utils.GetIP ignores X-Forwarded-For / X-Real-IP —
    an internet attacker can't spoof IPs to defeat rate-limits or
    poison audit logs.

== Env secret containment + cross-env defense ==

pkg/types: new TLSEnvironmentView — the low-privilege env projection.
  Omits Secret, EnrollSecretPath, RemoveSecretPath, Certificate, Flags,
  and every other field that materially contributes to enrolling a node.

cmd/api/handlers/environments.go:
  - EnvironmentHandler now branches on access level: AdminLevel (or
    super-admin) gets the full storage struct; UserLevel gets the
    low-priv view.
  - EnvEnrollHandler / EnvRemoveHandler raised from UserLevel to
    AdminLevel — both embed the env's enroll/remove secret.
  - Both handlers log only the target name, not returnData.
  - EnvActionsHandler 'create' branch validates caller-supplied UUID
    via EnvUUIDFilter (rejects malformed) and EnvExists (rejects
    collision). 'delete' branch gets the same validation for symmetry.

cmd/api/handlers/queries.go: QueryResultsHandler now precheck-validates
  the named query belongs to env.ID via h.Queries.Exists(name, env.ID)
  and returns 404 otherwise. logging.GetQueryResults filtered on 'name'
  only, so without this gate a user with QueryLevel on env A could
  pull results from env B by passing B's query name in A's URL.

pkg/environments/environments.go: tighten EnvUUIDFilter regex and add
  axis-pure Exists/UUIDExists helpers so handler checks can match the
  router's expectations exactly.

== Shared rate-limit + audit-log infrastructure ==

pkg/ratelimit (new): per-key token-bucket rate limiter with idle
  eviction. Used by osctrl-api for /login here, and by osctrl-tls for
  /enroll in the companion PR. Tunable burst, window, and key
  function (KeyByIP today; KeyByIPAndEnv available).

pkg/auditlog/audit.go: FailedLogin + FailedEnroll helpers — a clean
  stream of authn/enrol failures for SoC tooling to alert on
  brute-force, password-spray, and enroll abuse.

pkg/utils/http-utils.go: SetTrustedProxies + an updated GetIP that
  honors the trusted-proxies set. Empty (default) ignores
  X-Forwarded-For / X-Real-IP entirely.

== SQL hardening + carve path safety ==

pkg/carves/utils.go: new ValidCarvePath regexp gate. Without this gate
  a CarveLevel operator could pass \`'; SELECT 1; --\` and pivot 'carve
  a file' into 'run any SELECT against your fleet' via GenCarveQuery's
  string concat.

cmd/api/handlers/carves.go (CarvesRunHandler): path validated before
  the SQL splice. Rejected paths return 400.

== Authz + audit-log hardening ==

pkg/users:
  - bcrypt cost raised from default (10) to 12. CheckLoginCredentials
    opportunistically re-hashes existing users at next login (no
    password reset needed). Rehash failure is non-fatal.
  - New ClearToken empties APIToken AND CSRFToken so any existing JWT
    + CSRF cookie pair stops validating. Used by future
    DELETE /api/v1/users/{username}/token in a follow-up PR.

cmd/api/handlers/{users,settings,environments}.go: authz tightenings
  around permission writes, settings PATCH, and env-action service-name
  validation.

pkg/environments/env-cache.go: keep the 2h cleanup interval; introduce
  an envCacheTTL constant so the value is self-documenting and tunable
  locally without changing runtime defaults.

== Defaults + ops ==

deploy/config/{api,admin}.yml: flip --audit-log default to true so
  audit log writes are on by default. Operators can disable with
  --audit-log=false.

Verified: go build ./... clean, go vet ./... clean, go test ./pkg/...
./cmd/api/... ./cmd/tls/... all green.
Round 2 of 3 (round 1: security; round 3: frontend). Adds the API
surface the SPA needs to fully replace the legacy admin templates.
No existing routes are removed or repurposed — every new endpoint is
additive. The new shapes are SPA-canonical (paginated envelope,
projections, typed PATCH bodies).

== New endpoints ==

Stats / dashboard:
  GET /api/v1/stats                                  cross-env summary KPIs
  GET /api/v1/stats/osquery-versions                 fleet agent versions
  GET /api/v1/stats/activity/{env}                   env-scoped audit-log activity heatmap
  GET /api/v1/stats/activity/node/{env}/{uuid}       per-node activity heatmap
  GET /api/v1/stats/activity/node-batch/{env}        per-node heatmap, up to 100 uuids

Logs (live SPA log viewer):
  GET /api/v1/logs/{type}/{env}/{uuid}               paginated, since-aware

Saved queries (full CRUD):
  GET    /api/v1/saved-queries/{env}
  POST   /api/v1/saved-queries/{env}
  PATCH  /api/v1/saved-queries/{env}/{name}
  DELETE /api/v1/saved-queries/{env}/{name}

User profile + token + permissions:
  GET    /api/v1/users/me
  PATCH  /api/v1/users/me
  POST   /api/v1/users/me/password
  POST   /api/v1/users/{username}/permissions
  POST   /api/v1/users/{username}/token/refresh
  DELETE /api/v1/users/{username}/token

Environment CRUD + config PATCHes:
  POST   /api/v1/environments
  PATCH  /api/v1/environments/{env}
  DELETE /api/v1/environments/{env}
  GET    /api/v1/environments/{env}/config
  PATCH  /api/v1/environments/{env}/config
  PATCH  /api/v1/environments/{env}/intervals
  PATCH  /api/v1/environments/{env}/expiration

Settings PATCH:
  PATCH  /api/v1/settings/{service}/{name}

Audit log filters + pagination:
  GET    /api/v1/audit-logs?service=&username=&type=&envUuid=&since=&until=&page=&pageSize=

Login envs (pre-auth env list):
  GET    /api/v1/login/environments                  pre-auth-safe UUID+name only

Sample libraries (operator starter packs):
  GET    /api/v1/queries/samples
  GET    /api/v1/carves/samples
  GET    /api/v1/osquery/tables

== Pagination + sort + search ==

Every list endpoint accepts ?page=&page_size= (default 50, max 500) and
returns the envelope:
  { "items": [...], "page": N, "page_size": N, "total_items": N, "total_pages": N }

Sortable fields use a per-resource SortableColumns allowlist enforced
at the package layer (pkg/nodes, pkg/queries, pkg/carves). Unknown sort
keys fall back to the resource's default order without 400ing.

Search is ?q= free-text against a per-resource field set (case-insensitive
LIKE). Wildcards are escaped server-side.

== New package: pkg/dbutil ==

Dialect-aware SQL bucket-expression helper (postgres / mysql / sqlite)
used by the activity heatmap endpoints. Each category (status logs /
result logs / distributed queries / carves) issues a single SQL
GROUP BY rather than plucking every timestamp — at 50k+ nodes the
table-page heatmap query is bounded by the index instead of the
chatty-row count.

== Package-layer additions ==

  pkg/nodes: GetByEnvPaged, NodeView projection, SortableColumns,
             platform-bucket helpers, GetOsqueryVersionCounts.
  pkg/queries: GetByEnvTargetPaged, GetSaved* CRUD, SortableColumns,
               sample-template loader, GetNodeQueryBucketed.
  pkg/carves: GetByEnvPaged, sample-template loader,
              GetNodeCarveBucketed.
  pkg/environments: Create / Update / Delete, UpdateConfig /
                    UpdateIntervals / UpdateExpiration helpers.
  pkg/auditlog: GetPaged with PageFilter; FailedLogin / FailedEnroll
                hooks; GetEnvActivityBucketed for the heatmap.
  pkg/logging: GetNodeLogs with ?q= search filter,
               GetNode{Status,Result}Bucketed for the heatmap.
  pkg/osquery: LoadTables (osquery schema for the SPA query editor).
  pkg/types: NodeView, paginated response envelopes, EnvCreate /
             EnvUpdate / EnvConfig* request types, SettingPatchRequest,
             SavedQueryView, AdminUserView.

Verified: go build ./... clean, go vet ./... clean, go test ./... all
packages pass. End-to-end tested against a Kali docker deployment.

== What this depends on ==

This PR is stacked on the security-hardening PR (auth bedrock, env
secret containment, TLS-side rate-limit). When that PR is merged
upstream, this branch will be re-targeted at the new main HEAD.

== What this enables ==

A separate round-3 PR will land the React admin SPA under a new
`frontend/` directory at the repo root. The SPA consumes only the
endpoints in this PR — no admin-template surface is touched.
@alvarofraguas alvarofraguas force-pushed the pr/round-2-api-extensions branch from 9daefb3 to b8f83ff Compare May 14, 2026 17:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

✨ enhancement New feature or request ⭐️ frontend Frontend related issues osctrl-api osctrl-api related changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants