osctrl-api: API extensions for a React admin frontend (round 2 of 3)#814
Open
alvarofraguas wants to merge 2 commits into
Open
osctrl-api: API extensions for a React admin frontend (round 2 of 3)#814alvarofraguas wants to merge 2 commits into
alvarofraguas wants to merge 2 commits into
Conversation
c7b8c25 to
75b36e1
Compare
5 tasks
75b36e1 to
9daefb3
Compare
…, shared rate-limit + audit-log infra
Server-side hardening for osctrl-api, plus shared infrastructure
(rate-limit package, audit-log helpers, trusted-proxies plumbing)
that osctrl-tls also consumes — its consumer-side changes ship in a
companion PR so the TLS-facing surface can be tested in isolation.
== Auth bedrock ==
cmd/api:
- --auth=jwt is now the default. Refuse to start with --auth=none
unless OSCTRL_INSECURE_NO_AUTH=1 is set. When opted in, a 60s
warning ticker keeps the deployment from drifting into
'auth-off forever'.
- HttpOnly + Secure cookie session for SPA-style clients
(osctrl_token). CLI clients with Authorization: Bearer continue
to work unchanged.
- Double-submit CSRF (osctrl_csrf cookie + X-CSRF-Token header) for
mutating cookie-authenticated requests. CLI Bearer flows exempt.
- JWT signing-algorithm pin (HMAC only) to defeat alg-confusion
attacks (alg:none / RS256-with-HS256-verify).
- JWT secret minimum 32 bytes (HS256 needs HMAC key ≥ hash output).
Startup fails fast with the openssl one-liner if too short.
- Strict 'forwarded headers' trust via --trusted-proxies. Empty
default means utils.GetIP ignores X-Forwarded-For / X-Real-IP —
an internet attacker can't spoof IPs to defeat rate-limits or
poison audit logs.
== Env secret containment + cross-env defense ==
pkg/types: new TLSEnvironmentView — the low-privilege env projection.
Omits Secret, EnrollSecretPath, RemoveSecretPath, Certificate, Flags,
and every other field that materially contributes to enrolling a node.
cmd/api/handlers/environments.go:
- EnvironmentHandler now branches on access level: AdminLevel (or
super-admin) gets the full storage struct; UserLevel gets the
low-priv view.
- EnvEnrollHandler / EnvRemoveHandler raised from UserLevel to
AdminLevel — both embed the env's enroll/remove secret.
- Both handlers log only the target name, not returnData.
- EnvActionsHandler 'create' branch validates caller-supplied UUID
via EnvUUIDFilter (rejects malformed) and EnvExists (rejects
collision). 'delete' branch gets the same validation for symmetry.
cmd/api/handlers/queries.go: QueryResultsHandler now precheck-validates
the named query belongs to env.ID via h.Queries.Exists(name, env.ID)
and returns 404 otherwise. logging.GetQueryResults filtered on 'name'
only, so without this gate a user with QueryLevel on env A could
pull results from env B by passing B's query name in A's URL.
pkg/environments/environments.go: tighten EnvUUIDFilter regex and add
axis-pure Exists/UUIDExists helpers so handler checks can match the
router's expectations exactly.
== Shared rate-limit + audit-log infrastructure ==
pkg/ratelimit (new): per-key token-bucket rate limiter with idle
eviction. Used by osctrl-api for /login here, and by osctrl-tls for
/enroll in the companion PR. Tunable burst, window, and key
function (KeyByIP today; KeyByIPAndEnv available).
pkg/auditlog/audit.go: FailedLogin + FailedEnroll helpers — a clean
stream of authn/enrol failures for SoC tooling to alert on
brute-force, password-spray, and enroll abuse.
pkg/utils/http-utils.go: SetTrustedProxies + an updated GetIP that
honors the trusted-proxies set. Empty (default) ignores
X-Forwarded-For / X-Real-IP entirely.
== SQL hardening + carve path safety ==
pkg/carves/utils.go: new ValidCarvePath regexp gate. Without this gate
a CarveLevel operator could pass \`'; SELECT 1; --\` and pivot 'carve
a file' into 'run any SELECT against your fleet' via GenCarveQuery's
string concat.
cmd/api/handlers/carves.go (CarvesRunHandler): path validated before
the SQL splice. Rejected paths return 400.
== Authz + audit-log hardening ==
pkg/users:
- bcrypt cost raised from default (10) to 12. CheckLoginCredentials
opportunistically re-hashes existing users at next login (no
password reset needed). Rehash failure is non-fatal.
- New ClearToken empties APIToken AND CSRFToken so any existing JWT
+ CSRF cookie pair stops validating. Used by future
DELETE /api/v1/users/{username}/token in a follow-up PR.
cmd/api/handlers/{users,settings,environments}.go: authz tightenings
around permission writes, settings PATCH, and env-action service-name
validation.
pkg/environments/env-cache.go: keep the 2h cleanup interval; introduce
an envCacheTTL constant so the value is self-documenting and tunable
locally without changing runtime defaults.
== Defaults + ops ==
deploy/config/{api,admin}.yml: flip --audit-log default to true so
audit log writes are on by default. Operators can disable with
--audit-log=false.
Verified: go build ./... clean, go vet ./... clean, go test ./pkg/...
./cmd/api/... ./cmd/tls/... all green.
Round 2 of 3 (round 1: security; round 3: frontend). Adds the API
surface the SPA needs to fully replace the legacy admin templates.
No existing routes are removed or repurposed — every new endpoint is
additive. The new shapes are SPA-canonical (paginated envelope,
projections, typed PATCH bodies).
== New endpoints ==
Stats / dashboard:
GET /api/v1/stats cross-env summary KPIs
GET /api/v1/stats/osquery-versions fleet agent versions
GET /api/v1/stats/activity/{env} env-scoped audit-log activity heatmap
GET /api/v1/stats/activity/node/{env}/{uuid} per-node activity heatmap
GET /api/v1/stats/activity/node-batch/{env} per-node heatmap, up to 100 uuids
Logs (live SPA log viewer):
GET /api/v1/logs/{type}/{env}/{uuid} paginated, since-aware
Saved queries (full CRUD):
GET /api/v1/saved-queries/{env}
POST /api/v1/saved-queries/{env}
PATCH /api/v1/saved-queries/{env}/{name}
DELETE /api/v1/saved-queries/{env}/{name}
User profile + token + permissions:
GET /api/v1/users/me
PATCH /api/v1/users/me
POST /api/v1/users/me/password
POST /api/v1/users/{username}/permissions
POST /api/v1/users/{username}/token/refresh
DELETE /api/v1/users/{username}/token
Environment CRUD + config PATCHes:
POST /api/v1/environments
PATCH /api/v1/environments/{env}
DELETE /api/v1/environments/{env}
GET /api/v1/environments/{env}/config
PATCH /api/v1/environments/{env}/config
PATCH /api/v1/environments/{env}/intervals
PATCH /api/v1/environments/{env}/expiration
Settings PATCH:
PATCH /api/v1/settings/{service}/{name}
Audit log filters + pagination:
GET /api/v1/audit-logs?service=&username=&type=&envUuid=&since=&until=&page=&pageSize=
Login envs (pre-auth env list):
GET /api/v1/login/environments pre-auth-safe UUID+name only
Sample libraries (operator starter packs):
GET /api/v1/queries/samples
GET /api/v1/carves/samples
GET /api/v1/osquery/tables
== Pagination + sort + search ==
Every list endpoint accepts ?page=&page_size= (default 50, max 500) and
returns the envelope:
{ "items": [...], "page": N, "page_size": N, "total_items": N, "total_pages": N }
Sortable fields use a per-resource SortableColumns allowlist enforced
at the package layer (pkg/nodes, pkg/queries, pkg/carves). Unknown sort
keys fall back to the resource's default order without 400ing.
Search is ?q= free-text against a per-resource field set (case-insensitive
LIKE). Wildcards are escaped server-side.
== New package: pkg/dbutil ==
Dialect-aware SQL bucket-expression helper (postgres / mysql / sqlite)
used by the activity heatmap endpoints. Each category (status logs /
result logs / distributed queries / carves) issues a single SQL
GROUP BY rather than plucking every timestamp — at 50k+ nodes the
table-page heatmap query is bounded by the index instead of the
chatty-row count.
== Package-layer additions ==
pkg/nodes: GetByEnvPaged, NodeView projection, SortableColumns,
platform-bucket helpers, GetOsqueryVersionCounts.
pkg/queries: GetByEnvTargetPaged, GetSaved* CRUD, SortableColumns,
sample-template loader, GetNodeQueryBucketed.
pkg/carves: GetByEnvPaged, sample-template loader,
GetNodeCarveBucketed.
pkg/environments: Create / Update / Delete, UpdateConfig /
UpdateIntervals / UpdateExpiration helpers.
pkg/auditlog: GetPaged with PageFilter; FailedLogin / FailedEnroll
hooks; GetEnvActivityBucketed for the heatmap.
pkg/logging: GetNodeLogs with ?q= search filter,
GetNode{Status,Result}Bucketed for the heatmap.
pkg/osquery: LoadTables (osquery schema for the SPA query editor).
pkg/types: NodeView, paginated response envelopes, EnvCreate /
EnvUpdate / EnvConfig* request types, SettingPatchRequest,
SavedQueryView, AdminUserView.
Verified: go build ./... clean, go vet ./... clean, go test ./... all
packages pass. End-to-end tested against a Kali docker deployment.
== What this depends on ==
This PR is stacked on the security-hardening PR (auth bedrock, env
secret containment, TLS-side rate-limit). When that PR is merged
upstream, this branch will be re-targeted at the new main HEAD.
== What this enables ==
A separate round-3 PR will land the React admin SPA under a new
`frontend/` directory at the repo root. The SPA consumes only the
endpoints in this PR — no admin-template surface is touched.
9daefb3 to
b8f83ff
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Round 2 of 3. Adds the API surface a React admin frontend needs to fully cover what the current
osctrl-admintemplates do. Every new endpoint is additive — no existing routes are removed or repurposed. The legacy admin keeps working unchanged.End-to-end tested against a Kali docker deployment.
mainHEAD with no conflicts.New endpoints
Stats / dashboard
/api/v1/stats/api/v1/stats/osquery-versions/api/v1/stats/activity/{env}/api/v1/stats/activity/node/{env}/{uuid}/api/v1/stats/activity/node-batch/{env}Logs (live log viewer)
/api/v1/logs/{type}/{env}/{uuid}?q=searchSaved queries (full CRUD)
/api/v1/saved-queries/{env}/api/v1/saved-queries/{env}/api/v1/saved-queries/{env}/{name}/api/v1/saved-queries/{env}/{name}User profile + permissions + token
/api/v1/users/me/api/v1/users/me/api/v1/users/me/password/api/v1/users/{username}/permissions/api/v1/users/{username}/token/refresh/api/v1/users/{username}/tokenEnvironment CRUD + config PATCHes
/api/v1/environments/api/v1/environments/{env}/api/v1/environments/{env}/api/v1/environments/{env}/config/api/v1/environments/{env}/config/api/v1/environments/{env}/intervals/api/v1/environments/{env}/expirationSettings PATCH
/api/v1/settings/{service}/{name}Audit log filters + pagination
/api/v1/audit-logs?service=&username=&type=&envUuid=&since=&until=&page=&pageSize=Login envs (pre-auth env list for the SPA's login page env picker)
/api/v1/login/environments(UUID + name only; everything else stays behind auth)Sample libraries (operator starter packs)
/api/v1/queries/samples/api/v1/carves/samples/api/v1/osquery/tablesPagination, sort, search conventions
Every list endpoint accepts
?page=&page_size=(default 50, max 500) and returns the envelope:{ "items": [...], "page": N, "page_size": N, "total_items": N, "total_pages": N }Sortable fields use a per-resource
SortableColumnsallowlist enforced at the package layer (pkg/nodes,pkg/queries,pkg/carves). Unknown sort keys fall back to the resource's default order without 400ing.Search is
?q=free-text against a per-resource field set (case-insensitive LIKE). Wildcards are escaped server-side.New package:
pkg/dbutilDialect-aware SQL bucket-expression helper (postgres / mysql / sqlite) used by the activity heatmap endpoints. Each category (status logs / result logs / distributed queries / carves) issues a single SQL
GROUP BYrather than plucking every timestamp — at 50k+ nodes the table-page heatmap query is bounded by the index instead of the chatty-row count.Package-layer additions
pkg/nodes—GetByEnvPaged,NodeViewprojection,SortableColumns, platform-bucket helpers,GetOsqueryVersionCounts.pkg/queries—GetByEnvTargetPaged,GetSaved*CRUD,SortableColumns, sample-template loader,GetNodeQueryBucketed.pkg/carves—GetByEnvPaged, sample-template loader,GetNodeCarveBucketed.pkg/environments—Create/Update/Delete,UpdateConfig/UpdateIntervals/UpdateExpirationhelpers.pkg/auditlog—GetPagedwithPageFilter;GetEnvActivityBucketedfor the heatmap.pkg/logging—GetNodeLogswith?q=search filter,GetNode{Status,Result}Bucketedfor the heatmap.pkg/osquery—LoadTables(osquery schema for the SPA query editor).pkg/types—NodeView, paginated response envelopes,EnvCreate/EnvUpdate/EnvConfig*request types,SettingPatchRequest,SavedQueryView,AdminUserView.Test plan
go build ./...— cleango vet ./...— cleango test ./...— all packages passgofmt -l ./...— emptycmd/api/handlers/stats_test.go,pkg/nodes/nodes_test.go,pkg/queries/queries_test.go,pkg/queries/saved_test.goWhat this enables
Round 3 will land the React admin SPA under a new
frontend/directory at the repo root. The SPA consumes only the endpoints in this PR — no admin-template surface is touched.