theo explorations#6366
Draft
theosanderson-agent wants to merge 11 commits into
Draft
Conversation
…f LAPIS
Adds a new single-deployment service (query-service/) that proxies
/{organism}/{path} to the corresponding loculus-lapis-service-{organism}.
For now it is a transparent passthrough; the point is to give us a
single hop where we can rewrite LAPIS responses in future iterations
without changing the website or LAPIS.
Wires the website (server-side lapisUrls), the lapis ingress, and the
public lapisUrls (used by the browser and CLI) through the new
service, so all external and internal LAPIS calls now go through it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
claude
Bot
added
the
deployment
Contributor
|
This PR may be related to: #6265 The query-service introduced here acts as a central reverse proxy routing |
The query-service is now the only LAPIS-facing surface. The old
`/{organism}/{path}` passthrough is removed.
API shape (see query-service/README.md):
- Owned verbs under /v1/: aggregated, details, mutations, aaMutations,
insertions, aaInsertions, alignedSequences[/segment],
unalignedSequences[/segment], aaSequences/<protein>, info,
lineageDefinition.
- `?organism=` is required and single-valued.
- Reserved control keywords: organism, format (-> dataFormat),
download (-> downloadAsFile), fields, limit, offset, include,
reference. Anything else is a metadata-column filter.
- Implicit defaults applied centrally: versionStatus=LATEST_VERSION,
isRevocation=false. Override with `include=revoked|older-versions|all`.
Explicit version filters drop the defaults.
Helm:
- Removed `lapisUrls` (per-organism map) from the website runtime config;
replaced with a single `queryServiceUrl` plus an `organisms` list.
- lapis-ingress is now a single rule that routes everything on the
lapis hostname to query-service.
Website:
- Renamed Zodios endpoints to /v1/<verb>; segment / proteinName are
path components (Zodios needs unique paths per endpoint).
- LapisClient now takes (queryServiceUrl, organism, schema) and adds
`?organism=` to every call. Internal name unchanged to limit churn.
- DownloadUrlGenerator builds /v1/... URLs and adds ?organism=.
- Dropped manual versionStatus / isRevocation defaults from server-side
callers (GroupPage, getSeqSetStatistics, getOrganismStatistics) — the
query-service applies them.
- /loculus-info now exposes hosts.queryService instead of hosts.lapis.
CLI:
- Migrated to /v1/ paths with ?organism=.
- get_lapis_url() retained as a thin wrapper that returns the
query-service base URL.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… POSTs
Adds the missing pieces to keep the prior search-page UX (where users
can clear the version-status / revocation hidden-field defaults to see
all versions) working against the new query-service defaults.
- lapisApi.ts: every /v1/ endpoint now accepts an optional `include`
query parameter.
- serviceHooks.lapisClientHooks: takes an `options.include` and threads
it onto every request as a query string. The search UI passes
`'all'`; autocomplete and detail views leave it unset so query-service
defaults still apply there.
- SearchFullUI / serversideSearch / DownloadUrlGenerator: pass
`include=all`, since the search page manages its own version
defaults via hiddenFieldValues.
- LapisClient.getAllSequenceEntryHistoryForAccession: passes
`include=all` (version history needs every version + revocation).
- query-service:
- reads `include=` from the query string for POSTs too (was only
looking in the body), so the website's `?include=all` is honoured.
- parses `application/x-www-form-urlencoded` bodies for the long-query
download path that submits an HTML form. Uses `getlist` so repeated
keys (`fields=a&fields=b&fields=c`) are preserved as a list, not
collapsed to the first value.
- Adds python-multipart to requirements (Starlette's form parser
dependency).
- Updates DownloadDialog.spec.tsx assertions for the new
`?organism=ebola&include=all&...` URL prefix.
- Fixes one stale `/sample/details` reference in
download.dependent.spec.ts.
Integration suite: 98/98 passing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Add .github/workflows/query-service-image.yml so the integration tests can pull commit-tagged images of the query-service. Mirrors preprocessing-dummy-image.yml: hash-based caching, multi-tag pushes, ARM build on main. - Run prettier --write across files touched by the migration. - Restore the versionStatus / isRevocation regex segments to the DownloadDialog spec assertions that exercise hiddenFieldValues — those filters _are_ sent in that flow (the test passes them explicitly), and the implicit defaults stay opt-out via include=all. - Update vitest.setup.ts MSW mocks to point at the new sequence paths: /v1/alignedSequences[/segment], /v1/unalignedSequences[/segment]. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…e-all toggle Following up on the previous commit so the website stops shipping the versionStatus / isRevocation hidden defaults that the query-service is already applying: - Remove the `hiddenFieldValues` containing `versionStatus=LATEST_VERSION` and `isRevocation=false` from the search and submission/released pages. The query-service applies those defaults centrally — no need to also send them on the wire from the website. - Drop the blanket `?include=all` from `lapisClientHooks` / `serversideSearch` / `DownloadUrlGenerator`. Default search now relies on query-service's defaults (latest non-revoked). - Add an explicit "Include older versions and revocations" checkbox at the top of the search form. It writes `?include=all` into the URL, which the website then forwards as the `include=` query param. Toggle off and the URL drops the param so defaults reapply. - Update the override-hidden-fields integration test to flip the new toggle instead of clearing the (no longer present) hidden fields. - Replace the "hidden field values are kept in URL params" unit test with one that asserts the new `include=` toggle round-trips through the URL. - Bump query-service CPU request/limit a touch (200m / 2 cores). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… include=all
- SearchFullUI: stabilise the empty `hiddenFieldValues` default with
`useMemo`. The previous `??= {}` reassigned a fresh object every
render, which destabilised the `useMemo` chain feeding the
search-results `useEffect` and caused it to refire — the user observed
this as an infinite loop of /v1/aggregated requests on the search page.
- /[organism]/submission/[groupId]/released: default `?include=all` so
submitters see every version they've released (including revocations
they've just made). The `mySequencesPage` route helper builds the
link with that param and the page itself injects it into
`initialQueryDict` for direct navigation.
… state The React URL-syncing hook overwrites SSR-injected state with whatever is actually in window.location on hydration, so injecting include=all into initialQueryDict alone gets clobbered. Redirect server-side instead so the URL itself is the source of truth, and hydration sees include=all.
…do /released redirect
- The 'override hidden fields' test was specifically exercising the
removed clear-the-hidden-default mechanism. The replacement coverage
is in search.dependent.spec.ts ('include-all toggle puts include=all
in the URL'); the autocomplete-timeout flake on the deleted test was
not adding signal.
- Reverting the server-side redirect to ?include=all on /released:
some tests (e.g. file-sharing 'bulk revise 2 seqs with files') do
page.goto(page.url() + '?column_submissionId=true') after navigating
to /released, and the redirect made page.url() already carry an
?include=all so the appended ?column_... corrupted the URL into
?include=all?column_submissionId=true and query-service rejected the
malformed include= value with 400 (which then tripped the
console-warnings fixture).
The link helper (routes.mySequencesPage) still adds ?include=all so
the in-page 'released sequences' link gives submitters the
see-everything view they need; tests that navigate directly retain
default behaviour and can append params with ? safely.
theosanderson
added
preview
Conflicts resolved: - integration-tests/.../override-hidden-fields.spec.ts — kept the branch's deletion (the hidden-default override UX no longer exists; see search.dependent.spec.ts's 'include-all toggle' test). - website/src/components/SearchPage/SearchForm.tsx — main introduced a multi-field-search variant in the Metadata Filters loop; reapplied the branch's organism prop on the SearchField inside it. - website/src/components/SearchPage/fields/HierarchicalField.spec.tsx — main added this new spec; threaded the required `organism` prop into every <HierarchicalField> render.
theosanderson
added
preview
The new query-service exposes /v1/aggregated rather than
/{organism}/sample/aggregated, so multi-field-search interceptors were
matching nothing and observing undefined status.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Inserts a new
query-servicebetween the website / CLI / external callersand the per-organism LAPIS deployments, and migrates everyone onto it.
Implementation
query-service/— Python (FastAPI + httpx) reverse proxy, singledeployment, one image. Routes
/v1/<verb>?organism=<x>to the rightloculus-lapis-service-<x>. Owns its own verb names (/v1/aggregated,/v1/details,/v1/mutations,/v1/sequences, …) and reserved controlparams (
organism,format,download,fields,limit,offset,include). Accepts both JSON and form-encoded POST bodies.LapisClient,DownloadUrlGenerator,lapisClientHooks, all/loculus-infocallers migrated. Adds an"Include older versions and revocations" checkbox at the top of the
search filter panel, writing
?include=allto the URL.Behaviour changes
versionStatus=LATEST_VERSIONandisRevocation=falseserver-sideunless the caller opts out. The website's hidden-default fields for
these are gone; the CLI no longer silently shows revoked / older
versions.
?include=allon the API,matched in the UI by the new toggle.
?include=revoked/?include=older-versionsare also available. ExplicitversionStatus=/version=/accessionVersion=always wins.?include=all— when you just made a revocation or revision, it'svisible without flipping anything.
lapis.<host>/<organism>/sample/aggregatedis now<host>/v1/aggregated?organism=<organism>(verbs renamed, organismmoved from path to query param). Filter columns and response shapes
are unchanged.
User-facing writeup: https://gist.github.com/theosanderson-agent/b1e5a7375b865949cc6e3e0be9d1754d
🚀 Preview: Add
previewlabel to enable