Maintenance/document functions

.github/copilot-instructions.md

@@ -0,0 +1,71 @@
+# Newsletter Maker Project Instructions
+
+You are working in Newsletter Maker, a Django + DRF + Celery + Qdrant backend with a Next.js App Router frontend.
+
+## Repository Shape
+
+- Backend runtime code lives in `core/`.
+- Django project settings and top-level URLs live in `newsletter_maker/`.
+- Backend tests live primarily in `core/tests/` and `tests/`.
+- Frontend application code lives in `frontend/src/app/`, shared UI in `frontend/src/components/`, and shared API/types/helpers in `frontend/src/lib/`.
+- Operational and architecture docs live in `docs/`.
+
+## Working Norms
+
+- Prefer the smallest correct slice of work. Not every request requires both backend and frontend changes.
+- For user-facing product features, assess the full path: model or worker changes, serializer or API changes, frontend types and data access, UI updates, and tests.
+- For admin-only, ingestion-only, worker-only, documentation-only, or settings-only changes, stay in the affected layer. Do not scaffold unnecessary frontend code.
+- Preserve existing naming. This repo uses `project`, not `tenant`.
+
+## Backend Conventions
+
+- Project scoping is a core invariant. Most API resources are nested under `/api/v1/projects/{project_id}/...`.
+- Reuse the established DRF patterns in `core/api.py`, `core/api_urls.py`, and `core/serializers.py`:
+	- `ProjectOwnedQuerysetMixin` for nested viewsets
+	- serializer context containing `project`
+	- explicit validation for cross-project foreign keys
+- Keep viewsets and views thin. Put operational logic in `core/tasks.py`, `core/pipeline.py`, `core/newsletters.py`, `core/plugins/`, or nearby helpers.
+- Preserve existing API field shapes. Backend serializers and frontend types currently use `snake_case`; do not introduce ad hoc `camelCase` transforms.
+- When API behavior changes, update drf-spectacular schema metadata in `core/api.py`.
+- When changing ingestion, newsletter intake, AI processing, or embeddings, preserve the handoff between database state, Celery tasks, and Qdrant state.
+
+## Frontend Conventions
+
+- The frontend uses Next.js App Router.
+- Shared backend-facing types belong in `frontend/src/lib/types.ts`.
+- Shared backend-facing data access belongs in `frontend/src/lib/api.ts` unless there is a clear reason to add a route handler under `frontend/src/app/api/`.
+- Keep reusable UI in `frontend/src/components/` and page assembly in `frontend/src/app/`.
+- Preserve existing backend payload shapes in TypeScript types and UI code unless the backend contract is intentionally changing.
+
+## Documentation Standards
+
+- Python uses Google-style docstrings with PEP 257 conventions.
+- Add or improve module docstrings plus public classes, public functions, and non-obvious helpers.
+- Do not add noisy boilerplate to trivial `__str__` methods, simple properties, or obvious one-line helpers unless the surrounding file genuinely benefits.
+- TypeScript and React code should use JSDoc for exported utilities, hooks, route handlers, and non-trivial components when behavior is not obvious from the type signature alone.
+- If architecture or workflow behavior changes, update the most relevant docs in `docs/`, especially `docs/DEVELOPER_GUIDE.md`, `docs/IMPLEMENTATION_OVERVIEW.md`, `docs/MODELS.md`, `docs/RELEVANCE_SCORING.md`, or `docs/LOGGING.md`.
+
+## Testing And Validation
+
+- Backend tests use `pytest`.
+- Frontend tests use `vitest`.
+- Prefer focused validation commands over full-suite runs when the change is localized.
+- Common commands in this repo:
+	- `pytest core/tests/...`
+	- `python manage.py check`
+	- `just backend-lint`
+	- `cd frontend && npm run test:run`
+	- `cd frontend && npm run typecheck`
+	- `just frontend-lint`
+- Prefer existing `just` tasks when they cover the needed validation flow.
+
+## Skill Usage
+
+Use the workspace skills in `.github/skills/` when they match the task:
+
+- `docstring-enforcer`: documentation passes or doc cleanup across multiple files.
+- `coverage-auditor`: closing backend or frontend test gaps.
+- `bridge-scaffolder`: features that span Django API work and Next.js consumption.
+- `project-api-patterns`: adding or changing project-scoped DRF endpoints.
+- `source-plugin-patterns`: adding or changing ingestion plugins or source-config behavior.
+- `ai-pipeline-patterns`: changing embeddings, relevance scoring, newsletter intake, or Celery-driven AI workflow behavior.

.github/instructions/backend-python.instructions.md

@@ -0,0 +1,37 @@
+---
+name: "Backend Python Guidelines"
+description: "Use when editing Django, DRF, Celery, plugin, management command, or backend test code in Python. Covers project scoping, workflow placement, docstrings, and focused validation for core/, newsletter_maker/, tests/, and manage.py."
+applyTo:
+  - "core/**/*.py"
+  - "newsletter_maker/**/*.py"
+  - "tests/**/*.py"
+  - "manage.py"
+---
+
+# Backend Python Guidelines
+
+- Preserve `project` as the scoping boundary. Do not reintroduce `tenant` naming.
+- Keep Django views and DRF viewsets thin. Put operational logic in nearby helpers such as `core/tasks.py`, `core/pipeline.py`, `core/newsletters.py`, `core/plugins/`, or focused modules next to the owning workflow.
+- For nested API resources, follow the patterns in `core/api.py`, `core/api_urls.py`, and `core/serializers.py`.
+- Enforce cross-project relationship validation in serializers instead of trusting the client.
+- Preserve existing API field names in `snake_case` unless the contract is intentionally changing across backend and frontend.
+- Use Google-style docstrings with PEP 257 conventions for public modules, classes, functions, and non-obvious helpers.
+- Keep changes small and local. Do not create generic `services.py` or `utils.py` files unless the repo already needs that extraction.
+- When changing ingestion, embeddings, newsletter intake, or review behavior, keep the database, Celery, and Qdrant handoff coherent.
+
+## Validation
+
+- Prefer focused checks first:
+  - `pytest core/tests/...`
+  - `python manage.py check`
+  - `just backend-lint`
+- Mock external systems such as Reddit, feed parsing, OpenRouter, email providers, and Qdrant in tests.
+
+## Good Anchors
+
+- `core/models.py`
+- `core/serializers.py`
+- `core/api.py`
+- `core/tasks.py`
+- `core/pipeline.py`
+- `core/newsletters.py`

.github/instructions/documentation.instructions.md

@@ -0,0 +1,36 @@
+---
+name: "Documentation Guidelines"
+description: "Use when editing architecture, onboarding, operations, or product documentation in docs/**/*.md, README.md, or other repo markdown docs. Covers keeping docs aligned with code, preserving project terminology, and updating cross-links when workflows change."
+applyTo:
+  - "docs/**/*.md"
+  - "README.md"
+---
+
+# Documentation Guidelines
+
+- Write docs to match the current codebase, not an aspirational future design, unless the file is explicitly a roadmap or planning document.
+- Use `project`, not `tenant`, unless a historical or comparative note explicitly requires the old term.
+- Prefer concrete file and workflow references over vague architectural summaries.
+- When a behavior changes, update the closest existing document instead of adding a new overlapping explanation.
+- Keep `docs/DEVELOPER_GUIDE.md` current when the best "where to look first" path changes for contributors.
+- Keep `README.md` high-level. Put detailed runtime, workflow, or operator guidance in `docs/` and link to it.
+- When documenting backend behavior, align the wording with the real implementation in files like `core/models.py`, `core/tasks.py`, `core/pipeline.py`, `core/newsletters.py`, `core/api.py`, and `newsletter_maker/settings/`.
+- When documenting frontend behavior, align the wording with the real implementation in `frontend/src/app/`, `frontend/src/components/`, and `frontend/src/lib/`.
+- If a doc mentions commands, prefer the repo's real commands from `justfile`, `package.json`, or `manage.py`.
+- If a code change affects logging, relevance scoring, ingestion, newsletter intake, or onboarding, check whether `docs/LOGGING.md`, `docs/RELEVANCE_SCORING.md`, `docs/IMPLEMENTATION_OVERVIEW.md`, or `docs/DEVELOPER_GUIDE.md` should change too.
+
+## Style
+
+- Favor short sections and direct statements over long narrative paragraphs.
+- Use bullets for operational steps and comparisons.
+- Avoid copying large blocks of code into docs when a file path and a short explanation are enough.
+- Keep terminology consistent across docs, admin UI descriptions, and code comments.
+
+## Good Anchors
+
+- `docs/DEVELOPER_GUIDE.md`
+- `docs/IMPLEMENTATION_OVERVIEW.md`
+- `docs/MODELS.md`
+- `docs/RELEVANCE_SCORING.md`
+- `docs/LOGGING.md`
+- `README.md`

.github/instructions/frontend-app.instructions.md

@@ -0,0 +1,31 @@
+---
+name: "Frontend App Router Guidelines"
+description: "Use when editing Next.js App Router pages, route handlers, shared frontend API helpers, or TypeScript UI code in frontend/src/. Covers file placement, backend contract preservation, typing, and frontend validation."
+applyTo:
+  - "frontend/src/**/*.ts"
+  - "frontend/src/**/*.tsx"
+---
+
+# Frontend App Router Guidelines
+
+- This frontend uses Next.js App Router, not the Pages Router.
+- Keep backend-facing types in `frontend/src/lib/types.ts` and shared server-side API access in `frontend/src/lib/api.ts` unless a route handler in `frontend/src/app/api/` is the correct boundary.
+- Reuse the existing backend contract. This repo currently consumes `snake_case` fields from Django; do not silently rename payload keys in the frontend.
+- Keep reusable UI in `frontend/src/components/` and page composition in `frontend/src/app/`.
+- Prefer strong explicit types over loose `Record<string, unknown>` shapes when the contract is known.
+- Add JSDoc for exported utilities, route handlers, hooks, and non-trivial components when behavior is not obvious from the signature.
+- When a change depends on new backend fields or endpoints, update the corresponding types and API helpers in the same change.
+
+## Validation
+
+- Prefer focused checks first:
+  - `cd frontend && npm run test:run`
+  - `cd frontend && npm run typecheck`
+  - `just frontend-lint`
+
+## Good Anchors
+
+- `frontend/src/lib/types.ts`
+- `frontend/src/lib/api.ts`
+- `frontend/src/app/`
+- `frontend/src/components/`

.github/skills/ai-pipeline-patterns/SKILL.md

@@ -0,0 +1,53 @@
+---
+name: ai-pipeline-patterns
+description: "Use when changing the AI workflow, embeddings, Qdrant integration, newsletter intake, relevance scoring, summarization, classification, Celery handoff, or Anymail inbound processing. Trigger phrases include pipeline, relevance scoring, embeddings, Qdrant, newsletter intake, summarization, classification, OpenRouter, and review queue."
+---
+
+# AI Pipeline Patterns Skill
+
+Use this skill when working on the ingestion and AI processing pipeline.
+
+## Scope
+
+This skill covers the interaction between:
+
+- `core/pipeline.py`
+- `core/embeddings.py`
+- `core/tasks.py`
+- `core/newsletters.py`
+- `core/newsletter_extraction.py`
+- `core/signals.py`
+- `core/views.py`
+- related models, tests, and docs
+
+## Rules
+
+- Preserve the separation between persisted state, background execution, and vector storage.
+- Keep Qdrant access inside the embedding layer unless there is a strong reason to expand that boundary.
+- Keep newsletter intake routing and confirmation logic in the newsletter intake modules, not in unrelated views or serializers.
+- Prefer explicit fallbacks when LLM or external-service calls can fail.
+- Respect the current thresholds and routing semantics around relevance scoring, summarization eligibility, and review-queue creation unless the change intentionally redefines them.
+- When changing provider behavior, preserve provider-agnostic boundaries where they already exist, such as Anymail for inbound and Django mail abstractions for outbound.
+- Update docs when pipeline behavior changes in a way a new developer or operator would need to understand.
+
+## Implementation Guidance
+
+- For embedding or vector-search changes, start in `core/embeddings.py` and then verify the downstream callers in `core/pipeline.py` and `core/tasks.py`.
+- For newsletter intake changes, start in `core/newsletters.py` or `core/signals.py` and follow the path through `NewsletterIntake`, `IntakeAllowlist`, and `process_newsletter_intake`.
+- For project routing or user-facing confirmation behavior, verify both `core/views.py` and top-level URL registration.
+- For AI-skill output changes, check how `SkillResult` rows are persisted and how admin or frontend surfaces consume them.
+
+## Validation
+
+- Prefer focused checks first:
+  - `pytest core/tests/test_pipeline.py`
+  - `pytest core/tests/test_tasks.py`
+  - `pytest core/tests/test_newsletters.py`
+  - `python manage.py check`
+- If a change affects serializers, admin, or API surfaces too, expand validation to the nearest related test module.
+
+## References
+
+- `docs/DEVELOPER_GUIDE.md`
+- `docs/RELEVANCE_SCORING.md`
+- `docs/LOGGING.md`

.github/skills/bridge-scaffolder/SKILL.md

@@ -0,0 +1,28 @@
+---
+name: bridge-scaffolder
+description: "Use when creating or changing a feature that spans both the Django API and the Next.js frontend. Trigger phrases include full-stack feature, add endpoint and UI, wire frontend to backend, project dashboard change, and bridge serializer to frontend types."
+---
+
+# Bridge Scaffolder Skill
+
+Use this skill when a change genuinely crosses the backend and frontend boundary.
+
+## Rules
+
+- **Django Side:** Follow the existing patterns in `core/api.py`, `core/api_urls.py`, and `core/serializers.py`.
+- Most nested resources should stay project-scoped under `/api/v1/projects/{project_id}/...`.
+- Keep business logic out of viewsets. Use `core/tasks.py`, `core/pipeline.py`, `core/newsletters.py`, `core/plugins/`, or nearby helpers for real workflow logic.
+- **Next.js Side:** Update `frontend/src/lib/types.ts`, `frontend/src/lib/api.ts`, and the relevant pages, components, or route handlers under `frontend/src/app/`.
+- Preserve the existing `snake_case` payload shape unless the backend contract is intentionally changing.
+
+## Implementation Guidance
+
+- Check `core/api_urls.py` for the current route topology and `core/api.py` for the schema helper patterns.
+- Keep serializer validation aligned with project scoping and cross-project relationship rules.
+- If the frontend consumes the new field or endpoint, reflect it in `frontend/src/lib/types.ts` and the corresponding API helpers.
+- Update docs when the feature changes a core workflow or user-facing behavior.
+
+## Related Guidance
+
+- Use `project-api-patterns` when the backend portion is primarily a new or changed DRF resource.
+- Use `coverage-auditor` immediately after scaffolding to add targeted backend and frontend tests.

.github/skills/coverage-auditor/SKILL.md

@@ -0,0 +1,39 @@
+---
+name: coverage-auditor
+description: "Use when adding or updating tests for Django, DRF, Celery, admin, serializer, plugin, or Next.js code, or when closing a coverage gap. Trigger phrases include add tests, improve coverage, pytest, vitest, missing branch, serializer test, admin test, and route handler test."
+---
+
+# Coverage Auditor Skill
+
+Use this skill to add the smallest effective tests around the changed behavior.
+
+## Rules
+
+- **Backend:** Use `pytest` with the existing patterns in `core/tests/` and `tests/`.
+- **Frontend:** Use `vitest` and the existing co-located `__tests__` structure under `frontend/src/`.
+- Mock external integrations such as feed parsing, Reddit, email providers, Qdrant, or LLM calls instead of relying on live services.
+- Prefer focused behavior tests over broad snapshot-style tests.
+- For defensive branches that are intentionally unreachable, document exclusions explicitly with `# pragma: no cover` or `/* v8 ignore next */` only when justified.
+
+## Testing Guidance
+
+- Analyze the changed file first and cover the real branch points: validation failures, access control, empty states, duplicate handling, service failures, and success paths.
+- For backend work, add tests near the closest existing module, such as:
+	- `core/tests/test_serializers.py`
+	- `core/tests/test_admin.py`
+	- `core/tests/test_tasks.py`
+	- `core/tests/test_newsletters.py`
+	- `core/tests/test_pipeline.py`
+- For frontend work, extend the nearest existing `__tests__` file under `frontend/src/components/` or `frontend/src/lib/`.
+- When adding new frontend tests, keep imports sorted to satisfy the repo's ESLint import-order rules. If you hit `Run autofix to sort these imports!`, fix the import block or run file-scoped ESLint before moving on.
+- After changing tests, run the narrowest relevant validation command first.
+- For Provider/Wrapper components, do not test state or initialization. Instead, render a consumer child component and assert that the context values or library configurations (like QueryClient options) match the expected project defaults.
+- For Server Components, prioritize unit testing exported helper functions (like deriveSourceStatus) for logic, and use integration tests to verify that API data is mapped correctly to the UI components (badges, tables, etc.). Mock all library API calls using vi.mock.
+- Out of Scope: Root layouts, static metadata, font configurations, and pure "pass-through" providers.
+- In Scope: Layouts with conditional rendering, breadcrumb logic, or role-based access checks.
+
+## References
+
+- Backend commands are captured in `justfile` as `just backend-test`, `just backend-test-coverage`, and `just backend-lint`.
+- Frontend commands are captured in `frontend/package.json` and the repo `justfile` as `npm run test:run`, `npm run test:coverage`, and `just frontend-lint`.
+- Use `python manage.py check` after Django-side structural changes.

.github/skills/docstring-enforcer/SKILL.md

@@ -0,0 +1,37 @@
+---
+name: docstring-enforcer
+description: "Use when adding or revising documentation for Python modules/Django code or exported TypeScript/React utilities. Triggers: docstring, JSDoc, document this file, lib utilities, improve docs, and explain intent."
+---
+
+# Docstring Enforcer Skill
+
+Use this skill for meaningful documentation that explains *why* code exists and *how* it handles edge cases.
+
+## Rules
+
+### 🐍 Python (Google Style)
+
+- **Standard:** Use Google-style docstrings and PEP 257.
+- **Structure:** Include `Args:`, `Returns:`, and `Raises:` only when they provide value. Do not add empty sections.
+- **Django Specifics:** Document the "why" behind complex QuerySets or signal receivers. Trivial dunder methods or obvious model fields can remain undocumented if self-explanatory.
+- **Intent:** Favor workflow context over repeating the function name (e.g., "Invalidates the cache after user logout" vs "Does logout stuff").
+
+### ⚛️ TypeScript & Next.js (JSDoc Style)
+
+- **Standard:** Use JSDoc for exported utilities, hooks, and complex components.
+- **Frontend `lib/` Policy:** ALL shared utilities in `lib/` must include:
+    - **@example:** A brief code snippet showing typical usage.
+    - **Edge Cases:** Describe behavior for `null`, `undefined`, or empty strings in `@param` or `@returns`.
+- **Clarity:** Do not just restate TypeScript types. Explain constraints (e.g., "The string must be a valid ISO-8601 date").
+
+## References
+
+- **Gold Standard (Python):** See `core/models.py` and `core/pipeline.py`.
+- **Gold Standard (TS/lib):** See `frontend/src/lib/formatters.ts` (if applicable) for example-driven JSDoc.
+- **Context:** Check `docs/DEVELOPER_GUIDE.md` to ensure docs align with overall system architecture.
+
+## Workflow
+
+1. Analyze the file to understand its role in the Django/Next.js bridge.
+2. If documentation is missing or outdated, rewrite it using the styles above.
+3. Ensure that if a Django API field changes, the corresponding Next.js JSDoc for that field is also flagged for an update.