Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -76,12 +76,13 @@ DJANGO_SUPERUSER_USERNAME=admin
DJANGO_SUPERUSER_EMAIL=admin@example.com
DJANGO_SUPERUSER_PASSWORD=adminpass

NEWSLETTER_API_BASE_URL=http://nginx
NEWSLETTER_API_INTERNAL_URL=http://127.0.0.1:8080
NEWSLETTER_PUBLIC_URL=http://127.0.0.1:8080
NEWSLETTER_API_USERNAME=admin
NEWSLETTER_API_PASSWORD=adminpass

DEBUG=True

ALLOWED_HOSTS=localhost,127.0.0.1,newslettermaker.tech
ALLOWED_HOSTS=localhost,127.0.0.1,nginx,newslettermaker.tech

FRONTEND_URL=http://localhost:3000
3 changes: 3 additions & 0 deletions .env.test
Original file line number Diff line number Diff line change
Expand Up @@ -8,3 +8,6 @@ REDDIT_CLIENT_SECRET=secret
REDDIT_USER_AGENT=newsletter-maker/test
CELERY_BROKER_URL=memory://
CELERY_RESULT_BACKEND=cache+memory://
ALLOWED_HOSTS=localhost,127.0.0.1,nginx,testserver
NEWSLETTER_API_INTERNAL_URL=http://127.0.0.1:8080
NEWSLETTER_PUBLIC_URL=http://127.0.0.1:8080
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ kubectl port-forward svc/newsletter-maker-newsletter-maker-nginx 8080:80
> Frontend credentials (from seed):
>
> Username: demo_editor
> Password: demo_password
> Password: demo-password

**Command Summary:**

Expand Down
2 changes: 2 additions & 0 deletions core/settings_types.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ class CoreSettings(Protocol):
LINKEDIN_OAUTH_SCOPES: str
METRICS_TOKEN: str
NEWSLETTER_API_BASE_URL: str
NEWSLETTER_API_INTERNAL_URL: str
NEWSLETTER_PUBLIC_URL: str
QDRANT_URL: str
EMBEDDING_MODEL: str
EMBEDDING_PROVIDER: str
Expand Down
2 changes: 1 addition & 1 deletion deploy/helm/newsletter-maker/templates/configmap.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ data:
CHANNEL_LAYER_URL: {{ default (include "newsletter-maker.redisUrl" .) .Values.env.channelLayerUrl | quote }}
QDRANT_URL: {{ include "newsletter-maker.qdrantUrl" . | quote }}
MESSAGING_ENABLED: {{ .Values.env.messagingEnabled | quote }}
NEWSLETTER_API_BASE_URL: {{ .Values.env.newsletterApiBaseUrl | quote }}
NEWSLETTER_PUBLIC_URL: {{ .Values.env.newsletterPublicUrl | quote }}
EMAIL_BACKEND: {{ .Values.env.emailBackend | quote }}
DEFAULT_FROM_EMAIL: {{ .Values.env.defaultFromEmail | quote }}
SERVER_EMAIL: {{ .Values.env.serverEmail | quote }}
Expand Down
2 changes: 1 addition & 1 deletion deploy/helm/newsletter-maker/values-minikube.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ env:
debug: "true"
allowedHosts: "localhost,127.0.0.1,newsletter-maker.local"
csrfTrustedOrigins: "http://localhost,http://127.0.0.1,http://newsletter-maker.local"
newsletterApiBaseUrl: "http://newsletter-maker.local"
newsletterPublicUrl: "http://newsletter-maker.local"

nginx:
service:
Expand Down
2 changes: 1 addition & 1 deletion deploy/helm/newsletter-maker/values-staging.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ env:
allowedHosts: "staging.newsletter-maker.example.com"
csrfTrustedOrigins: "https://staging.newsletter-maker.example.com"
messagingEnabled: "true"
newsletterApiBaseUrl: "https://staging.newsletter-maker.example.com"
newsletterPublicUrl: "https://staging.newsletter-maker.example.com"
logLevel: INFO

secrets:
Expand Down
2 changes: 1 addition & 1 deletion deploy/helm/newsletter-maker/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ env:
siteId: "1"
channelLayerUrl: ""
messagingEnabled: "true"
newsletterApiBaseUrl: "http://newsletter-maker.local"
newsletterPublicUrl: "http://newsletter-maker.local"
emailBackend: anymail.backends.resend.EmailBackend
defaultFromEmail: onboarding@resend.dev
serverEmail: onboarding@resend.dev
Expand Down
2 changes: 1 addition & 1 deletion docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -139,7 +139,7 @@ services:
env_file:
- .env
environment:
NEWSLETTER_API_BASE_URL: http://nginx
NEWSLETTER_API_INTERNAL_URL: http://nginx
NEXT_TELEMETRY_DISABLED: "1"
depends_on:
nginx:
Expand Down
22 changes: 19 additions & 3 deletions docs/admin-guide/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,29 +3,45 @@
See the [Tunables Reference](../reference/tunables.md) for the exact list of algorithms and thresholds.

## Required vs Optional Variables
**Required**:
* `DATABASE_URL`, `REDIS_URL`, `QDRANT_URL`, `SECRET_KEY`, `NEWSLETTER_API_BASE_URL`.

**Required**:

* `DATABASE_URL`, `REDIS_URL`, `QDRANT_URL`, `SECRET_KEY`, `NEWSLETTER_PUBLIC_URL`.

**Optional but critical for AI**:

* `OPENROUTER_API_KEY` (Required for relevance tie-breaking and categorization).

## Secrets Handling

* In Docker Compose: Loaded tightly from the `.env` file mapped securely to the container.
* In Kubernetes: Expected to be mapped into the Pod `env` spec via Secrets.

## Internal vs Public URLs

Due to container networking:
* `NEWSLETTER_API_BASE_URL` (Internal) will reference inner hostnames like `http://nginx`.

* `NEWSLETTER_API_INTERNAL_URL` (Internal) should reference inner hostnames like `http://nginx` when the frontend talks to the backend over a private Docker or Kubernetes network.
* `NEWSLETTER_PUBLIC_URL` (Public) should point to your real FQDN (e.g. `https://news.mydomain.com`) used in emails.

For local Docker Compose development, the default split is usually:

* `NEWSLETTER_API_INTERNAL_URL=http://nginx`
* `NEWSLETTER_PUBLIC_URL=http://127.0.0.1:8080`

## Email Provider (Anymail)

Newsletter intake relies on Resend webhooks and Django Anymail forwarding.
Configured via:

* `RESEND_API_KEY`
* `RESEND_INBOUND_SECRET`
* `DEFAULT_FROM_EMAIL`

## LLM Provider Routing

Select between `local`, `ollama` or remote providers using `EMBEDDING_PROVIDER`. Set URLs correctly to point to either the internal container (`http://ollama:11434`) or external APIs (`https://api.openai.com/v1`).

## OAuth Provider Toggles

If `LINKEDIN_CLIENT_ID` or `REDDIT_CLIENT_ID` are present, their respective capabilities light up dynamically in the application.
40 changes: 40 additions & 0 deletions docs/developer-guide/deployment.md
Original file line number Diff line number Diff line change
@@ -1,19 +1,59 @@
# Deployment

## just build Contract

The `just build` target makes zero assumptions about the environment file. It uses `DOCKER_BUILDKIT=0` to ensure legacy build isolation and host image cache utilization. No `.env` copies are made during build time.

## Docker Compose

Used primarily for local testing and running the application on a single VPS. See [Admin Installation](../admin-guide/installation.md) for details.

## Helm Chart Layout

For Kubernetes deployments, a reusable Helm chart sits in `deploy/helm/`.

## Minikube Quick Start

Use this path when you want to run the stack on a local Kubernetes cluster instead of Docker Compose.

Prerequisites:

- `minikube`
- `kubectl`
- `helm`

Start Minikube and deploy the chart:

```bash
minikube start
just k8s-build-minikube
just helm-lint
just helm-template
just k8s-install-minikube
```

Forward the Nginx service locally:

```bash
kubectl port-forward svc/newsletter-maker-newsletter-maker-nginx 8080:80
```

Then open <http://localhost:8080/> in your browser.

To remove the local release:

```bash
just k8s-uninstall-minikube
```

## ArgoCD Application

We maintain an ArgoCD application manifest in `deploy/argocd/` to support GitOps continuous delivery.

## Staging Overlay

Staging branches utilize encrypted / sealed secrets (or external secret operators) pushed into the cluster.

## Prometheus ServiceMonitor

If deployed alongside the `kube-prometheus-stack`, the chart deploys a `ServiceMonitor` to scrape port 8000 for Django metrics exposed by `django-prometheus`.
19 changes: 12 additions & 7 deletions docs/developer-guide/local-development.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
Newsletter Maker uses a **two-workflow split** to isolate fast local iteration from full full-stack fidelity.

## The Two-Workflow Split
1. **Host-Side Track**: Used for fast linting, typechecking, and unit tests WITHOUT spinning up Docker.
1. **Host-Side Track**: Used for fast linting, typechecking, and unit tests WITHOUT spinning up Docker.
2. **Docker Track**: Used for running the application, seeing the UI, background workers, and Postgres.

## Host-Side Track
Expand All @@ -15,15 +15,20 @@ When you run commands on your local OS (e.g., `just lint`, `just test`, `just fr
## Docker Track
When you want to run the app:
```bash
just build # Env-free container build (DOCKER_BUILDKIT=0)
docker compose up -d
just build
just dev
```
When running the Docker track, all runtime commands must be executed **inside the container**:

`just dev` runs the full Docker Compose stack in the foreground and keeps streaming service logs. Leave it running in the first terminal.

Open a second terminal for follow-up commands against the running stack:
```bash
docker compose exec django python manage.py migrate
docker compose exec django python manage.py bootstrap_live_sources
source .venv/bin/activate
just seed
```

After seeding completes, open <http://localhost:8080/> in your browser.

## Celery Beat Schedule
The Celery beat schedule file (`celerybeat-schedule`) is written to `.cache/` to prevent dirtying the project root or colliding between host/container environments.

Expand All @@ -35,4 +40,4 @@ cd frontend && npm run dev

## When to Use Which Workflow
* **Writing code, running tests, checking types**: Host-side (`just lint`, `just test`).
* **Testing LLMs, seeing the UI, testing ingestion, full pipelines**: Docker Track (`docker compose up`).
* **Testing LLMs, seeing the UI, testing ingestion, full pipelines**: Docker Track (`just dev`).
24 changes: 21 additions & 3 deletions docs/reference/tunables.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,48 +3,66 @@
This document collects all parameters, thresholds, and variables that change how the system behaves. Most global tunables are configured via environment variables and loaded into Django settings, while project-specific algorithms use `ProjectConfig`.

## How Settings Are Read

1. Environment variables set at the Docker Compose / Kubernetes pod level.
2. Loaded in `newsletter_maker/settings/base.py` and combined with defaults.
3. Consumed via `django.conf.settings` across the project.

## LLM & Embeddings
These map directly to global inference capability.

These map directly to global inference capability.

* `EMBEDDING_PROVIDER`: Options include `local` (HuggingFace `sentence-transformers`), `ollama`, or `openai`/`openrouter`.
* `EMBEDDING_MODEL`: The identifier for the dense vector model.
* `OLLAMA_URL`: Local instance of Ollama, defaulting to `http://ollama:11434`.
* `OPENROUTER_API_KEY`: Fallback or primary inference provider key for OpenRouter or OpenAI compatible APIs.
* `OPENROUTER_API_BASE`: Endpoint for inference.

## Relevance & Scoring Thresholds

Relevance rules divide candidate articles into clear-match, ambiguous, and clear-non-match bands. See [Algorithms](algorithms.md) for how the pipeline evaluates these.

* **Similarity Thresholds**: Embedding cosine similarity above `0.85` assumes auto-relevant. Below `0.5` assumes irrelevant. The `0.5 - 0.85` band asks the LLM.

## Deduplication Thresholds

* Usually implemented via nearest-neighbor distance (e.g., threshold `< 0.05` means near duplication).

## Authority Weights

Configured per-project in `ProjectConfig`:

* `authority_decay_rate` (default: 0.95): The rate at which an entity's authority metric decays over time without recent mentions.

## Topic Centroid
## Topic Centroid

Configured per-project in `ProjectConfig`:

* `recompute_topic_centroid_on_feedback_save` (default: True): Determines if a user's thumbs up/down immediately recomputes the vector centroid representing the project's topic.

## URL Settings
* `NEWSLETTER_API_BASE_URL`: **Internal API base URL** (e.g. `http://nginx` within Compose) and historically used as a **Public API URL** for generated links (requires external DNS resolution). This is pending split into distinct explicit variables.

* `NEWSLETTER_API_INTERNAL_URL`: Internal frontend-to-backend base URL used by the Next.js app for API and WebSocket traffic. In Docker Compose this is usually `http://nginx`.
* `NEWSLETTER_PUBLIC_URL`: Public backend base URL used when Django builds absolute links for emails and OAuth callbacks.
* `NEWSLETTER_API_BASE_URL`: Deprecated compatibility fallback. If present, it is used as the default for both explicit URL settings until those are configured separately.
* `FRONTEND_BASE_URL`: Where the Next.js app sits.

## Observability Retention

Keeps the database from ballooning over time.

* `OBSERVABILITY_SNAPSHOT_RETENTION_DAYS` (default: 90)
* `OBSERVABILITY_TREND_TASK_RUN_RETENTION_DAYS` (default: 30)
* `OBSERVABILITY_REVIEW_QUEUE_RETENTION_DAYS` (default: 30)

## OAuth Provider Toggles

Requires specific API keys to be populated to become available:

* **LinkedIn**: `LINKEDIN_CLIENT_ID`, `LINKEDIN_CLIENT_SECRET`, `LINKEDIN_OAUTH_SCOPES`
* **Reddit**: `REDDIT_CLIENT_ID`, `REDDIT_CLIENT_SECRET`, `REDDIT_USER_AGENT`

## Channels / Messaging

* `CHANNEL_LAYER_URL`: URL to the Redis instance used by Django Channels for ASGI web socket propagation (e.g., `redis://redis:6379/1`).
* `MESSAGING_ENABLED` (frontend/build feature flags).
2 changes: 1 addition & 1 deletion frontend/.env.example
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# Copy this file to .env.local when running the Next.js app outside Docker.
NEWSLETTER_API_BASE_URL=http://127.0.0.1:8080
NEWSLETTER_API_INTERNAL_URL=http://127.0.0.1:8080
NEWSLETTER_API_USERNAME=admin
NEWSLETTER_API_PASSWORD=adminpass
NEXT_TELEMETRY_DISABLED=1
Expand Down
2 changes: 1 addition & 1 deletion frontend/next-env.d.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
/// <reference types="next" />
/// <reference types="next/image-types/global" />
import "./.next/types/routes.d.ts";
import "./.next/dev/types/routes.d.ts";

// NOTE: This file should not be edited
// see https://nextjs.org/docs/app/api-reference/config/typescript for more information.
2 changes: 1 addition & 1 deletion frontend/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
"private": true,
"version": "0.1.0",
"scripts": {
"dev": "next dev",
"dev": "next dev --webpack",
"build": "next build",
"start": "next start",
"typecheck": "tsc --noEmit",
Expand Down
15 changes: 10 additions & 5 deletions frontend/src/app/(home)/_components/ContentFeed/index.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,12 @@ import { Button, buttonVariants } from "@/components/ui/button"
import { Card, CardContent } from "@/components/ui/card"
import type { Content } from "@/lib/types"
import { cn } from "@/lib/utils"
import { formatDate, formatPercentScore, truncateText } from "@/lib/view-helpers"
import {
formatDate,
formatDisplayLabel,
formatPercentScore,
truncateText,
} from "@/lib/view-helpers"

import type { ContentClusterBadge } from "../shared"

Expand Down Expand Up @@ -41,10 +46,10 @@ export function ContentFeed({
<div className="flex flex-col gap-4 md:flex-row md:items-start md:justify-between">
<div className="space-y-3">
<h3 className="font-display text-title-md font-bold">{content.title}</h3>
<div className="flex flex-wrap gap-2 text-sm text-muted">
<div className="flex flex-wrap gap-2 text-sm text-muted-foreground">
<span>{formatDate(content.published_date)}</span>
<span>{content.author || "Unknown author"}</span>
<span>{content.source_plugin}</span>
<span>{formatDisplayLabel(content.source_plugin)}</span>
</div>
</div>
<StatusBadge
Expand Down Expand Up @@ -73,7 +78,7 @@ export function ContentFeed({
</span>
) : null}
<span className="inline-flex items-center rounded-full border border-border/12 bg-muted/55 px-3 py-1 text-sm text-foreground">
{content.content_type || "unclassified"}
{formatDisplayLabel(content.content_type || "unclassified")}
</span>
{content.duplicate_signal_count > 0 ? (
<span className="inline-flex items-center rounded-full border border-border/12 bg-muted/55 px-3 py-1 text-sm text-foreground">
Expand Down Expand Up @@ -102,7 +107,7 @@ export function ContentFeed({
) : null}
</div>

<p className="text-sm leading-6 text-muted">{truncateText(content.content_text)}</p>
<p className="text-sm leading-6 text-muted-foreground">{truncateText(content.content_text)}</p>

<div className="flex flex-wrap items-center gap-3">
<Link
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,15 @@ describe("DashboardFilterToolbar", () => {
/>,
)

expect(screen.getByRole("button", { name: "Apply filters" })).toBeInTheDocument()
expect(screen.getByRole("button", { name: "Apply filters" })).toHaveClass(
"hover:bg-primary/84",
)
expect(screen.getByRole("link", { name: "Reset" })).toHaveAttribute("href", "/?project=1")
expect(container.querySelector("#dashboard-view-filter")).toHaveClass(
"border-border/45",
"bg-card/95",
"hover:bg-secondary/88",
)
expect(container.querySelector('input[name="project"]')).toHaveValue("1")
expect(container.querySelector('input[name="contentType"]')).toHaveValue("article")
expect(container.querySelector('input[name="source"]')).toHaveValue("rss")
Expand Down
Loading
Loading