Refactor dev workflow

.env.example

@@ -3,9 +3,10 @@ DEBUG=true
 ALLOWED_HOSTS=localhost,127.0.0.1
 CSRF_TRUSTED_ORIGINS=http://localhost,http://127.0.0.1,http://localhost:8080,http://127.0.0.1:8080
 
-DATABASE_URL=sqlite:///db.sqlite3
-REDIS_URL=redis://localhost:6379/0
-QDRANT_URL=http://localhost:6333
+# Docker Compose runtime defaults. Host-side lint and test use .env.test instead.
+DATABASE_URL=postgresql://newsletter:newsletter@postgres:5432/newsletter_maker
+REDIS_URL=redis://redis:6379/0
+QDRANT_URL=http://qdrant:6333
 
 OPENROUTER_API_KEY=
 OPENROUTER_API_BASE=https://openrouter.ai/api/v1
@@ -27,7 +28,7 @@ EMBEDDING_PROVIDER=sentence-transformers
 EMBEDDING_MODEL=sentence-transformers/all-MiniLM-L6-v2
 EMBEDDING_TRUST_REMOTE_CODE=false
 
-OLLAMA_URL=http://localhost:11434
+OLLAMA_URL=http://ollama:11434
 
 REDDIT_CLIENT_ID=
 REDDIT_CLIENT_SECRET=
@@ -75,7 +76,7 @@ DJANGO_SUPERUSER_USERNAME=admin
 DJANGO_SUPERUSER_EMAIL=admin@example.com
 DJANGO_SUPERUSER_PASSWORD=adminpass
 
-NEWSLETTER_API_BASE_URL=http://127.0.0.1:8080
+NEWSLETTER_API_BASE_URL=http://nginx
 NEWSLETTER_API_USERNAME=admin
 NEWSLETTER_API_PASSWORD=adminpass
 

.env.test

@@ -1,3 +1,4 @@
+DATABASE_URL=sqlite:///:memory:
 OPENROUTER_API_KEY=test-key
 OPENROUTER_API_BASE=https://openrouter.ai/api/v1
 OPENROUTER_APP_NAME=newsletter-maker

.gitignore

@@ -3,6 +3,7 @@ __pycache__/
 .pytest_cache/
 .mypy_cache/
 .ruff_cache/
+.cache/
 .venv/
 venv/
 .coverage

README.md

@@ -100,13 +100,17 @@ just install
 
 `just install` installs the backend and frontend dependencies and registers the repository's `pre-commit` hooks, so `git commit` runs the configured lint and test hooks locally.
 
+There are two intentionally separate workflows:
+
+- `just lint` and `just test` run on the host without Docker. The backend half of those commands uses `.env.test`.
+- Runtime, data, and Django management commands run against the Docker Compose stack.
+
 1. Run `just dev` to start Django, Celery, Postgres, Redis, Qdrant, and Nginx. On the first run Docker builds the app image automatically. After that, `just dev` reuses the existing image so normal restarts are fast. If `.env` is missing, the `just` command copies `.env.example` automatically.
-2. Run `just build` after changing `requirements.txt` or `docker/web/Dockerfile`.
+2. Run `just build` after changing `requirements.txt` or `docker/web/Dockerfile`. It does not copy or depend on local env files.
 3. For a fully fresh local stack after schema changes, run `just reset-volumes` before starting the containers again. This drops the Docker-backed Postgres, Redis, and Qdrant state so regenerated migrations apply cleanly.
-4. Update `.env` with non-default secrets before using the stack outside local development. The example file uses SQLite and localhost URLs so host-side `manage.py` commands work even without Docker.
-5. Open `http://localhost:8080/healthz/` for a liveness check and `http://localhost:8080/admin/` for Django admin. Use `just seed` after the stack is up if you want the demo project and sample content.
-
-For host-based development without Docker, install `requirements.txt`, then use `python3 manage.py migrate` and `python3 manage.py runserver`. The default `.env.example` is host-safe; Docker Compose overrides the service URLs inside containers.
+4. Run Django management commands against the running backend container. `just migrate`, `just shell`, `just embed-all`, `just embed-project <project_id>`, `just embed-smoke`, `just embed-smoke-content <content_id>`, and `just bootstrap-live-sources <project_id>` all use `docker compose exec django ...`.
+5. `.env.example` is Compose-oriented and uses Docker service hostnames for the backend runtime. Update `.env` with non-default secrets before using the stack outside local development.
+6. Open `http://localhost:8080/healthz/` for a liveness check and `http://localhost:8080/admin/` for Django admin. Use `just seed` after the stack is up if you want the demo project and sample content.
 
 ### Testing
 
@@ -118,6 +122,10 @@ just test
 
 Pytest auto-loads `.env.test` during test startup. That file is intentionally checked in and only contains non-sensitive placeholder values used by tests, such as fake API keys, fake Reddit credentials, and localhost service URLs.
 
+`.env.test` also pins Django tests to an explicit SQLite configuration so backend tests stay independent from the Compose-backed Postgres development database.
+
+`backend-lint` also runs Django-aware host-side checks (`mypy` with the Django plugin and `manage.py check`) under `.env.test`, so `just lint` stays independent from Docker.
+
 Use `.env.test` for stable dummy values that make tests deterministic. Do not put real secrets in it. Real local or production secrets belong in `.env`, which remains ignored.
 
 ### Embedding Backends
@@ -157,11 +165,11 @@ Use these commands to backfill or refresh embeddings for existing content:
 ```bash
 just embed-all
 just embed-project 1
-python3 manage.py sync_embeddings --content-id 42
-python3 manage.py sync_embeddings --references-only
+docker compose exec django python manage.py sync_embeddings --content-id 42
+docker compose exec django python manage.py sync_embeddings --references-only
 ```
 
-When `just dev` is running, Django admin uses the Postgres database inside Docker, not the host SQLite database. That means host commands like `python manage.py createsuperuser` create users in SQLite and will not let you log into the Docker-backed admin site.
+When `just dev` is running, Django admin and the developer-facing `just` wrappers all operate against the Compose-backed Postgres database.
 
 Create or update an admin user for the running Docker stack with:
 

docker-compose.yml

@@ -20,7 +20,16 @@ services:
     environment:
       BOOTSTRAP_APP: "true"
       <<: *shared-env
-    command: ["gunicorn", "newsletter_maker.wsgi:application", "--bind", "0.0.0.0:8000", "--workers", "2", "--reload"]
+    command:
+      [
+        "gunicorn",
+        "newsletter_maker.wsgi:application",
+        "--bind",
+        "0.0.0.0:8000",
+        "--workers",
+        "2",
+        "--reload",
+      ]
     depends_on:
       postgres:
         condition: service_healthy
@@ -29,7 +38,7 @@ services:
       qdrant:
         condition: service_started
     healthcheck:
-      test: ["CMD", "python", "manage.py", "check"]
+      test: [ "CMD", "python", "manage.py", "check" ]
       interval: 30s
       timeout: 10s
       retries: 5
@@ -39,7 +48,20 @@ services:
     environment:
       BOOTSTRAP_APP: "false"
       <<: *shared-env
-    command: ["watchmedo", "auto-restart", "--directory=.", "--pattern=*.py", "--recursive", "--", "celery", "-A", "newsletter_maker", "worker", "--loglevel=info"]
+    command:
+      [
+        "watchmedo",
+        "auto-restart",
+        "--directory=.",
+        "--pattern=*.py",
+        "--recursive",
+        "--",
+        "celery",
+        "-A",
+        "newsletter_maker",
+        "worker",
+        "--loglevel=info",
+      ]
     depends_on:
       postgres:
         condition: service_healthy
@@ -51,7 +73,14 @@ services:
     environment:
       BOOTSTRAP_APP: "false"
       <<: *shared-env
-    command: ["watchmedo", "auto-restart", "--directory=.", "--pattern=*.py", "--recursive", "--", "celery", "-A", "newsletter_maker", "beat", "--loglevel=info"]
+    command:
+      [
+        "sh",
+        "-lc",
+        "mkdir -p .cache && exec watchmedo auto-restart --directory=.
+          --pattern=*.py --recursive -- celery -A newsletter_maker beat
+          --loglevel=info --schedule=.cache/celerybeat-schedule",
+      ]
     depends_on:
       postgres:
         condition: service_healthy
@@ -67,18 +96,18 @@ services:
     volumes:
       - postgres_data:/var/lib/postgresql/data
     healthcheck:
-      test: ["CMD-SHELL", "pg_isready -U newsletter -d newsletter_maker"]
+      test: [ "CMD-SHELL", "pg_isready -U newsletter -d newsletter_maker" ]
       interval: 10s
       timeout: 5s
       retries: 5
 
   redis:
     image: redis:7-alpine
-    command: ["redis-server", "--save", "", "--appendonly", "no"]
+    command: [ "redis-server", "--save", "", "--appendonly", "no" ]
     volumes:
       - redis_data:/data
     healthcheck:
-      test: ["CMD", "redis-cli", "ping"]
+      test: [ "CMD", "redis-cli", "ping" ]
       interval: 10s
       timeout: 5s
       retries: 5
@@ -101,7 +130,12 @@ services:
   frontend:
     image: node:22-alpine
     working_dir: /app/frontend
-    command: ["sh", "-lc", "npm install && npm run dev -- --hostname 0.0.0.0 --port 3000"]
+    command:
+      [
+        "sh",
+        "-lc",
+        "npm install && npm run dev -- --hostname 0.0.0.0 --port 3000",
+      ]
     env_file:
       - .env
     environment:

justfile

@@ -4,6 +4,8 @@ compose := "docker compose"
 backend_env := "if [ ! -f .env ]; then cp .env.example .env; fi"
 frontend_env := "if [ ! -f frontend/.env.local ]; then cp frontend/.env.example frontend/.env.local; fi"
 frontend_cd := "cd frontend &&"
+django_manage := "docker compose exec django python manage.py"
+host_backend_test_env := "set -a && . ./.env.test && set +a &&"
 
 # -----------------------------------------------------------------------------
 # Setup
@@ -56,12 +58,10 @@ dev:
 
 # Build the backend Docker image used by the local stack
 backend-build:
-    @{{backend_env}}
-    @{{compose}} build django
+    @DOCKER_BUILDKIT=0 docker compose build django
 
 # Build the frontend production bundle
 frontend-build:
-    @{{frontend_env}}
     @{{frontend_cd}} npm run build
 
 # Build both backend and frontend deliverables
@@ -78,12 +78,11 @@ frontend-typecheck:
 
 # Lint and validate the backend Python and template code
 backend-lint:
-    @{{backend_env}}
     @ruff check manage.py core newsletter_maker tests
     @djlint core/templates --check
-    @python3 -m mypy
+    @{{host_backend_test_env}} python3 -m mypy
     @pre-commit run --all-files check-yaml
-    @python3 manage.py check
+    @{{host_backend_test_env}} python3 manage.py check
 
 # Lint and typecheck the frontend codebase
 frontend-lint:
@@ -96,7 +95,6 @@ lint: backend-lint frontend-lint helm-lint
 
 # Auto-fix backend lint issues where supported, then re-run backend validation
 backend-lint-fix:
-    @{{backend_env}}
     @ruff check manage.py core newsletter_maker tests --fix
     @djlint core/templates --reformat
     @pre-commit run --all-files end-of-file-fixer
@@ -138,12 +136,12 @@ frontend-test-all:
 
 # Run the backend test suite
 backend-test:
-    @python3 -m pytest
+    @{{host_backend_test_env}} python3 -m pytest
 
 # Run backend tests with terminal coverage output
 backend-test-coverage:
     @python3 -m coverage erase
-    @python3 -m coverage run -m pytest
+    @{{host_backend_test_env}} python3 -m coverage run -m pytest
     @python3 -m coverage report -m
 
 # Generate backend HTML coverage output
@@ -209,51 +207,51 @@ changepassword username:
     @{{backend_env}}
     @{{compose}} exec django python manage.py changepassword {{username}}
 
-# Apply Django database migrations locally
+# Apply Django database migrations in the running backend container
 migrate:
     @{{backend_env}}
-    @python3 manage.py migrate
+    @{{django_manage}} migrate
 
 # Seed demo data into the running backend container
 seed:
     @{{backend_env}}
     @{{compose}} exec django python manage.py seed_demo
 
-# Bootstrap RSS and Reddit source configs for one project in local development
+# Bootstrap RSS and Reddit source configs for one project in the running backend container
 bootstrap-live-sources project_id:
     @{{backend_env}}
-    @python3 manage.py bootstrap_live_sources \
+    @{{django_manage}} bootstrap_live_sources \
         --project-id {{project_id}} \
         ${RSS_FEEDS:+--rss-feed "$RSS_FEEDS"} \
         ${SUBREDDITS:+--subreddit "$SUBREDDITS"} \
         ${REDDIT_LISTING:+--reddit-listing "$REDDIT_LISTING"} \
         ${REDDIT_LIMIT:+--reddit-limit "$REDDIT_LIMIT"} \
         ${RUN_NOW:+--run-now}
 
-# Sync embeddings for all eligible content
+# Sync embeddings for all eligible content in the running backend container
 embed-all:
     @{{backend_env}}
-    @python3 manage.py sync_embeddings
+    @{{django_manage}} sync_embeddings
 
-# Sync embeddings for a single project
+# Sync embeddings for a single project in the running backend container
 embed-project project_id:
     @{{backend_env}}
-    @python3 manage.py sync_embeddings --project-id {{project_id}}
+    @{{django_manage}} sync_embeddings --project-id {{project_id}}
 
-# Run the embedding smoke test across the default sample content
+# Run the embedding smoke test across the default sample content in the running backend container
 embed-smoke:
     @{{backend_env}}
-    @python3 manage.py embedding_smoke
+    @{{django_manage}} embedding_smoke
 
-# Run the embedding smoke test for a single content item
+# Run the embedding smoke test for a single content item in the running backend container
 embed-smoke-content content_id:
     @{{backend_env}}
-    @python3 manage.py embedding_smoke --content-id {{content_id}}
+    @{{django_manage}} embedding_smoke --content-id {{content_id}}
 
-# Open a local Django shell
+# Open a Django shell in the running backend container
 shell:
     @{{backend_env}}
-    @python3 manage.py shell
+    @{{django_manage}} shell
 
 # Run the staged disaster recovery rehearsal workflow against the configured cluster
 disaster-recovery-rehearsal: