Skip to content

Conversation

@XxUnkn0wnxX
Copy link

@XxUnkn0wnxX XxUnkn0wnxX commented Nov 8, 2025

Fix optimizer, docs, and defaults: stabilize dev HTTPS + APPSI HiGHS

Summary

This PR bundles three developer‑experience fixes:

  1. Optimizer stability in dev (HTTPS): switch to the in‑process APPSI HiGHS backend and silence solver console output to avoid Pyomo capture deadlocks that produced intermittent 500 on POST /optimize/.
  2. Cross‑platform local HTTPS docs: add clear instructions for running locally over HTTPS without touching production settings (pip‑only sslserver quickstart, plus uvicorn+cert alternative; consistent port 8100; certify/ignore local certs).
  3. Defaults seeding cleanup: move default‑settings bootstrap out of AppConfig.ready() into a post_migrate signal; add a logger warning when ui/data/default.json is missing. This removes the Django warning about DB access during app init and makes first‑run behavior explicit.

Branch: feat/dev-https-docs-and-appsi-hardening


Motivation

  • Dev HTTPS runs (Werkzeug/ASGI with TLS) occasionally deadlocked when HiGHS (APPSI) wrote to stdout/stderr while the dev server also captured output. Symptoms included:

    • DeveloperError: Deadlock closing capture_output
    • RuntimeError: TeeStream: deadlock observed joining reader threads
    • Follow‑on ssl.SSLError … _ssl.c:2427 (debug page write after socket closed)
  • New contributors also tripped on enforced HTTPS locally; we needed a one‑command, cross‑platform dev path.

  • Django 5.2 warns against DB access in AppConfig.ready(); defaults bootstrap lived there.


Changes

1) Optimizer: explicit APPSI + quiet logs

  • Use the explicit APPSI driver (pure‑pip, Windows/macOS/Linux) and reduce console output to avoid capture contention under dev servers.
- from pyomo.opt import SolverFactory
+ from pyomo.environ import SolverFactory
@@
- solver = SolverFactory("highs")
+ solver = SolverFactory("appsi_highs")  # Pyomo APPSI HiGHS (in‑process)
+ try:
+     solver.options["log_to_console"] = False
+ except Exception:
+     pass

Functional behavior (solutions) is unchanged; only the integration/logging path differs.

2) Docs: cross‑platform local HTTPS

  • README quickstart now shows an all‑pip HTTPS command and links to the new dev guide.

  • docs/dev-local.md (new):

    • Option A (pip‑only): django-sslserver self‑signed on https://127.0.0.1:8100/.
    • Option B (trusted cert): uvicorn with a local cert (mkcert optional; NSS note for Firefox).
    • Recommends port 8100 to avoid collisions; ensures certs/*.pem are git‑ignored.

3) Defaults seeding: post_migrate + logging

  • Move defaults/bootstrap to a post_migrate receiver scoped to the ui app and guarded by dispatch_uid.
  • Make the seeder idempotent; skip if defaults exist.
  • Add a logger warning when ui/data/default.json is missing so contributors immediately see why defaults weren’t created.
  • Result: no more “accessing DB during app initialization” warning; first‑run experience is deterministic.

Testing

Solver smoke test (no server)

python - <<'PY'
import pyomo.environ as pyo
m = pyo.ConcreteModel(); m.x = pyo.Var(bounds=(0,None))
m.obj = pyo.Objective(expr=m.x, sense=pyo.minimize)
m.c   = pyo.Constraint(expr=m.x >= 1)
s = pyo.SolverFactory("appsi_highs")
print("available:", s.available(False))
s.options["log_to_console"] = False
s.solve(m)
print("x =", pyo.value(m.x))
PY

Expect: available: True and x = 1.0.

Dev HTTPS flows

  • sslserver: python manage.py runsslserver 127.0.0.1:8100
  • runserver_plus: python manage.py runserver_plus --cert-file certs/localhost.pem --key-file certs/localhost-key.pem 127.0.0.1:8100
  • ASGI: DJANGO_ALLOW_ASYNC_UNSAFE=true uvicorn satopt.asgi:application --host 127.0.0.1 --port 8100 --ssl-certfile certs/localhost.pem --ssl-keyfile certs/localhost-key.pem
    • While logged in and logged out, submit Optimize repeatedly; expect stable 200 responses with no capture/deadlock traces.

Defaults bootstrap

  • Fresh DB (rm db.sqlite3 && python manage.py migrate) creates defaults if ui/data/default.json exists.
  • If the JSON is absent, a clear warning appears in logs; app starts cleanly.

Backwards compatibility

  • No production behavior changes. Solver selection/logging only affects dev; docs target contributor setup.
  • APPSI path is pure‑pip and cross‑platform. Users who prefer external executables can still configure CBC/GLPK locally without code changes.

Notes for maintainers

  • If desired, we can add a small helper to try appsi_highs and fall back to an external CLI solver (CBC/GLPK) automatically.

Checklist

  • Optimizer uses appsi_highs; solver console logging silenced in dev
  • README quickstart updated; new docs/dev-local.md
  • certs/*.pem ignored by git
  • Defaults seeding moved to post_migrate; idempotent; warning on missing JSON
  • Verified stable /optimize/ under sslserver and uvicorn

Ready for review.


Related

- ignore tmp/, AGENTS.md, .venv/, and dev TLS certs

- switch UI optimizer to appsi_highs with quiet logging fallback
- add sslserver and django-extensions to INSTALLED_APPS

- extend requirements with local HTTPS dev dependencies
- replace plain text README with Markdown version

- archive original README.txt under docs/oldreadme.txt
- add cross-platform dev-local.md guide covering mkcert and self-signed flows
- update running instructions to reference docs/dev-local.md via relative link
- add Homebrew reference link beside macOS commands
- collapse identical Bash/PowerShell snippets for runsslserver and runserver_plus
- move DefaultSettings bootstrap from AppConfig.ready() into a post_migrate signal

- prevents Django 5.2 RuntimeWarning about database access during app init

- retains existing default.json import and error handling
- add logging to the post_migrate seeding hook so missing data/default.json surfaces immediately

- helps devs understand why defaults were not created after migrations
- switch optimizer import to pyomo.environ so the appsi_highs plugin loads reliably

- prevents 'solver plugin was not registered' errors seen with direct pyomo.opt import
- include official GNU GPL v3 text in COPYING.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add docs for macOS HTTPS local development (mkcert or OpenSSL)

1 participant