Skip to content

documentation.md

Latest commit

 

History

History
68 lines (57 loc) · 3.96 KB

documentation.md

File metadata and controls

68 lines (57 loc) · 3.96 KB

CLARITY Convertor Technical Documentation

Architecture Overview

  • Frontend: React (Create React App v5) with Material UI. Ships as static assets served by nginx in production.
  • Backend: FastAPI application with JWT authentication, Postgres persistence (SQLAlchemy ORM), and PDF processing that renders each page to PNG (Poppler/pdf2image) before prompting an Ollama vision-capable model for structured LaTeX output.
  • Database: PostgreSQL 13 running in a container.
  • Background Processing: Synchronous for now; upload requests perform PDF extraction inline.
  • Storage: Converted page outputs stored in Postgres. Uploaded PDFs are streamed to temporary files during processing and removed immediately afterwards.

Data Flow

  1. User signs in and obtains a short-lived JWT stored in localStorage.
  2. Dashboard loads conversion sessions via authenticated requests to /conversions.
  3. Uploading a PDF streams the file to a temporary location, renders each page to PNG, sends the image plus contextual instructions to Ollama, and persists the structured LaTeX response alongside the raw extracted text.
  4. The frontend caches session metadata locally while fetching conversion outputs from the API on demand.
  5. Deleting a conversion removes the database record and all associated page outputs.

API Reference (selected endpoints)

  • POST /register — create a new user (requires admin activation).
  • POST /token — obtain an access token.
  • GET /conversions — list conversion sessions for the current user.
  • GET /conversions/{session_id} — detailed view with page outputs.
  • POST /upload-pdf/ — upload a PDF for conversion (requires file and model_name form-data fields).
  • DELETE /conversions/{session_id} — remove a session and its page outputs.
  • GET /ollama/models — list available Ollama models.

Detailed schema definitions live in backend/app/schemas.py.

Environment Variables

  • DATABASE_URL — SQLAlchemy connection string (docker-compose provides one).
  • SECRET_KEY — JWT signing secret (set in docker-compose for production).
  • OLLAMA_API_URL — base URL for the Ollama service.

Uploaded files are stored only temporarily in backend/uploads/ during processing and removed immediately afterwards. Each conversion now stores the following per-page payload in Postgres:

{
  "page_title": "...",
  "sub_heading": "...",
  "page_text": "...",
  "equations": {
    "1": {"latex": "...", "text": "..."}
  },
  "source_text": "..."
}

Timeouts

  • OLLAMA_TIMEOUT_SECONDS governs how long the backend waits for each Ollama request. Values <= 0 disable the timeout to accommodate large PDFs; otherwise use seconds.
  • nginx (frontend/nginx.conf) mirrors the timeout window so the reverse proxy doesn’t cut off long-running requests first.

Testing

  • Frontend: npm test (Jest). Note that Create React App's Jest configuration needs a transformer for react-router-dom@7 ESM modules before tests can run successfully under Node 18/19.
  • Frontend Build: npm run build (used by Docker image).
  • Backend: python -m compileall app provides a quick syntax check. Add pytest-based suites as the service grows.

Docker Images

  • clarityconvertor-backend — FastAPI app + gunicorn/uvicorn (dev server). Mounts the source tree for live reload during development.
  • clarityconvertor-frontend — nginx serving compiled React assets.
  • postgres:13 — database with persistent volume postgres_data.

Rebuild images when dependencies change:

docker-compose build

Maintenance Notes

  • The backend seeds an admin user (lakshay / 1312) on first startup if none exists.
  • If you relocate the backend service, update the frontend API base URLs (currently default to http://localhost:8000). Parameterize them using environment variables or .env files for multi-environment deployments.
  • Poppler is required locally for pdf2image; when running in Docker the backend image installs poppler-utils automatically.