Release Candidate v5.0.21

.agent/rules/code-style.md

@@ -14,6 +14,10 @@ Always start with modifying the main code. When done, look for tests and update
 
 In Python, I want you to use the latest type syntax (`type | None`) instead of `Optional`. I also want you to use a single space (`=`) around the equals sign (`=`) in function argument calls. It's important to use double quotation marks (`"`) instead of single quotations (`'`). And finally, we want to always use trailing commas in multi-line function declarations and calls. There's never a reason to write `unittest.main()` manually, we have a script for running tests. Never use inline imports inside of functions (use file header even in tests), and always use `from ... import ...` syntax at the top of the file.
 
+#### Error Handling
+
+Never use generic `ValueError`, `AssertionError`, or bare `Exception` for raising errors. Always use the structured exceptions from `util.errors` (`ValidationError`, `NotFoundError`, `AuthorizationError`, `ExternalServiceError`, `RateLimitError`, `ConfigurationError`, `InternalError`). Each raise must include an error code from `util.error_codes`. When re-raising from a caught exception, always use `raise ... from e` to preserve the chain. When calling external services (LLMs, image APIs, web fetchers), always guard against empty/null/empty-array responses with `ExternalServiceError`.
+
 #### JavaScript/TypeScript
 
 In JavaScript and TypeScript, use types as much as possible: strict mode will be turned on! If in doubt, follow Java standard formatting. Finally, we also always want trailing commas in multi-line code blocks.

.agent/rules/tooling.md

@@ -18,14 +18,14 @@ trigger: always_on
 
 #### Development Workflow
 
-- Use `./tools/run_dev.sh` for development server (includes hot reload, verbose logging, dev API key)
-- Use `./tools/run_lint.sh --fix` for code quality checks with `ruff` (this flag will auto-fix issues it can)
-- Use `./tools/run_prod.sh` for production runs
+- Use `pipenv install --dev` and `pipenv run python src/main.py --dev` for development server (includes hot reload, verbose logging, dev API key)
+- Use `pipenv run pre-commit run --all-files --show-diff-on-failure` for code quality checks
+- Use `pipenv install` and `pipenv run python src/main.py` for production runs
 - For all other operations like testing, always run inside of `pipenv`
 
 #### Code Quality
 
-- Always run linting before commits: `./tools/run_lint.sh --fix`
+- Always run linting before commits: `pipenv run pre-commit run`
 - All scripts handle environment setup automatically (PYTHONPATH, .env files)
 
 #### Project Structure

.claude/CLAUDE.md

@@ -10,6 +10,10 @@ Always start with modifying the main code. When done, look for tests and update
 
 In Python, I want you to use the latest type syntax (`type | None`) instead of `Optional`. I also want you to use a single space (`=`) around the equals sign (`=`) in function argument calls. It's important to use double quotation marks (`"`) instead of single quotations (`'`). And finally, we want to always use trailing commas in multi-line function declarations and calls. There's never a reason to write `unittest.main()` manually, we have a script for running tests. Never use inline imports inside of functions (use file header even in tests), and always use `from ... import ...` syntax at the top of the file.
 
+#### Error Handling
+
+Never use generic `ValueError`, `AssertionError`, or bare `Exception` for raising errors. Always use the structured exceptions from `util.errors` (`ValidationError`, `NotFoundError`, `AuthorizationError`, `ExternalServiceError`, `RateLimitError`, `ConfigurationError`, `InternalError`). Each raise must include an error code from `util.error_codes`. When re-raising from a caught exception, always use `raise ... from e` to preserve the chain. When calling external services (LLMs, image APIs, web fetchers), always guard against empty/null/empty-array responses with `ExternalServiceError`.
+
 #### JavaScript/TypeScript
 
 In JavaScript and TypeScript, use types as much as possible: strict mode will be turned on! If in doubt, follow Java standard formatting. Finally, we also always want trailing commas in multi-line code blocks.
@@ -30,14 +34,14 @@ In JavaScript and TypeScript, use types as much as possible: strict mode will be
 
 #### Development Workflow
 
-- Use `./tools/run_dev.sh` for development server (includes hot reload, verbose logging, dev API key)
-- Use `./tools/run_lint.sh --fix` for code quality checks with `ruff` (this flag will auto-fix issues it can)
-- Use `./tools/run_prod.sh` for production runs
+- Use `pipenv install --dev` and `pipenv run python src/main.py --dev` for development server (includes hot reload, verbose logging, dev API key)
+- Use `pipenv run pre-commit run --all-files --show-diff-on-failure` for code quality checks
+- Use `pipenv install` and `pipenv run python src/main.py` for production runs
 - For all other operations like testing, always run inside of `pipenv`
 
 #### Code Quality
 
-- Always run linting before commits: `./tools/run_lint.sh --fix`
+- Always run linting before commits: `pipenv run pre-commit run`
 - All scripts handle environment setup automatically (PYTHONPATH, .env files)
 
 #### Project Structure

.cursor/rules/style.mdc

@@ -15,6 +15,10 @@ Always start with modifying the main code. When done, look for tests and update
 
 In Python, I want you to use the latest type syntax (`type | None`) instead of `Optional`. I also want you to use a single space (`=`) around the equals sign (`=`) in function argument calls. It's important to use double quotation marks (`"`) instead of single quotations (`'`). And finally, we want to always use trailing commas in multi-line function declarations and calls. There's never a reason to write `unittest.main()` manually, we have a script for running tests. Never use inline imports inside of functions (use file header even in tests), and always use `from ... import ...` syntax at the top of the file.
 
+#### Error Handling
+
+Never use generic `ValueError`, `AssertionError`, or bare `Exception` for raising errors. Always use the structured exceptions from `util.errors` (`ValidationError`, `NotFoundError`, `AuthorizationError`, `ExternalServiceError`, `RateLimitError`, `ConfigurationError`, `InternalError`). Each raise must include an error code from `util.error_codes`. When re-raising from a caught exception, always use `raise ... from e` to preserve the chain. When calling external services (LLMs, image APIs, web fetchers), always guard against empty/null/empty-array responses with `ExternalServiceError`.
+
 #### JavaScript/TypeScript
 
 In JavaScript and TypeScript, use types as much as possible: strict mode will be turned on! If in doubt, follow Java standard formatting. Finally, we also always want trailing commas in multi-line code blocks.

.cursor/rules/tooling.mdc

@@ -19,14 +19,14 @@ alwaysApply: true
 
 #### Development Workflow
 
-- Use `./tools/run_dev.sh` for development server (includes hot reload, verbose logging, dev API key)
-- Use `./tools/run_lint.sh --fix` for code quality checks with `ruff` (this flag will auto-fix issues it can)
-- Use `./tools/run_prod.sh` for production runs
+- Use `pipenv install --dev` and `pipenv run python src/main.py --dev` for development server (includes hot reload, verbose logging, dev API key)
+- Use `pipenv run pre-commit run --all-files --show-diff-on-failure` for code quality checks
+- Use `pipenv install` and `pipenv run python src/main.py` for production runs
 - For all other operations like testing, always run inside of `pipenv`
 
 #### Code Quality
 
-- Always run linting before commits: `./tools/run_lint.sh --fix`
+- Always run linting before commits: `pipenv run pre-commit run`
 - All scripts handle environment setup automatically (PYTHONPATH, .env files)
 
 #### Project Structure

.dockerignore

@@ -9,4 +9,5 @@
 !alembic.ini
 !.env
 !pyproject.toml
+!config/
 !.github/ISSUE_TEMPLATE/

.github/workflows/beta.yml

@@ -9,34 +9,39 @@ jobs:
   build:
     runs-on: ubuntu-latest
     if: github.event.pull_request.user.login != 'dependabot[bot]'
+    permissions:
+      contents: write
+      packages: write
+      pull-requests: write
     steps:
       - uses: actions/checkout@v2
         with:
-          fetch-depth: 100
+          fetch-depth: 200
 
       - name: Print build name
         run: echo "$GITHUB_ACTOR is building '$GITHUB_REPOSITORY' (commit $GITHUB_SHA)"
 
+      - name: Set up Python
+        id: setup-python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12.12"
+
       - name: Cache dependencies
         uses: actions/cache@v4
         with:
           path: |
             ~/.cache/pip
             ~/.local/share/virtualenvs
-          key: ${{ runner.os }}-dependency-cache
-
-      - name: Set up Python
-        uses: actions/setup-python@v5
-        with:
-          python-version: "3.12.6"
+          key: ${{ runner.os }}-python-${{ steps.setup-python.outputs.python-version }}-dependency-cache
 
       - name: Install dependencies
         run: |
           pip install --no-cache-dir pipenv
           pipenv install --dev
 
       - name: Lint checks
-        run: ./tools/run_lint.sh
+        run: pipenv run pre-commit run --all-files --show-diff-on-failure
 
       - name: Test the service
         env:
@@ -47,27 +52,24 @@ jobs:
       - name: Build Docker image
         run: docker build . --file Dockerfile --tag the-agent
 
-      - name: Auth for GitHub's Docker repository
-        env:
-          USER: ${{ secrets.PACKAGES_USER }}
-          TOKEN: ${{ secrets.PACKAGES_TOKEN }}
-        run: echo $TOKEN | docker login docker.pkg.github.com --username $USER --password-stdin
+      - name: Auth for GitHub's Container Registry
+        run: echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io --username "${{ github.actor }}" --password-stdin
 
       - name: Tag Docker image
         run: |
           VERSION=$(python -c "import tomllib; print(tomllib.load(open('pyproject.toml', 'rb'))['project']['version'])")
           echo "Tagging images with version '$VERSION'..."
-          docker tag the-agent docker.pkg.github.com/$GITHUB_REPOSITORY/the-agent:"$VERSION".beta
-          docker tag the-agent docker.pkg.github.com/$GITHUB_REPOSITORY/the-agent:latest_beta
+          docker tag the-agent ghcr.io/$GITHUB_REPOSITORY:"$VERSION".beta
+          docker tag the-agent ghcr.io/$GITHUB_REPOSITORY:latest_beta
           docker tag the-agent appifyhub/the-agent:"$VERSION".beta
           docker tag the-agent appifyhub/the-agent:latest_beta
 
       - name: Publish Docker images to GitHub
         run: |
           VERSION=$(python -c "import tomllib; print(tomllib.load(open('pyproject.toml', 'rb'))['project']['version'])")
           echo "Publishing images with version '$VERSION'..."
-          docker push docker.pkg.github.com/$GITHUB_REPOSITORY/the-agent:"$VERSION".beta
-          docker push docker.pkg.github.com/$GITHUB_REPOSITORY/the-agent:latest_beta
+          docker push ghcr.io/$GITHUB_REPOSITORY:"$VERSION".beta
+          docker push ghcr.io/$GITHUB_REPOSITORY:latest_beta
 
       - name: Publish Docker images to DockerHub
         id: docker_publish
@@ -155,10 +157,10 @@ jobs:
               -H "Content-Type: application/json" \
               -d "{\"release_output_b64\": \"$RELEASE_OUTPUT_B64\"}" \
               -s || true)
-            SHOULD_RETRY=$(echo "$SUMMARY_RESPONSE" | grep -o '"should_retry"[[:space:]]*:[[:space:]]*[^,}]*' | head -n1 | sed 's/.*:[[:space:]]*\([^,}]*\).*/\1/')
+            SHOULD_RETRY=$(echo "$SUMMARY_RESPONSE" | jq -r '.should_retry // false')
             if [[ "$SHOULD_RETRY" != "true" ]]; then
               echo "Notification successful (should_retry = $SHOULD_RETRY)"
-              SUMMARY=$(echo "$SUMMARY_RESPONSE" | grep -o '"summary"[[:space:]]*:[[:space:]]*"[^"]*"' | head -n1 | sed 's/.*: *"\([^"]*\)".*/\1/')
+              SUMMARY=$(echo "$SUMMARY_RESPONSE" | jq -r '.summary // ""')
               SUMMARY_B64=$(printf "%s" "$SUMMARY" | base64 -w 0)
               echo "summary_b64=$SUMMARY_B64" >> $GITHUB_OUTPUT
               exit 0

.github/workflows/dependabot-main-pr-lock.yml

@@ -30,14 +30,6 @@ jobs:
       - name: Checkout repository
         uses: actions/checkout@v4
 
-      - name: Cache dependencies
-        uses: actions/cache@v4
-        with:
-          path: |
-            ~/.cache/pip
-            ~/.local/share/virtualenvs
-          key: ${{ runner.os }}-dependency-cache
-
       - name: Remove the locked and verified labels if present
         uses: actions-ecosystem/action-remove-labels@v1
         continue-on-error: true
@@ -48,9 +40,18 @@ jobs:
             Verified
 
       - name: Set up Python
+        id: setup-python
         uses: actions/setup-python@v5
         with:
-          python-version: "3.12.6"
+          python-version: "3.12.12"
+
+      - name: Cache dependencies
+        uses: actions/cache@v4
+        with:
+          path: |
+            ~/.cache/pip
+            ~/.local/share/virtualenvs
+          key: ${{ runner.os }}-python-${{ steps.setup-python.outputs.python-version }}-dependency-cache
 
       - name: Install and lock dependencies
         run: |

.github/workflows/dependabot-main-pr-verify.yml

@@ -33,14 +33,6 @@ jobs:
       - name: Checkout repository
         uses: actions/checkout@v4
 
-      - name: Cache dependencies
-        uses: actions/cache@v4
-        with:
-          path: |
-            ~/.cache/pip
-            ~/.local/share/virtualenvs
-          key: ${{ runner.os }}-dependency-cache
-
       - name: Remove the verified label if present
         uses: actions-ecosystem/action-remove-labels@v1
         continue-on-error: true
@@ -49,9 +41,18 @@ jobs:
           labels: Verified
 
       - name: Set up Python
+        id: setup-python
         uses: actions/setup-python@v5
         with:
-          python-version: "3.12.6"
+          python-version: "3.12.12"
+
+      - name: Cache dependencies
+        uses: actions/cache@v4
+        with:
+          path: |
+            ~/.cache/pip
+            ~/.local/share/virtualenvs
+          key: ${{ runner.os }}-python-${{ steps.setup-python.outputs.python-version }}-dependency-cache
 
       - name: Install dependencies
         run: |

.github/workflows/qa.yml

@@ -8,6 +8,10 @@ jobs:
   build:
     runs-on: ubuntu-latest
     if: github.event.pull_request.user.login != 'dependabot[bot]'
+    permissions:
+      contents: write
+      packages: write
+      pull-requests: write
     steps:
       - uses: actions/checkout@v2
         with:
@@ -16,26 +20,27 @@ jobs:
       - name: Print build name
         run: echo "$GITHUB_ACTOR is building PR#$PR_NUMBER in '$GITHUB_REPOSITORY' (commit $GITHUB_SHA)"
 
+      - name: Set up Python
+        id: setup-python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12.12"
+
       - name: Cache dependencies
         uses: actions/cache@v4
         with:
           path: |
             ~/.cache/pip
             ~/.local/share/virtualenvs
-          key: ${{ runner.os }}-dependency-cache
-
-      - name: Set up Python
-        uses: actions/setup-python@v5
-        with:
-          python-version: "3.12.6"
+          key: ${{ runner.os }}-python-${{ steps.setup-python.outputs.python-version }}-dependency-cache
 
       - name: Install dependencies
         run: |
           pip install --no-cache-dir pipenv
           pipenv install --dev
 
       - name: Lint checks
-        run: ./tools/run_lint.sh
+        run: pipenv run pre-commit run --all-files --show-diff-on-failure
 
       - name: Test the service
         env:
@@ -46,20 +51,17 @@ jobs:
       - name: Build Docker image
         run: docker build . --file Dockerfile --tag the-agent
 
-      - name: Auth for GitHub's Docker repository
-        env:
-          USER: ${{ secrets.PACKAGES_USER }}
-          TOKEN: ${{ secrets.PACKAGES_TOKEN }}
-        run: echo $TOKEN | docker login docker.pkg.github.com --username $USER --password-stdin
+      - name: Auth for GitHub's Container Registry
+        run: echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io --username "${{ github.actor }}" --password-stdin
 
       - name: Tag Docker image
         env:
           PR_NUMBER: ${{ github.event.number }}
         run: |
           VERSION=$(python -c "import tomllib; print(tomllib.load(open('pyproject.toml', 'rb'))['project']['version'])")
           echo "Tagging images with version '$VERSION'..."
-          docker tag the-agent docker.pkg.github.com/$GITHUB_REPOSITORY/the-agent:"$VERSION".pr_$PR_NUMBER
-          docker tag the-agent docker.pkg.github.com/$GITHUB_REPOSITORY/the-agent:latest_pr
+          docker tag the-agent ghcr.io/$GITHUB_REPOSITORY:"$VERSION".pr_$PR_NUMBER
+          docker tag the-agent ghcr.io/$GITHUB_REPOSITORY:latest_pr
           docker tag the-agent appifyhub/the-agent:"$VERSION".pr_$PR_NUMBER
           docker tag the-agent appifyhub/the-agent:latest_pr
 
@@ -69,8 +71,8 @@ jobs:
         run: |
           VERSION=$(python -c "import tomllib; print(tomllib.load(open('pyproject.toml', 'rb'))['project']['version'])")
           echo "Publishing images with version '$VERSION'..."
-          docker push docker.pkg.github.com/$GITHUB_REPOSITORY/the-agent:"$VERSION".pr_$PR_NUMBER
-          docker push docker.pkg.github.com/$GITHUB_REPOSITORY/the-agent:latest_pr
+          docker push ghcr.io/$GITHUB_REPOSITORY:"$VERSION".pr_$PR_NUMBER
+          docker push ghcr.io/$GITHUB_REPOSITORY:latest_pr
 
       - name: Publish Docker images to DockerHub
         id: docker_publish