Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
# Environment and secrets
.env
*.env
config_files/credentials_config.json

# Version control
.git/
Expand All @@ -22,7 +21,6 @@ venv/
.venv/

# Docker files (not needed inside the image)
dockerfile
docker-compose.yml

# Documentation and non-runtime files
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,7 @@ jobs:
sudo chmod +x /usr/local/bin/gitleaks

- name: Run Gitleaks
run: gitleaks detect --source . --redact --verbose --baseline-path .gitleaks-baseline.json --exit-code 1
run: gitleaks detect --source . --config .gitleaks.toml --baseline-path .gitleaks-baseline.json --redact --verbose --exit-code 1

semgrep:
runs-on: ubuntu-latest
Expand Down
271 changes: 188 additions & 83 deletions .gitleaks-baseline.json

Large diffs are not rendered by default.

10 changes: 10 additions & 0 deletions .gitleaks.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
title = "Gitleaks config"

[extend]
useDefault = true

[[allowlists]]
description = "Ignore generated Gitleaks baseline report"
paths = [
'''^\.gitleaks-baseline\.json$'''
]
6 changes: 3 additions & 3 deletions dockerfile → Dockerfile
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# ── Stage 1: install dependencies in a throwaway builder ──────────────
FROM python:3.9-slim AS builder
FROM python:3.14-slim AS builder

# Prevent .pyc files and enable unbuffered stdout/stderr
ENV PYTHONDONTWRITEBYTECODE=1 \
Expand All @@ -13,7 +13,7 @@ RUN pip install --no-cache-dir --prefix=/install -r requirements.txt


# ── Stage 2: lean runtime image ──────────────────────────────────────
FROM python:3.9-slim AS runtime
FROM python:3.14-slim AS runtime

LABEL maintainer="Learning Dashboard team" \
description="LD Connect Event – webhook ingestion service"
Expand Down Expand Up @@ -41,7 +41,7 @@ EXPOSE 5000

# Healthcheck so Docker / Compose can monitor the service
HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:5000/')" || exit 1
CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:5000/health')" || exit 1

# Run gunicorn with the create_app() factory
CMD ["gunicorn", \
Expand Down
23 changes: 22 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,9 @@ pip install -r requirements.txt
# copy sample env and edit credentials / secrets
cp template.env .env

# create the directory that will contain your per-project API credentials
mkdir -p config_files

# run development server (single worker)
python app.py
```
Expand All @@ -91,9 +94,14 @@ docker compose up -d --build ld_connect
```

* Exposes the service on port **5000** inside the container
* Mounts `./config_files` into `/app/config_files` as read-only
* Behind Nginx / Traefik, route
`https://<your-domain>/webhook/{github|taiga|excel}` → `ld_connect:5000`

Before building or starting the service, make sure
`config_files/credentials_config.json` exists. It is used during local image builds
and can also be provided at runtime through the `./config_files` mount.

---

## Environment variables
Expand Down Expand Up @@ -186,7 +194,20 @@ Configured hooks:

### What's the origin and purpouse of credentials_config.json?

Basically, when LD Connect receives an event from GitHub or Taiga, it often needs to fetch additional details about the event (e.g., commit info, issue details) by calling the respective APIs. To authenticate these API calls, LD Connect uses tokens that are specific to each project or team. The `credentials_config.json` file serves as a mapping between project identifiers (like "TeamA") and their corresponding API tokens. This way, when an event comes in with a `prj` parameter, LD Connect can look up the correct token to use for any API requests related to that event.
Basically, when LD Connect receives an event from GitHub or Taiga, it often needs to fetch additional details about the event (e.g., commit info, issue details) by calling the respective APIs. To authenticate these API calls, LD Connect uses tokens that are specific to each project or team. The `credentials_config.json` file serves as a mapping between project identifiers (like "TeamA") and their corresponding API tokens. This way, when an event comes in with a `prj` parameter, LD Connect can look up the correct token to use for any API requests related to that event.

Minimal example:

```json
{
"course_a": {
"github_token": "ghp_replace_me",
"taiga_user": "replace-me",
"taiga_password": "replace-me",
"teams": ["TeamAlpha", "TeamBeta"]
}
}
```

## Can i use LD Connect alone, without LD-infrastructure?

Expand Down
54 changes: 53 additions & 1 deletion app.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,12 @@
from flask import Flask
from flask import Flask, jsonify
from routes.github_routes import github_bp
from routes.taiga_routes import taiga_bp
from routes.excel_routes import excel_bp
from config.credentials_loader import (
CredentialsConfigError,
ProjectCredentialsNotFoundError,
validate_config_file,
)
from config.logger_config import setup_logging
import logging
import os
Expand All @@ -10,13 +15,60 @@
logger = logging.getLogger(__name__)


def _register_error_handlers(app: Flask) -> None:
@app.errorhandler(CredentialsConfigError)
def handle_credentials_config_error(exc):
logger.error("Credentials configuration error: %s", exc)
return (
jsonify(
{
"error": "Credentials configuration error",
"details": str(exc),
}
),
500,
)

Check warning on line 30 in app.py

View check run for this annotation

SonarQubeCloud / SonarCloud Code Analysis

Specify an explicit HTTP status code for this error handler.

See more on https://sonarcloud.io/project/issues?id=Learning-Dashboard_LD_Connect_Event&issues=AZ0kki0ywFGn-mp-3Keg&open=AZ0kki0ywFGn-mp-3Keg&pullRequest=18

@app.errorhandler(ProjectCredentialsNotFoundError)
def handle_unknown_project(exc):
logger.warning("Unknown project in credentials config: %s", exc)
return (
jsonify(
{
"error": "Unknown project credentials",
"details": str(exc),
}
),
400,
)

Check warning on line 43 in app.py

View check run for this annotation

SonarQubeCloud / SonarCloud Code Analysis

Specify an explicit HTTP status code for this error handler.

See more on https://sonarcloud.io/project/issues?id=Learning-Dashboard_LD_Connect_Event&issues=AZ0kki0ywFGn-mp-3Keh&open=AZ0kki0ywFGn-mp-3Keh&pullRequest=18


def _log_credentials_config_status() -> None:
try:
config_path = validate_config_file()
except CredentialsConfigError as exc:
logger.warning("%s", exc)
else:
logger.info("Using credentials config at %s", config_path)


def create_app():
app = Flask(__name__)

@app.get("/health")
def health():
return jsonify({"status": "ok"}), 200

# Register blueprint routes
app.register_blueprint(github_bp)
app.register_blueprint(taiga_bp)
app.register_blueprint(excel_bp)

app.register_blueprint(github_bp, url_prefix="/webhooks", name="github_bp_prefixed")

Check failure on line 67 in app.py

View check run for this annotation

SonarQubeCloud / SonarCloud Code Analysis

Define a constant instead of duplicating this literal "/webhooks" 3 times.

See more on https://sonarcloud.io/project/issues?id=Learning-Dashboard_LD_Connect_Event&issues=AZ2nOTPmdnLmPxB3drzT&open=AZ2nOTPmdnLmPxB3drzT&pullRequest=18
app.register_blueprint(taiga_bp, url_prefix="/webhooks", name="taiga_bp_prefixed")
app.register_blueprint(excel_bp, url_prefix="/webhooks", name="excel_bp_prefixed")
_register_error_handlers(app)
_log_credentials_config_status()
logger.info("Flask created and Blueprints registered successfully.")
return app

Expand Down
73 changes: 64 additions & 9 deletions config/credentials_loader.py
Original file line number Diff line number Diff line change
@@ -1,13 +1,65 @@
import json
import os
from typing import Optional
from pathlib import Path
from typing import Any, Optional

CONFIG_FILE = os.getenv("CREDENTIALS_FILE", "config_files/credentials_config.json")
PROJECT_ROOT = Path(__file__).resolve().parent.parent
CONFIG_FILE_ENV = "CREDENTIALS_FILE"
DEFAULT_CONFIG_FILE = Path("config_files/credentials_config.json")


def load():
with open(CONFIG_FILE, "r", encoding="utf-8") as fh:
return json.load(fh)
class CredentialsConfigError(RuntimeError):
"""Base error for credentials configuration issues."""


class CredentialsConfigNotFoundError(FileNotFoundError, CredentialsConfigError):
"""Raised when the credentials config file cannot be found."""


class CredentialsConfigInvalidError(CredentialsConfigError, ValueError):
"""Raised when the credentials config file cannot be parsed."""


class ProjectCredentialsNotFoundError(KeyError):
"""Raised when a team/project is not present in the credentials config."""


def _raw_config_value(config_file: Optional[str] = None) -> str:
return config_file or os.getenv(CONFIG_FILE_ENV) or str(DEFAULT_CONFIG_FILE)


def get_config_path(config_file: Optional[str] = None) -> Path:
raw_value = _raw_config_value(config_file)
raw_path = Path(raw_value).expanduser()
if raw_path.is_absolute():
return raw_path
return (PROJECT_ROOT / raw_path).resolve()


def validate_config_file(config_file: Optional[str] = None) -> Path:
raw_value = _raw_config_value(config_file)
config_path = get_config_path(raw_value)
if config_path.is_file():
return config_path

raise CredentialsConfigNotFoundError(
"Credentials config file not found. "
f"Resolved {CONFIG_FILE_ENV}={raw_value!r} to {config_path}. "
"Create config_files/credentials_config.json, mount ./config_files "
f"into /app/config_files, or set {CONFIG_FILE_ENV} to an absolute path."
)


def load() -> dict[str, Any]:
config_path = validate_config_file()
try:
with config_path.open("r", encoding="utf-8") as fh:
return json.load(fh)
except json.JSONDecodeError as exc:
raise CredentialsConfigInvalidError(
f"Credentials config file at {config_path} contains invalid JSON: "
f"{exc.msg}"
) from exc


def resolve(prj: str, field: str) -> Optional[str]:
Expand All @@ -16,8 +68,11 @@ def resolve(prj: str, field: str) -> Optional[str]:
that corresponds to <prj>. Raise KeyError if not configured.
"""
cfg = load()
for course, props in cfg.items():
if prj in props["teams"]:

config_path = get_config_path()
for props in cfg.values():
if prj in props.get("teams", []):
return props.get(field)
raise KeyError(f"Project {prj!r} not found in {CONFIG_FILE}")

raise ProjectCredentialsNotFoundError(
f"Project {prj!r} not found in credentials config {config_path}"
)
7 changes: 4 additions & 3 deletions datasources/github_handler.py
Original file line number Diff line number Diff line change
Expand Up @@ -35,9 +35,10 @@ def parse_github_push_event(raw_payload: Dict, prj: str) -> Dict:

# The 'sender' object is at the top level
sender = raw_payload.get("sender", {})
sender_login = sender.get("login") or "anonymous"
sender_info = {
"id": sender.get("id", ""),
"login": sender.get("login", ""),
"login": sender_login,
"url": sender.get("url", ""),
"type": sender.get("type", ""),
"site_admin": sender.get("site_admin", False),
Expand All @@ -54,8 +55,8 @@ def parse_github_push_event(raw_payload: Dict, prj: str) -> Dict:
date = to_madrid_local(c.get("timestamp"))

# Built author information
author_login = c.get("author", {}).get("username", "")
author_name = c.get("author", {}).get("name", "")
author_login = c.get("author", {}).get("username") or sender_login
author_name = c.get("author", {}).get("name", "")
author_email = c.get("author", {}).get("email", "")

# Compute message stats
Expand Down
Loading
Loading