Content idea generation

SESSION.md

@@ -1 +1,73 @@
 # Session Restore Point
+
+## Current focus
+
+Phase 3 WP5: original content idea generation in `trends/`.
+
+## What landed today
+
+- Added `OriginalContentIdeaStatus` and `OriginalContentIdea` to `trends/models.py`.
+- Added migration `trends/migrations/0005_original_content_idea.py`.
+- Added new prompt skill folder `skills/original_content_ideation/` with:
+	- `SKILL.md`
+	- `resources/gap_detect.md`
+	- `resources/generate.md`
+	- `resources/critique.md`
+- Added partial WP5 task implementation in `trends/tasks.py`:
+	- `run_all_original_content_idea_generations`
+	- `generate_original_content_ideas`
+	- workflow helpers for accept / dismiss / mark written
+	- heuristic gap detection + fallback generation + heuristic critique
+	- optional OpenRouter-backed prompt-resource calls for gap detect / generate / critique
+- Added partial task coverage in `trends/tests/test_tasks.py` for:
+	- pending-idea creation
+	- weekly cap
+	- mark-written workflow
+
+## What was validated
+
+- `python manage.py check`
+- `python manage.py makemigrations --check --dry-run trends`
+
+Both passed after naming the new model index `core_idea_project_7f21_idx`.
+
+## Known incomplete state
+
+- WP5 is not finished.
+- No serializer/API route/test work has been added yet for `OriginalContentIdea`.
+- `docs/IMPLEMENTATION_PHASE_3.md` still needs a current implementation note for WP5 once the backend slice is complete.
+- The focused task validation was interrupted:
+	- first `pytest trends/tests/test_tasks.py -q` failed on a duplicate `_build_theme_cluster_context` definition introduced during editing
+	- that duplicate definition was removed
+	- the rerun was cancelled because we paused for shutdown
+- After that, additional external edits were reported in:
+	- `docs/IMPLEMENTATION_PHASE_3.md`
+	- `trends/tasks.py`
+	- `trends/tests/test_tasks.py`
+	Re-read those files before making new edits tomorrow.
+
+## First steps tomorrow
+
+1. Confirm the active branch. The repo attachment reported `maintenance/finish-core-refactor`, but an earlier terminal action created `feature/original-content-idea-generation`, so verify before continuing.
+2. Re-read:
+	 - `trends/tasks.py`
+	 - `trends/tests/test_tasks.py`
+	 - `docs/IMPLEMENTATION_PHASE_3.md`
+3. Run the focused validation that was interrupted:
+	 - `pytest trends/tests/test_tasks.py -q`
+4. If task tests pass, finish the remaining WP5 slice:
+	 - `trends/serializers.py`
+	 - `trends/api.py`
+	 - `trends/api_urls.py`
+	 - `trends/tests/test_api.py`
+	 - optional admin only if needed
+	 - add WP5 implementation note to `docs/IMPLEMENTATION_PHASE_3.md`
+5. After API work, run focused validation again:
+	 - `pytest trends/tests/test_tasks.py trends/tests/test_api.py -q`
+	 - `python3 -m mypy trends/tasks.py`
+
+## Likely follow-up checks
+
+- Watch for unused imports in `trends/tasks.py` (`build_skill_user_prompt` / `get_skill_definition` may no longer be needed for WP5 helpers).
+- Confirm the new heuristic tests still pass after any formatter changes.
+- Keep WP5 scoped to backend + docs for now; frontend `/ideas` work belongs to WP6.

skills/original_content_ideation/SKILL.md

@@ -0,0 +1,15 @@
+---
+name: original-content-ideation
+description: Generate grounded original article ideas from project trend gaps.
+input: project_topic_description, cluster_context, supporting_contents, recent_themes_accepted, recent_themes_dismissed
+output: angle_title, summary, suggested_outline, why_now, self_critique_score
+---
+
+This skill generates editor-facing original content ideas from project-scoped trend gaps.
+
+The runtime flow is split into three prompt resources under `resources/`:
+- `gap_detect.md`
+- `generate.md`
+- `critique.md`
+
+Each step must stay grounded in real project context, accepted and dismissed theme history, and the supplied supporting content.

skills/original_content_ideation/resources/critique.md

@@ -0,0 +1,7 @@
+You critique a generated original content idea before it reaches an editor.
+
+Score the idea for topic alignment, redundancy with recent project content, redundancy with theme history, and plausibility of the `why_now` rationale.
+
+Return structured JSON with these fields:
+- `self_critique_score`: a number between 0 and 1, where 1 is a strong editor-ready idea
+- `critique_summary`: one short sentence summarizing the critique

skills/original_content_ideation/resources/gap_detect.md

@@ -0,0 +1,9 @@
+You identify promising editorial gaps around one high-velocity project topic cluster.
+
+Use the supplied project topic, cluster context, supporting contents, and recent theme history.
+
+Prioritize clusters that are accelerating, still adjacent to the project's topic centroid, and not yet dominated by high-authority voices.
+
+Return structured JSON with these fields:
+- `gap_description`: one concise paragraph describing the undercovered opportunity
+- `gap_score`: a number between 0 and 1, where 1 is a very strong original-content opportunity

skills/original_content_ideation/resources/generate.md

@@ -0,0 +1,9 @@
+You generate one editor-facing original content idea from a validated project gap.
+
+Use only the supplied project topic, cluster context, gap analysis, and supporting contents. Keep the idea specific enough that an editor could plausibly assign or write it now.
+
+Return structured JSON with these fields:
+- `angle_title`: a short editorial angle title
+- `summary`: a concise explanation of the proposed article
+- `suggested_outline`: a short outline the editor could follow
+- `why_now`: one short paragraph explaining why the angle is timely

trends/api.py

@@ -2,9 +2,10 @@
 
 from typing import Any
 
+from django.conf import settings
 from django.db.models import Avg, Count, OuterRef, Prefetch, Q, Subquery
-from drf_spectacular.utils import OpenApiParameter, extend_schema
-from rest_framework import serializers, viewsets
+from drf_spectacular.utils import OpenApiParameter, extend_schema, inline_serializer
+from rest_framework import serializers, status, viewsets
 from rest_framework.decorators import action
 from rest_framework.filters import OrderingFilter
 from rest_framework.response import Response
@@ -19,6 +20,7 @@
 from core.permissions import IsProjectContributor, IsProjectMember
 from trends.models import (
     ContentClusterMembership,
+    OriginalContentIdea,
     SourceDiversitySnapshot,
     ThemeSuggestion,
     ThemeSuggestionStatus,
@@ -27,6 +29,8 @@
     TopicVelocitySnapshot,
 )
 from trends.serializers import (
+    OriginalContentIdeaDismissSerializer,
+    OriginalContentIdeaSerializer,
     SourceDiversityObservabilitySummarySerializer,
     SourceDiversitySnapshotSerializer,
     ThemeSuggestionDismissSerializer,
@@ -38,6 +42,12 @@
     TopicVelocitySnapshotSerializer,
 )
 from trends.tasks import accept_theme_suggestion, dismiss_theme_suggestion
+from trends.tasks import (
+    accept_original_content_idea,
+    dismiss_original_content_idea,
+    generate_original_content_ideas,
+    mark_original_content_idea_written,
+)
 
 
 def _require_pk(instance: Any) -> int:
@@ -278,6 +288,192 @@ def dismiss(self, request, *args, **kwargs):
         return Response(response_serializer.data)
 
 
+@document_project_owned_viewset(
+    resource_plural="original content ideas",
+    resource_singular="original content idea",
+    create_description="Original content ideas are pipeline-managed rows and are exposed read-only aside from editorial workflow actions.",
+    tag="Trend Analysis",
+    action_overrides=build_crud_action_overrides(
+        OriginalContentIdeaSerializer,
+        resource_plural="original content ideas for the selected project",
+        resource_singular="original content idea",
+    ),
+)
+class OriginalContentIdeaViewSet(
+    ProjectOwnedQuerysetMixin, viewsets.ReadOnlyModelViewSet
+):
+    """Inspect and resolve project-scoped original-content ideas."""
+
+    serializer_class = OriginalContentIdeaSerializer
+    filter_backends = [OrderingFilter]
+    ordering_fields = ["created_at", "self_critique_score", "status"]
+    ordering = ["status", "-self_critique_score", "-created_at"]
+    queryset = OriginalContentIdea.objects.select_related(
+        "project", "related_cluster", "decided_by"
+    )
+
+    def get_queryset(self):
+        """Prefetch supporting contents for original-content idea responses."""
+
+        return (
+            super()
+            .get_queryset()
+            .select_related("related_cluster__dominant_entity")
+            .prefetch_related(
+                Prefetch(
+                    "supporting_contents",
+                    queryset=Content.objects.order_by("-published_date", "-id"),
+                )
+            )
+        )
+
+    def get_permissions(self):
+        """Allow members to read ideas and contributors to resolve them."""
+
+        if self.action in {"accept", "dismiss", "mark_written", "generate"}:
+            return [IsProjectContributor()]
+        return [IsProjectMember()]
+
+    @extend_schema(
+        summary="Generate original content ideas",
+        description=(
+            "Trigger original-content idea generation for the selected project. "
+            "When Celery runs eagerly, ideas are generated before the response is returned. "
+            "Otherwise, the generation task is queued for background execution."
+        ),
+        request=None,
+        responses={
+            200: inline_serializer(
+                name="OriginalContentIdeaGenerateCompletedResponse",
+                fields={
+                    "status": serializers.CharField(),
+                    "project_id": serializers.IntegerField(),
+                    "result": inline_serializer(
+                        name="OriginalContentIdeaGenerateResult",
+                        fields={
+                            "project_id": serializers.IntegerField(),
+                            "clusters_considered": serializers.IntegerField(),
+                            "created": serializers.IntegerField(),
+                            "skipped": serializers.IntegerField(),
+                        },
+                    ),
+                },
+            ),
+            202: inline_serializer(
+                name="OriginalContentIdeaGenerateQueuedResponse",
+                fields={
+                    "status": serializers.CharField(),
+                    "project_id": serializers.IntegerField(),
+                },
+            ),
+            403: AUTHENTICATION_REQUIRED_RESPONSE,
+        },
+        tags=["Trend Analysis"],
+    )
+    @action(detail=False, methods=["post"], url_path="generate")
+    def generate(self, request, *args, **kwargs):
+        """Trigger original-content idea generation for the current project."""
+
+        project = self.get_project()
+        project_id = _require_pk(project)
+        if settings.CELERY_TASK_ALWAYS_EAGER:
+            result = generate_original_content_ideas(project_id)
+            return Response(
+                {
+                    "status": "completed",
+                    "project_id": project_id,
+                    "result": result,
+                }
+            )
+        generate_original_content_ideas.delay(project_id)
+        return Response(
+            {"status": "queued", "project_id": project_id},
+            status=status.HTTP_202_ACCEPTED,
+        )
+
+    @extend_schema(
+        summary="Accept original content idea",
+        description="Mark a pending original content idea as accepted by the current editor.",
+        request=None,
+        responses={
+            200: OriginalContentIdeaSerializer,
+            403: AUTHENTICATION_REQUIRED_RESPONSE,
+        },
+        tags=["Trend Analysis"],
+    )
+    @action(detail=True, methods=["post"], url_path="accept")
+    def accept(self, request, *args, **kwargs):
+        """Accept the selected pending original-content idea."""
+
+        idea = self.get_object()
+        try:
+            accept_original_content_idea(idea, user_id=request.user.id)
+        except ValueError as exc:
+            raise serializers.ValidationError(
+                {"status": "Unable to accept this original content idea."}
+            ) from exc
+        response_serializer = self.get_serializer(idea)
+        return Response(response_serializer.data)
+
+    @extend_schema(
+        summary="Dismiss original content idea",
+        description="Dismiss a pending original content idea and persist the editor's reason.",
+        request=OriginalContentIdeaDismissSerializer,
+        responses={
+            200: OriginalContentIdeaSerializer,
+            400: OriginalContentIdeaDismissSerializer,
+            403: AUTHENTICATION_REQUIRED_RESPONSE,
+        },
+        tags=["Trend Analysis"],
+    )
+    @action(detail=True, methods=["post"], url_path="dismiss")
+    def dismiss(self, request, *args, **kwargs):
+        """Dismiss the selected pending original-content idea."""
+
+        idea = self.get_object()
+        serializer = OriginalContentIdeaDismissSerializer(
+            data=request.data,
+            context=self.get_serializer_context(),
+        )
+        serializer.is_valid(raise_exception=True)
+        try:
+            dismiss_original_content_idea(
+                idea,
+                user_id=request.user.id,
+                reason=serializer.validated_data["reason"],
+            )
+        except ValueError as exc:
+            raise serializers.ValidationError(
+                {"status": "Unable to dismiss this original content idea."}
+            ) from exc
+        response_serializer = self.get_serializer(idea)
+        return Response(response_serializer.data)
+
+    @extend_schema(
+        summary="Mark original content idea written",
+        description="Mark an accepted original content idea as written by the current editor.",
+        request=None,
+        responses={
+            200: OriginalContentIdeaSerializer,
+            403: AUTHENTICATION_REQUIRED_RESPONSE,
+        },
+        tags=["Trend Analysis"],
+    )
+    @action(detail=True, methods=["post"], url_path="mark_written")
+    def mark_written(self, request, *args, **kwargs):
+        """Mark the selected accepted original-content idea as written."""
+
+        idea = self.get_object()
+        try:
+            mark_original_content_idea_written(idea, user_id=request.user.id)
+        except ValueError as exc:
+            raise serializers.ValidationError(
+                {"status": "Unable to mark this original content idea as written."}
+            ) from exc
+        response_serializer = self.get_serializer(idea)
+        return Response(response_serializer.data)
+
+
 @document_project_owned_viewset(
     resource_plural="topic centroid snapshots",
     resource_singular="topic centroid snapshot",

trends/api_urls.py

@@ -3,6 +3,7 @@
 from rest_framework_nested.routers import NestedSimpleRouter
 
 from trends.api import (
+    OriginalContentIdeaViewSet,
     SourceDiversitySnapshotViewSet,
     ThemeSuggestionViewSet,
     TopicCentroidSnapshotViewSet,
@@ -23,6 +24,11 @@ def register_project_routes(project_router: NestedSimpleRouter) -> None:
         ThemeSuggestionViewSet,
         basename="project-theme-suggestion",
     )
+    project_router.register(
+        r"ideas",
+        OriginalContentIdeaViewSet,
+        basename="project-original-content-idea",
+    )
     project_router.register(
         r"topic-centroid-snapshots",
         TopicCentroidSnapshotViewSet,