Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion .github/workflows/lint.yml
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,10 @@
cache-dependency-path: frontend/package-lock.json

- name: Install just
uses: extractions/setup-just@v4
uses: extractions/setup-crate@v2

Check warning

Code scanning / CodeQL

Unpinned tag for a non-immutable Action in workflow Medium

Unpinned 3rd party Action 'Lint' step
Uses Step
uses 'extractions/setup-crate' with ref 'v2', not a pinned commit hash
with:
repo: casey/just@1.50.0
github-token: ${{ github.token }}

- name: Install dependencies
run: just install
Expand Down
5 changes: 4 additions & 1 deletion .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,10 @@
cache-dependency-path: frontend/package-lock.json

- name: Install just
uses: extractions/setup-just@v4
uses: extractions/setup-crate@v2

Check warning

Code scanning / CodeQL

Unpinned tag for a non-immutable Action in workflow Medium test

Unpinned 3rd party Action 'Test' step
Uses Step
uses 'extractions/setup-crate' with ref 'v2', not a pinned commit hash
with:
repo: casey/just@1.50.0
github-token: ${{ github.token }}

- name: Install dependencies
run: just install
Expand Down
11 changes: 6 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,14 @@

An AI-powered content curation platform for technically-oriented newsletters. Newsletter Maker ingests content from dozens of sources, builds authority models of people and companies in a domain, and surfaces the most relevant articles, trends, and themes for each edition — so editors spend their time writing, not searching.

The system is multi-tenant: each newsletter has its own tracked entities, relevance model, and content pipeline. Designed for non-technical editors who don't know what a vector database is and don't need to.
The system is organized into projects: each newsletter project has its own tracked entities, relevance model, and content pipeline. Projects are assigned to Django groups so editorial access can be shared cleanly. Designed for non-technical editors who don't know what a vector database is and don't need to.

## What This Does That Existing Tools Don't

Tools like Feedly, UpContent, and ContentStudio handle parts of the content curation problem. Newsletter Maker combines several capabilities none of them offer:

- **Authority scoring from newsletter cross-referencing.** By ingesting peer newsletters, the system builds an authority model based on who real editors actually link to — a human-curated endorsement signal no existing tool provides.
- **Per-editor relevance training.** Upvote/downvote feedback trains a personalized relevance model per tenant. The tool learns what *you* consider valuable.
- **Per-project relevance training.** Upvote/downvote feedback trains a personalized relevance model per project. The tool learns what each editorial project considers valuable.
- **Unified entity model.** A person's blog, LinkedIn, Bluesky, GitHub, and conference talks are linked into a single profile with an authority score — a holistic view of who matters in a space.
- **Competitive intelligence.** "These 3 peer newsletters all covered topic X this week, but you haven't." A natural output of newsletter ingestion that no curation tool provides.
- **Historical trend analysis.** Not just what's trending now, but trajectories over weeks. Content is retained indefinitely for long-term pattern detection.
Expand Down Expand Up @@ -102,8 +102,9 @@ python3 -m pip install -r requirements.txt

1. Run `just dev` to start Django, Celery, Postgres, Redis, Qdrant, and Nginx. On the first run Docker builds the app image automatically. After that, `just dev` reuses the existing image so normal restarts are fast. If `.env` is missing, the `just` command copies `.env.example` automatically.
2. Run `just build` after changing `requirements.txt` or `docker/web/Dockerfile`.
3. Update `.env` with non-default secrets before using the stack outside local development. The example file uses SQLite and localhost URLs so host-side `manage.py` commands work even without Docker.
4. Open `http://localhost:8080/healthz/` for a liveness check and `http://localhost:8080/admin/` for Django admin.
3. For a fully fresh local stack after schema changes, run `just reset-volumes` before starting the containers again. This drops the Docker-backed Postgres, Redis, and Qdrant state so regenerated migrations apply cleanly.
4. Update `.env` with non-default secrets before using the stack outside local development. The example file uses SQLite and localhost URLs so host-side `manage.py` commands work even without Docker.
5. Open `http://localhost:8080/healthz/` for a liveness check and `http://localhost:8080/admin/` for Django admin. Use `just seed` after the stack is up if you want the demo project and sample content.

For host-based development without Docker, install `requirements.txt`, then use `python3 manage.py migrate` and `python3 manage.py runserver`. The default `.env.example` is host-safe; Docker Compose overrides the service URLs inside containers.

Expand Down Expand Up @@ -155,7 +156,7 @@ Use these commands to backfill or refresh embeddings for existing content:

```bash
just embed-all
just embed-tenant 1
just embed-project 1
python3 manage.py sync_embeddings --content-id 42
python3 manage.py sync_embeddings --references-only
```
Expand Down
106 changes: 53 additions & 53 deletions core/admin.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,60 +12,60 @@
Content,
Entity,
IngestionRun,
Project,
ProjectConfig,
ReviewQueue,
SkillResult,
SourceConfig,
Tenant,
TenantConfig,
UserFeedback,
)
from core.plugins import get_plugin_for_source_config, validate_plugin_config
from core.tasks import process_content


@admin.register(Tenant)
class TenantAdmin(ExportActionMixin, admin.ModelAdmin):
list_display = ("name", "user", "content_retention_days", "created_at")
@admin.register(Project)
class ProjectAdmin(ExportActionMixin, admin.ModelAdmin):
list_display = ("name", "group", "content_retention_days", "created_at")

# Better navigation
date_hierarchy = "created_at"
list_filter = ("created_at",)
# Better navigation
date_hierarchy = "created_at"
list_filter = ("created_at",)

# Faster searching
search_fields = ("name", "user__username", "user__email")
# Faster searching
search_fields = ("name", "group__name")

# Performance for large user lists
autocomplete_fields = ("user",)
# Performance for large user lists
autocomplete_fields = ("group",)

# Quick editing
list_editable = ("content_retention_days",)
# Quick editing
list_editable = ("content_retention_days",)


@admin.register(TenantConfig)
class TenantConfigAdmin(admin.ModelAdmin):
list_display = ("tenant", "upvote_authority_weight", "downvote_authority_weight", "authority_decay_rate")
@admin.register(ProjectConfig)
class ProjectConfigAdmin(admin.ModelAdmin):
list_display = ("project", "upvote_authority_weight", "downvote_authority_weight", "authority_decay_rate")


@admin.register(Entity)
class EntityAdmin(admin.ModelAdmin):
# Replace 'authority_score' with your new method name
list_display = ("name", "tenant", "type", "colored_score", "created_at")

@admin.display(description="Authority Score", ordering="authority_score")
def colored_score(self, obj):
# Choose a color based on the value
if obj.authority_score >= 80:
color = "green"
elif obj.authority_score >= 50:
color = "orange"
else:
color = "red"

return format_html(
'<span style="color: {}; font-weight: bold;">{}</span>',
color,
obj.authority_score,
)
# Replace 'authority_score' with your new method name
list_display = ("name", "project", "type", "colored_score", "created_at")

@admin.display(description="Authority Score", ordering="authority_score")
def colored_score(self, obj):
# Choose a color based on the value
if obj.authority_score >= 80:
color = "green"
elif obj.authority_score >= 50:
color = "orange"
else:
color = "red"

return format_html(
'<span style="color: {}; font-weight: bold;">{}</span>',
color,
obj.authority_score,
)


class HighValueFilter(admin.SimpleListFilter):
Expand All @@ -91,14 +91,14 @@ class ContentAdmin(admin.ModelAdmin):
"is_reference",
"preview_content",
"source_plugin",
"tenant",
"project",
"title",
"view_trace",
)
list_editable = ("is_reference", "is_active")
list_filter = (
HighValueFilter,
("tenant", admin.RelatedOnlyFieldListFilter),
("project", admin.RelatedOnlyFieldListFilter),
"source_plugin",
"is_active"
)
Expand Down Expand Up @@ -159,7 +159,7 @@ def view_trace(self, obj):
run_id=trace_id,
skill_name=latest_skill_result.skill_name,
skill_result_id=latest_skill_result.id,
tenant_id=obj.tenant_id,
project_id=obj.project_id,
trace_id=trace_id,
)

Expand Down Expand Up @@ -233,13 +233,13 @@ class SkillResultAdmin(ModelAdmin):
"model_used",
"created_at",
)
list_filter = ("status", "skill_name", "tenant", "model_used")
list_filter = ("status", "skill_name", "project", "model_used")
search_fields = ("skill_name", "content__title", "model_used", "error_message")
actions = ["retry_selected_skills"]
readonly_fields = ("pretty_result_data", "latency_ms", "created_at", "superseded_by")
fieldsets = (
("Execution Details", {
"fields": ("skill_name", "content", "tenant", "status", "model_used")
"fields": ("skill_name", "content", "project", "status", "model_used")
}),
("AI Output", {
"fields": ("pretty_result_data", "error_message"),
Expand Down Expand Up @@ -334,11 +334,11 @@ class UserFeedbackAdmin(ModelAdmin):
"display_feedback",
"get_content_title",
"get_ai_score",
"tenant",
"project",
"user",
"created_at"
)
list_filter = ("feedback_type", ("tenant", admin.RelatedOnlyFieldListFilter))
list_filter = ("feedback_type", ("project", admin.RelatedOnlyFieldListFilter))
search_fields = ("content__title", "user__email", "user__username")

@admin.display(description="Type")
Expand Down Expand Up @@ -387,18 +387,18 @@ def changelist_view(self, request, extra_context=None):
class IngestionRunAdmin(ModelAdmin):
list_display = (
"plugin_name",
"tenant",
"project",
"display_status",
"display_efficiency",
"display_duration",
"started_at",
)
list_filter = ("plugin_name", "status", ("tenant", admin.RelatedOnlyFieldListFilter))
search_fields = ("plugin_name", "error_message", "tenant__name")
list_filter = ("plugin_name", "status", ("project", admin.RelatedOnlyFieldListFilter))
search_fields = ("plugin_name", "error_message", "project__name")
readonly_fields = ("display_duration", "started_at", "completed_at")
fieldsets = (
("Run Info", {
"fields": ("plugin_name", "tenant", "status")
"fields": ("plugin_name", "project", "status")
}),
("Data Metrics", {
"fields": ("items_fetched", "items_ingested", "display_efficiency")
Expand Down Expand Up @@ -468,19 +468,19 @@ def changelist_view(self, request, extra_context=None):
class SourceConfigAdmin(ModelAdmin):
list_display = (
"plugin_name",
"tenant",
"project",
"display_health",
"is_active",
"last_fetched_at",
)
list_filter = ("is_active", "plugin_name", ("tenant", admin.RelatedOnlyFieldListFilter))
list_filter = ("is_active", "plugin_name", ("project", admin.RelatedOnlyFieldListFilter))
list_editable = ("is_active",)
search_fields = ("plugin_name", "tenant__name")
search_fields = ("plugin_name", "project__name")
actions = ["test_source_connection"]
readonly_fields = ("last_fetched_at", "pretty_config")
fieldsets = (
("Core Settings", {
"fields": ("plugin_name", "tenant", "is_active")
"fields": ("plugin_name", "project", "is_active")
}),
("Configuration", {
"fields": ("pretty_config", "config"),
Expand Down Expand Up @@ -519,7 +519,7 @@ def test_source_connection(self, request, queryset):
healthy_sources = []
failed_sources = []

for source_config in queryset.select_related("tenant"):
for source_config in queryset.select_related("project"):
try:
source_config.config = validate_plugin_config(
source_config.plugin_name,
Expand Down Expand Up @@ -573,14 +573,14 @@ def changelist_view(self, request, extra_context=None):
class ReviewQueueAdmin(ModelAdmin):
list_display = (
"get_content_title",
"tenant",
"project",
"reason",
"display_confidence",
"resolved",
"resolution",
"created_at",
)
list_filter = ("resolved", "reason", ("tenant", admin.RelatedOnlyFieldListFilter))
list_filter = ("resolved", "reason", ("project", admin.RelatedOnlyFieldListFilter))
list_editable = ("resolved", "resolution")
actions = ["mark_as_approved", "mark_as_rejected"]

Expand Down
Loading
Loading