release: v1.2.0 — Performance at Scale

[vishaltps]

Summary

• Resolves gateway timeouts on overview page with large datasets (millions of rows in solid_queue_jobs)
• Eliminates all expensive queries on the jobs table — stats derived entirely from execution tables
• Adds config.show_chart toggle to optionally disable chart queries

Problem

Issue #27 — Users with ~4M rows in solid_queue_jobs experienced gateway timeouts (502/504) when loading the overview page. Root causes:

1. Stats calculation: 3× COUNT(*) on solid_queue_jobs taking 52+ seconds each
2. Chart data: Plucking all timestamps into Ruby memory, then iterating in a loop
3. Filters: .pluck(:job_id) loading millions of IDs into Ruby arrays for WHERE IN
4. Queue stats: N+1 queries — 3 COUNT queries per queue row in QueuesPresenter

Solution

1. Stats from execution tables only

Replaced COUNT(*) on solid_queue_jobs with counts from small execution tables (ready_executions, scheduled_executions, claimed_executions, failed_executions). Dashboard now shows "Active Jobs" instead of "Total Jobs" and "Completed".

2. SQL GROUP BY chart bucketing

Replaced in-memory timestamp bucketing with SQL GROUP BY using computed bucket index. Cross-DB compatible (PostgreSQL + SQLite).

3. Subquery filters

Changed all .pluck(:job_id) to .select(:job_id) so filtering stays as a DB subquery instead of loading arrays into memory.

4. Pre-aggregated queue stats

Replaced per-queue COUNT queries with 3 GROUP BY queue_name queries in the controller, passed to presenter.

5. config.show_chart toggle

New configuration option to disable chart queries entirely for users who don't need the visualization.

Benchmark Results (100,000 jobs on PostgreSQL)

Pain Point	Before	After	Speedup
Stats calculation	75.18ms	3.10ms	24.3x
Chart data (1d)	147.96ms	22.02ms	6.7x
Chart data (1w)	771.35ms	89.27ms	8.6x
Filter (pluck → subquery)	7.43ms	1.57ms	4.7x
Queue stats (N+1)	36.51ms	7.22ms	5.1x

Note: Speedup grows dramatically with more rows. At 4M rows (the reporter's scale), the old stats calculation took 52+ seconds causing gateway timeouts. The new approach stays constant regardless of solid_queue_jobs table size.

Changes

• app/services/solid_queue_monitor/stats_calculator.rb — Rewritten to use execution tables only
• app/presenters/solid_queue_monitor/stats_presenter.rb — "Active Jobs" replaces "Total Jobs"/"Completed"
• app/controllers/solid_queue_monitor/base_controller.rb — All .pluck(:job_id) → .select(:job_id)
• app/controllers/solid_queue_monitor/queues_controller.rb — Pre-aggregate queue stats with GROUP BY
• app/controllers/solid_queue_monitor/in_progress_jobs_controller.rb — Fix pluck in filters
• app/presenters/solid_queue_monitor/queues_presenter.rb — Use pre-aggregated stats hash
• app/services/solid_queue_monitor/chart_data_service.rb — SQL GROUP BY bucketing (cross-DB)
• app/controllers/solid_queue_monitor/overview_controller.rb — Conditional chart loading
• lib/solid_queue_monitor.rb — Add show_chart config attribute
• lib/generators/solid_queue_monitor/templates/initializer.rb — Add show_chart config option

Breaking Change

Dashboard "Total Jobs" and "Completed" stats are replaced with "Active Jobs" (sum of ready + scheduled + in-progress + failed). This avoids expensive COUNT(*) on the jobs table that caused timeouts at scale.

Testing

• 255 examples, 0 failures, 5 pending
• New spec: no_unbounded_pluck_spec.rb — source-code scanning test preventing pluck regression
• Rewritten chart_data_service_spec.rb — behaviour-based tests on SQLite
• Updated stats_calculator_spec.rb — verifies SolidQueue::Job never receives :count
• Updated overview_spec.rb — tests show_chart = false skips chart queries
• Manual testing on PostgreSQL with 100K+ jobs (benchmark script)
• Cross-DB: tests pass on SQLite, benchmarked on PostgreSQL

Checklist

• Code follows project patterns
• Self-review completed
• Tests pass (255 examples, 0 failures)
• No debug statements left
• Version bumped to 1.2.0
• CHANGELOG updated
• README updated with Performance at Scale section
• ROADMAP updated

Closes #27