release: v1.2.0 — Performance at Scale

CHANGELOG.md

@@ -1,5 +1,22 @@
 # Changelog
 
+## [1.2.0] - 2026-03-07
+
+### Changed
+
+- **BREAKING**: Dashboard "Total Jobs" and "Completed" stats replaced with "Active Jobs" (sum of ready + scheduled + in-progress + failed). This avoids expensive `COUNT(*)` on the jobs table at scale.
+
+### Fixed
+
+- **Performance**: Overview page no longer queries `solid_queue_jobs` for stats — all counts derived from execution tables (resolves gateway timeouts with millions of rows) ([#27](https://github.com/vishaltps/solid_queue_monitor/issues/27))
+- **Performance**: Chart data service uses SQL `GROUP BY` bucketing instead of loading all timestamps into Ruby memory
+- **Performance**: All filter methods use `.select(:job_id)` subqueries instead of unbounded `.pluck(:job_id)`
+- **Performance**: Queue stats pre-aggregated with 3 `GROUP BY` queries, eliminating N+1 per-queue COUNT queries
+
+### Added
+
+- `config.show_chart` option to disable the job activity chart and skip chart queries entirely
+
 ## [1.1.0] - 2026-02-07
 
 ### Added

Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    solid_queue_monitor (1.1.0)
+    solid_queue_monitor (1.2.0)
       rails (>= 7.0)
       solid_queue (>= 0.1.0)
 

README.md

@@ -72,7 +72,7 @@ A lightweight, zero-dependency web interface for monitoring Solid Queue backgrou
 Add this line to your application's Gemfile:
 
 ```ruby
-gem 'solid_queue_monitor', '~> 1.1'
+gem 'solid_queue_monitor', '~> 1.2'
 ```
 
 Then execute:
@@ -118,9 +118,27 @@ SolidQueueMonitor.setup do |config|
 
   # Auto-refresh interval in seconds (default: 30)
   config.auto_refresh_interval = 30
+
+  # Disable the chart on the overview page to skip chart queries entirely
+  # config.show_chart = true
 end
 ```
 
+### Performance at Scale
+
+SolidQueueMonitor is optimized for large datasets (millions of rows in `solid_queue_jobs`):
+
+- **Overview stats** are derived entirely from execution tables (`ready_executions`, `scheduled_executions`, `claimed_executions`, `failed_executions`), avoiding expensive `COUNT(*)` queries on the jobs table.
+- **Chart data** uses SQL `GROUP BY` bucketing instead of loading timestamps into Ruby memory.
+- **Filters** use subqueries (`.select(:job_id)`) instead of loading ID arrays into memory.
+- **Queue stats** are pre-aggregated with `GROUP BY` to avoid N+1 queries.
+
+If you don't need the job activity chart, disable it to skip chart queries entirely:
+
+```ruby
+config.show_chart = false
+```
+
 ### Authentication
 
 By default, Solid Queue Monitor does not require authentication to access the dashboard. This makes it easy to get started in development environments.

ROADMAP.md

@@ -22,6 +22,7 @@ This document tracks planned features for solid_queue_monitor, comparing with ot
 | Job Details Page | Dedicated page for single job with full context | ✅ Done |
 | Search/Full-text Search | Better search across all job data | ✅ Done |
 | Sorting Options | Sort by various columns | ✅ Done |
+| Large Dataset Performance | Optimized for millions of rows — no jobs table scans | ✅ Done |
 | Backtrace Cleaner | Remove framework noise from error backtraces | ⬚ Planned |
 | Manual Job Triggering | Enqueue a job directly from the dashboard | ⬚ Planned |
 | Cancel Running Jobs | Stop long-running jobs | ⬚ Planned |

app/controllers/solid_queue_monitor/base_controller.rb

@@ -91,17 +91,13 @@ def filter_jobs(relation)
         when 'completed'
           relation = relation.where.not(finished_at: nil)
         when 'failed'
-          failed_job_ids = SolidQueue::FailedExecution.pluck(:job_id)
-          relation = relation.where(id: failed_job_ids)
+          relation = relation.where(id: SolidQueue::FailedExecution.select(:job_id))
         when 'scheduled'
-          scheduled_job_ids = SolidQueue::ScheduledExecution.pluck(:job_id)
-          relation = relation.where(id: scheduled_job_ids)
+          relation = relation.where(id: SolidQueue::ScheduledExecution.select(:job_id))
         when 'pending'
-          # Pending jobs are those that are not completed, failed, or scheduled
-          failed_job_ids = SolidQueue::FailedExecution.pluck(:job_id)
-          scheduled_job_ids = SolidQueue::ScheduledExecution.pluck(:job_id)
           relation = relation.where(finished_at: nil)
-                             .where.not(id: failed_job_ids + scheduled_job_ids)
+                             .where.not(id: SolidQueue::FailedExecution.select(:job_id))
+                             .where.not(id: SolidQueue::ScheduledExecution.select(:job_id))
         end
       end
 
@@ -117,16 +113,13 @@ def filter_ready_jobs(relation)
       return relation unless params[:class_name].present? || params[:queue_name].present? || params[:arguments].present?
 
       if params[:class_name].present?
-        job_ids = SolidQueue::Job.where('class_name LIKE ?', "%#{params[:class_name]}%").pluck(:id)
-        relation = relation.where(job_id: job_ids)
+        relation = relation.where(job_id: SolidQueue::Job.where('class_name LIKE ?', "%#{params[:class_name]}%").select(:id))
       end
 
       relation = relation.where('queue_name LIKE ?', "%#{params[:queue_name]}%") if params[:queue_name].present?
 
-      # Add arguments filtering
       if params[:arguments].present?
-        job_ids = SolidQueue::Job.where('arguments::text ILIKE ?', "%#{params[:arguments]}%").pluck(:id)
-        relation = relation.where(job_id: job_ids)
+        relation = relation.where(job_id: SolidQueue::Job.where('arguments::text ILIKE ?', "%#{params[:arguments]}%").select(:id))
       end
 
       relation
@@ -136,16 +129,13 @@ def filter_scheduled_jobs(relation)
       return relation unless params[:class_name].present? || params[:queue_name].present? || params[:arguments].present?
 
       if params[:class_name].present?
-        job_ids = SolidQueue::Job.where('class_name LIKE ?', "%#{params[:class_name]}%").pluck(:id)
-        relation = relation.where(job_id: job_ids)
+        relation = relation.where(job_id: SolidQueue::Job.where('class_name LIKE ?', "%#{params[:class_name]}%").select(:id))
       end
 
       relation = relation.where('queue_name LIKE ?', "%#{params[:queue_name]}%") if params[:queue_name].present?
 
-      # Add arguments filtering
       if params[:arguments].present?
-        job_ids = SolidQueue::Job.where('arguments::text ILIKE ?', "%#{params[:arguments]}%").pluck(:id)
-        relation = relation.where(job_id: job_ids)
+        relation = relation.where(job_id: SolidQueue::Job.where('arguments::text ILIKE ?', "%#{params[:arguments]}%").select(:id))
       end
 
       relation
@@ -170,25 +160,19 @@ def filter_failed_jobs(relation)
       return relation unless params[:class_name].present? || params[:queue_name].present? || params[:arguments].present?
 
       if params[:class_name].present?
-        job_ids = SolidQueue::Job.where('class_name LIKE ?', "%#{params[:class_name]}%").pluck(:id)
-        relation = relation.where(job_id: job_ids)
+        relation = relation.where(job_id: SolidQueue::Job.where('class_name LIKE ?', "%#{params[:class_name]}%").select(:id))
       end
 
       if params[:queue_name].present?
-        # Check if FailedExecution has queue_name column
-        if relation.column_names.include?('queue_name')
-          relation = relation.where('queue_name LIKE ?', "%#{params[:queue_name]}%")
-        else
-          # If not, filter by job's queue_name
-          job_ids = SolidQueue::Job.where('queue_name LIKE ?', "%#{params[:queue_name]}%").pluck(:id)
-          relation = relation.where(job_id: job_ids)
-        end
+        relation = if relation.column_names.include?('queue_name')
+                     relation.where('queue_name LIKE ?', "%#{params[:queue_name]}%")
+                   else
+                     relation.where(job_id: SolidQueue::Job.where('queue_name LIKE ?', "%#{params[:queue_name]}%").select(:id))
+                   end
       end
 
-      # Add arguments filtering
       if params[:arguments].present?
-        job_ids = SolidQueue::Job.where('arguments::text ILIKE ?', "%#{params[:arguments]}%").pluck(:id)
-        relation = relation.where(job_id: job_ids)
+        relation = relation.where(job_id: SolidQueue::Job.where('arguments::text ILIKE ?', "%#{params[:arguments]}%").select(:id))
       end
 
       relation

app/controllers/solid_queue_monitor/in_progress_jobs_controller.rb

@@ -22,13 +22,11 @@ def filter_in_progress_jobs(relation)
       return relation if params[:class_name].blank? && params[:arguments].blank?
 
       if params[:class_name].present?
-        job_ids = SolidQueue::Job.where('class_name LIKE ?', "%#{params[:class_name]}%").pluck(:id)
-        relation = relation.where(job_id: job_ids)
+        relation = relation.where(job_id: SolidQueue::Job.where('class_name LIKE ?', "%#{params[:class_name]}%").select(:id))
       end
 
       if params[:arguments].present?
-        job_ids = SolidQueue::Job.where('arguments::text ILIKE ?', "%#{params[:arguments]}%").pluck(:id)
-        relation = relation.where(job_id: job_ids)
+        relation = relation.where(job_id: SolidQueue::Job.where('arguments::text ILIKE ?', "%#{params[:arguments]}%").select(:id))
       end
 
       relation

app/controllers/solid_queue_monitor/overview_controller.rb

@@ -6,7 +6,7 @@ class OverviewController < BaseController
 
     def index
       @stats = SolidQueueMonitor::StatsCalculator.calculate
-      @chart_data = SolidQueueMonitor::ChartDataService.new(time_range: time_range_param).calculate
+      @chart_data = SolidQueueMonitor.show_chart ? SolidQueueMonitor::ChartDataService.new(time_range: time_range_param).calculate : nil
 
       recent_jobs_query = SolidQueue::Job.limit(100)
       sorted_query = apply_sorting(filter_jobs(recent_jobs_query), SORTABLE_COLUMNS, 'created_at', :desc)
@@ -29,13 +29,13 @@ def time_range_param
     end
 
     def generate_overview_content
-      SolidQueueMonitor::StatsPresenter.new(@stats).render +
-        SolidQueueMonitor::ChartPresenter.new(@chart_data).render +
-        SolidQueueMonitor::JobsPresenter.new(@recent_jobs[:records],
-                                             current_page: @recent_jobs[:current_page],
-                                             total_pages: @recent_jobs[:total_pages],
-                                             filters: filter_params,
-                                             sort: sort_params).render
+      html = SolidQueueMonitor::StatsPresenter.new(@stats).render
+      html += SolidQueueMonitor::ChartPresenter.new(@chart_data).render if @chart_data
+      html + SolidQueueMonitor::JobsPresenter.new(@recent_jobs[:records],
+                                                  current_page: @recent_jobs[:current_page],
+                                                  total_pages: @recent_jobs[:total_pages],
+                                                  filters: filter_params,
+                                                  sort: sort_params).render
     end
   end
 end

app/controllers/solid_queue_monitor/queues_controller.rb

@@ -10,8 +10,13 @@ def index
                                   .select('queue_name, COUNT(*) as job_count')
       @queues = apply_queue_sorting(base_query)
       @paused_queues = QueuePauseService.paused_queues
+      @queue_stats = aggregate_queue_stats
 
-      render_page('Queues', SolidQueueMonitor::QueuesPresenter.new(@queues, @paused_queues, sort: sort_params).render)
+      render_page('Queues', SolidQueueMonitor::QueuesPresenter.new(
+        @queues, @paused_queues,
+        queue_stats: @queue_stats,
+        sort: sort_params
+      ).render)
     end
 
     def show
@@ -57,6 +62,15 @@ def resume
 
     private
 
+    def aggregate_queue_stats
+      {
+        ready: SolidQueue::ReadyExecution.group(:queue_name).count,
+        scheduled: SolidQueue::ScheduledExecution.group(:queue_name).count,
+        failed: SolidQueue::FailedExecution.joins(:job)
+                                           .group('solid_queue_jobs.queue_name').count
+      }
+    end
+
     def calculate_queue_counts(queue_name)
       {
         total: SolidQueue::Job.where(queue_name: queue_name).count,
@@ -77,17 +91,13 @@ def filter_queue_jobs(relation)
         when 'completed'
           relation = relation.where.not(finished_at: nil)
         when 'failed'
-          failed_job_ids = SolidQueue::FailedExecution.pluck(:job_id)
-          relation = relation.where(id: failed_job_ids)
+          relation = relation.where(id: SolidQueue::FailedExecution.select(:job_id))
         when 'scheduled'
-          scheduled_job_ids = SolidQueue::ScheduledExecution.pluck(:job_id)
-          relation = relation.where(id: scheduled_job_ids)
+          relation = relation.where(id: SolidQueue::ScheduledExecution.select(:job_id))
         when 'pending'
-          ready_job_ids = SolidQueue::ReadyExecution.pluck(:job_id)
-          relation = relation.where(id: ready_job_ids)
+          relation = relation.where(id: SolidQueue::ReadyExecution.select(:job_id))
         when 'in_progress'
-          claimed_job_ids = SolidQueue::ClaimedExecution.pluck(:job_id)
-          relation = relation.where(id: claimed_job_ids)
+          relation = relation.where(id: SolidQueue::ClaimedExecution.select(:job_id))
         end
       end
 

app/presenters/solid_queue_monitor/queues_presenter.rb

@@ -2,10 +2,11 @@
 
 module SolidQueueMonitor
   class QueuesPresenter < BasePresenter
-    def initialize(records, paused_queues = [], sort: {})
-      @records = records
+    def initialize(records, paused_queues = [], sort: {}, queue_stats: {})
+      @records       = records
       @paused_queues = paused_queues
-      @sort = sort
+      @sort          = sort
+      @queue_stats   = queue_stats
     end
 
     def render
@@ -39,16 +40,16 @@ def generate_table
 
     def generate_row(queue)
       queue_name = queue.queue_name || 'default'
-      paused = @paused_queues.include?(queue_name)
+      paused     = @paused_queues.include?(queue_name)
 
       <<-HTML
         <tr class="#{paused ? 'queue-paused' : ''}">
           <td>#{queue_link(queue_name)}</td>
           <td>#{status_badge(paused)}</td>
           <td>#{queue.job_count}</td>
-          <td>#{ready_jobs_count(queue_name)}</td>
-          <td>#{scheduled_jobs_count(queue_name)}</td>
-          <td>#{failed_jobs_count(queue_name)}</td>
+          <td>#{@queue_stats.dig(:ready, queue_name) || 0}</td>
+          <td>#{@queue_stats.dig(:scheduled, queue_name) || 0}</td>
+          <td>#{@queue_stats.dig(:failed, queue_name) || 0}</td>
           <td class="actions-cell">#{action_button(queue_name, paused)}</td>
         </tr>
       HTML
@@ -84,19 +85,5 @@ def action_button(queue_name, paused)
         HTML
       end
     end
-
-    def ready_jobs_count(queue_name)
-      SolidQueue::ReadyExecution.where(queue_name: queue_name).count
-    end
-
-    def scheduled_jobs_count(queue_name)
-      SolidQueue::ScheduledExecution.where(queue_name: queue_name).count
-    end
-
-    def failed_jobs_count(queue_name)
-      SolidQueue::FailedExecution.joins(:job)
-                                 .where(solid_queue_jobs: { queue_name: queue_name })
-                                 .count
-    end
   end
 end

app/presenters/solid_queue_monitor/stats_presenter.rb

@@ -11,13 +11,12 @@ def render
         <div class="stats-container">
           <h3>Queue Statistics</h3>
           <div class="stats">
-            #{generate_stat_card('Total Jobs', @stats[:total_jobs])}
+            #{generate_stat_card('Active Jobs', @stats[:active_jobs])}
             #{generate_stat_card('Ready', @stats[:ready])}
             #{generate_stat_card('In Progress', @stats[:in_progress])}
             #{generate_stat_card('Scheduled', @stats[:scheduled])}
             #{generate_stat_card('Recurring', @stats[:recurring])}
             #{generate_stat_card('Failed', @stats[:failed])}
-            #{generate_stat_card('Completed', @stats[:completed])}
           </div>
         </div>
       HTML