[Bug]: Jobs are not being processed in a long running sqlite server

[IgnisDa]

What happened?

Hi! I recently updated apalis from 0.7.4 to 1.0.0-rc.4 and have been noticing reports on Ryot's github and discord about jobs not being processed (example). I also switched the job backend from memory to sqlite. This was done because the rc versions don't allow cloning the MemoryStorage backend.

Earlier I thought that it was because the queue was being choked by a lot of jobs being added to it (which is by design). This queue has a concurrency of 1. To that end, I "sharded" the queue so that more jobs are processed in parallel (commit).

But people are still reporting this issue, so this was not the fix. I am stumped on what may be the actual issue.

Any suggestions on what I can do to debug this issue of jobs stopped being processed after some time?

Expected behavior

Jobs to be processed correctly.

Steps to reproduce

I have a standard apalis setup (https://github.com/IgnisDa/ryot/blob/2074b8e72d295a9bd4e3deaffdc2fd8812a802aa/apps/backend/src/main.rs#L187).

Minimal code example

Version

1.0.0-rc.x

Environment

• OS: Any
• Rust version: 1.93.1
• Cargo version: 1.93.1

Relevant log output

Additional context

No response

[IgnisDa]

Here is the commit when apalis was updated from 0.7.4 to rc-* version: IgnisDa/ryot@6adbd15.

[geofmureithi]

Could you do a dump of an sqlite file for one of the hanging queues, this can help me look at what is exactly hanging. I will move this issue to apalis-sqlite repo because I feel this is sqlite specific. Before I move it though I would also ask you to try JsonStorage just to help confirm its an sqlite issue.

…

On Tue, 3 Mar 2026 at 04:04, Diptesh ***@***.***> wrote: *IgnisDa* left a comment (apalis-dev/apalis-sqlite#58) <#58 (comment)> Here is the commit when apalis was updated from 0.7.4 to rc-* version: ***@***.*** <IgnisDa/ryot@6adbd15> . — Reply to this email directly, view it on GitHub <#58 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AWXVRGQU65DTMG53FYECDRL4OYVTFAVCNFSM6AAAAACWE3HWGSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTSOBXHE2DKMZQGU> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[IgnisDa]

Hi @geofmureithi. I have forwarded you the db file on Discord.

Something I want to mention which makes me believe this is an apalis issue: I have set the db path to /tmp/ryot_jobs in the ryot container. And this is not mounted to a volume so every time the container is restarted, the file is made afresh. This is by design since I don't need the jobs to be persisted across restarts.

The reason I mention this is that users are reporting that everything starts working fine after the container is restarted.

[IgnisDa]

Just wanted to update this is indeed a sqlite issue. I switched over to memory storage in this PR: IgnisDa/ryot#1722

Additionally I tried json storage and it errored out when there were a lot of concurrent requests:

[geofmureithi]

Could you try using the SharedSqliteStorage here?

for (shard_idx, single_application_job_storage) in
            single_application_job_storages.into_iter().enumerate()
        {
            let shard_ss = supporting_service.clone();
            monitor = monitor.register(move |_runs| {
                WorkerBuilder::new(format!("perform_single_application_job_{shard_idx}"))
                    .backend(single_application_job_storage.clone())
                    .catch_panic()
                    .enable_tracing()
                    .concurrency(1)
                    .data(shard_ss.clone())
                    .build(perform_single_application_job)
            });
        }