High CPU usage on v1.7.8

[hanjo]

Hi,

I'm running crowdsec in docker on two Linux (Ubuntu) servers. After updating to v1.7.8, the CPU usage on both servers roughly doubled to tripled.

There is nothing of relevance in the logs.

After moving back to v1.7.7, CPU usage dropped to expected values.

Based on the release notes, I cannot pinpoint the issue. Any suggestions?

Thanks!

[github-actions]

@hanjo: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

1. Check Crowdsec Documentation to see if your issue can be self resolved.
2. You can also join our Discord.
3. Check Releases to make sure your agent is on the latest version.

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[github-actions]

@hanjo: There are no 'kind' label on this issue. You need a 'kind' label to start the triage process.

• /kind feature
• /kind enhancement
• /kind refactoring
• /kind bug
• /kind packaging

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[hanjo]

/kind bug

[blotus]

Hello,

Could you give more informations about the servers, CPU type, ...

What was the baseline CPU usage ? Did you went from 10% to 30% or from 2% to 6% ?

How many bouncers do you have ? We made a change in 1.7.8 to improve the response time and greatly reduce the memory consumption of the decisions stream, but this comes at the cost of slightly higher CPU usage on LAPI.

Could you as well provide a CPU profile of crowdsec ?
Assuming you run the command on the machine where crowdsec is running:
curl http://localhost:6060/debug/pprof/profile -o profile.gz.

The command will take 30s to run.

[hanjo]

Thanks for the quick response.
Server A (Intel Core i5-9500T CPU @ 2.20GHz, 6 cores) went from 18-19% to 38-42%, memory dropped from 91-94MB to 55-57MB.
Server B (some VPS, using QEMU Virtual CPU version 2.5+, 4 cores) went from 10-11% to 40-47%, memory dropped from 130-450MB to 105-390MB.

On Server A I have 12 bouncers, but only 5 are active (there are duplicates due to multiple source IP addresses).
On Server B I have 4 bouncers (all active).

I have attached the CPU profiles, but I created them on v1.7.7 now, since I rolled back. Do you need them from v1.7.8, or doesn't it matter?

profile-serverA.gz
profile-serverB.gz

[tibouh]

Same issue on my side.
Moreover crowdsec_web_ui cannot connect saying CrowdSec LAPI is offline.
Reverted to v1.7.7 and it's working again.

[blotus]

Thanks for the quick response. Server A (Intel Core i5-9500T CPU @ 2.20GHz, 6 cores) went from 18-19% to 38-42%, memory dropped from 91-94MB to 55-57MB. Server B (some VPS, using QEMU Virtual CPU version 2.5+, 4 cores) went from 10-11% to 40-47%, memory dropped from 130-450MB to 105-390MB.

On Server A I have 12 bouncers, but only 5 are active (there are duplicates due to multiple source IP addresses). On Server B I have 4 bouncers (all active).

I have attached the CPU profiles, but I created them on v1.7.7 now, since I rolled back. Do you need them from v1.7.8, or doesn't it matter?

profile-serverA.gz profile-serverB.gz

I need them on 1.7.8 to see where the time is being spent. But since you have 5 bouncers, I'm pretty sure the difference is coming from the stream change.

[hanjo]

I suppose this makes sense 🙃

Here you go:
profile-serverA-v178.gz
profile-serverB-v178.gz

[baragoon]

Same for me, crowdsec-web-ui can't connect to LAPI, a lot of "database is locked" + 500's errors in the logs and constant ~100% CPU usage. Was running fine 1.7.7 on RPI 4.

[jimstrang]

Seeing the same on v1.7.8 (Docker, TrueNAS SCALE). LAPI returns database is locked errors and 500s under normal load — one firewall bouncer polling every 10s plus a Homepage dashboard widget hitting /v1/alerts. CPU spikes and the SQLite DB locks up within minutes of a restart.

Logs show the localhost machine entry flip-flopping between 127.0.0.1 (internal healthcheck) and the Docker bridge IP on every request, triggering a DB write each time (new IP address detected for machine 'localhost'). This compounds the contention from the stream change.

Pinned back to v1.7.7 and the issue is resolved.

[jimstrang]

Update: Migrated from SQLite to PostgreSQL (pgx driver) and v1.7.8 runs without issues. The chunked decisions stream (#4413) holds connections open while iterating batches, and concurrent operations (bouncer polls, heartbeats, alert inserts) compete for SQLite's single write lock. Postgres handles this natively via MVCC — no lock contention at all on the same code.