Multi-agent scanning support and other improvements#148
Multi-agent scanning support and other improvements#148colin-stubbs wants to merge 8 commits intoblacklanternsecurity:devfrom
Conversation
packages and related dependencies required so that bbot can operate normally
Introduces a `build.sh` script for multi-platform Docker image builds and optional registry pushes. Refactors the `Dockerfile` to consolidate `RUN` commands into a single layer, reducing image size and layers. Adds Docker Compose files `compose-server.yml` and `compose-agent.yml` to show how to run bbot-server in a more modular, closer to production ready, environment with the ability to scale out/up the number of available bbot agents that can run scans. The `compose-server.yml` externalizes database and message queue dependencies, leveraging environment variables for connection URIs and improving architectural flexibility. It also adds Traefik and autoupdate label examples for better integration with existing infrastructure. The new `compose-agent.yml` provides a template for running `bbot` agents, offering explicit ID configuration and server connectivity options. Improves `.gitignore`
Improves scan agent assignment by ensuring agents are removed from the ready pool after successfully dispatching a scan. Prevents scans from getting stuck by clearing the assigned agent when the `start_scan` command is rejected or if the pinned agent is offline/not ready. Adds a guard to re-fetch agent status if no ready agents remain during dispatch.
Refactors the scan error handling update operation for conciseness. Clarifies comments in the Docker Compose file regarding the injection of custom CA bundles, providing better guidance for future configuration.
Runs a default agent alongside everything else
Ensure sudo available so agent instances of containers can install necessary dependencies etc
|
@colin-stubbs thanks for the PR and welcome back! It's true that the agent functionality has been unreliable and somewhat half-baked. We aren't using it yet in production, but it's likely the agent, preset, and scan dispatching functionality won't survive in the open source version. Of course it'll always be possible to stream events from a scanner directly to BBOT server (via HTTP, websocket, etc.), but we aren't really happy with the existing agent system, as it needs a way to sync the BBOT version between the server and agents, perform updates remotely, pull scan logs, terminate misbehaving modules and scans, etc. This feature will be heavily integrated into the new frontend and because of that, it'll probably be a feature of the upcoming BBOT Pro, which will have affordable pricing akin to Burpsuite Pro. This is the main reason why it hasn't been a focus. The Redis updates are appreciated although TaskIQ has been a pain in the ass and probably will get the ax as well, leaving it up to users to schedule scans, archive jobs, etc. via cron or however they see fit. Overall the goal is to keep the BBOT Server codebase clean and lean, and while we hoped agents would fit the bill, they've turned out to be quite a bit more work than expected, making them an enterprise feature. I'm still happy to merge this, but just as an FYI, some parts may perish during the agent purge! |
2 participants
bbot-server surprisingly still couldn't handle dispatching scans to multiple agents.
Three changes were needed in start_scans_loop() to get dispatching scans to all/any available agent:
Adds compose-server.yml and compose-agent.yml files to help show how to modularise and scale out by running many agents on many hosts.
Adds python3-pip pipx openssl tcpdump packages to container build, this ensures that a variety of dependencies will exist and that installation of packages is not needed and that scanning modules will no longer fail.
Improves Dockerfile to reduce layers and resulting image size.
Connecting to a Redis cluster was not working. Now it does. Example URL now in README.md
Improves .gitignore so that git clients will ignore .env files and local mongodb content.