Use ai-robots-txt agents list

[EchedelleLR]

I was thinking that this special configuration could be tweaked to add tainted user-agents such as the ones listed at https://github.com/ai-robots-txt/ai.robots.txt.

Is a source that can be used for it.

[EchedelleLR]

In my case for this, I define a Mod_Security custom rule (it felt easier for me) and I load the modsecurity2 module into Apache HTTP server.

root@server:/etc/modsecurity/custom$ cat 000-badbots.conf 
SecRule REQUEST_HEADERS:User-Agent "@pmFromFile lists/000-badbots_ai_robots.txt" \
	"id:2,\
	phase:1,\
	deny,\
	status:403,\
	log,\
	msg:'AI bot blocked by User-Agent

I have the following script to download the agents:
sh -c 'curl --silent https://raw.githubusercontent.com/ai-robots-txt/ai.robots.txt/refs/heads/main/robots.json | jq -r "keys[]" > 000-badbots_ai_robots.txt'

The file looks like:

root@server:/etc/modsecurity/custom/lists$ head 000-badbots_ai_robots.txt 
AI2Bot
AI2Bot-DeepResearchEval
AddSearchBot
Ai2Bot-Dolma
AmazonBuyForMe
Amazonbot
Andibot
Anomura
Applebot
Applebot-Extended

In security2.conf file I make a little tweak to include my custom rules.

root@server:/etc/modsecurity/custom/lists$ cat /etc/apache2/mods-enabled/security2.conf 
<IfModule security2_module>
	# Default Debian dir for modsecurity's persistent data
	SecDataDir /var/cache/modsecurity

	# Include all the *.conf files in /etc/modsecurity.
	# Keeping your local configuration in that directory
	# will allow for an easy upgrade of THIS file and
	# make your life easier
        IncludeOptional /etc/modsecurity/*.conf

	# Include OWASP ModSecurity CRS rules if installed
	#IncludeOptional /usr/share/modsecurity-crs/*.load

	# Include custom ModSecurity rules
	IncludeOptional /etc/modsecurity/custom/*.conf
</IfModule>