I found a nice, albeit a bit inaccurate tutorial on centralizing an IP-blacklist, referrer-blacklist and especially user-agent blacklist on a rooted Apache.

Goal is of course to maintain those blacklists only once per real apache installation, with minimum implications on dozens or hundreds of different virtual servers (domains) setup on the instance.

Why block "users" (bot's) out? Because they:

• put your server under load
• scrabe your content to place them in some kind of "search engine" or directory only made to trigger any duplicate content filters or gain own PPC click ads with your content
• place referrer urls in your referrer listing (altough that has no effect after NOFOLLOW tag implementation they do it… and again put your server under load without necessity

My answer: 403 – forbidden :-)

Actually this is a work-around for Apache not being abled to inherit the main server configuration on mod-rewrite from each and every virtual server…

How to centralize administration of things to block is a great start and I set it up to block gazillions of chinese, korean etc. spam bots last night – with a server load from from 20-25 to 3-4 :) that's a protection of my investments, eh :)

the Agents currently blocked:

• Alexibot
• BackDoorbot
• Black.Hole
• BlackWidow
• BlowFish
• BuiltbotTough
• Bullseye
• BunnySlippers
• Cegbfeieh
• Cheesebot
• CherryPicker
• ChinaClaw
• CopyRightCheck
• Crescent
• Custo
• DIIbot
• DISCo
• DittoSpyder
• Download
• DownloadsDemon
• EirGrabber
• EmailCollector
• EmailSiphon
• EmailWolf
• EroCrawler
• Express
• ExpresssWebPictures
• ExtractorPro
• EyeNetIE
• FlashGet
• Foobot
• FrontPage
• GetRight
• GetWeb!
• Gigabot/2.0
• Go!Zilla
• Go-Ahead-Got-It
• GrabNet
• Grafula
• HMView
• HTTrack
• Harvest
• Image
• ImagesStripper
• ImagesSucker
• IndysLibrary
• InfonaviRobot
• InterGET
• InternetSeer.com
• Internet\sNinja
• JOC\sWeb\sSpider
• Jennybot
• JetCar
• Kenjin.Spider
• Keyword.Density
• LeechFTP
• Lexibot
• LinkScan/8.1a.Unix
• LinkWalker
• LinkextractorPro
• MIDown
• MIIxpc
• MSFrontPage
• Mass
• Mata.Hari
• Microsoft.URL
• Mister.PiX
• Mozilla/3.Mozilla/2.01
• NICErsPRO
• NPbot
• Navroad
• NearSite
• NetAnts
• NetMechanic
• NetSpider
• NetZIP
• Net\sVampire
• Octopus
• Offline.Explorer
• Openfind
• PageGrabber
• Pagerabber
• Papa
• Ping
• ProPowerbot/2.14
• ProWebWalker
• QueryN.Metasearch
• RMA
• ReGet
• RepoMonkey
• Siphon
• SiteSnagger
• SlySearch
• SmartDownload
• Spankbot
• SuperBot
• SuperHTTP
• Superbot
• Surfbot
• Szukacz/1.4
• Teleport
• Teleport\sPro
• Telesoft
• The.Intraformant
• TheNomad
• TightTwatbot
• Titan
• True_Robot
• Turnitinbot/1.5
• URLy.Warning
• VCI
• VoidEYE
• WWW-Collector-E
• WWWOFFLE
• Web.Image.Collector
• WebAuto
• WebBandit
• WebCopier
• WebEMailExtrac.*
• WebEnhancer
• WebFetch
• WebGo
• WebLeacher
• WebReaper
• WebSauger
• WebStripper
• WebWhacker
• WebZIP
• WebZip
• WebmasterWorldForumbot
• Website.Quester
• Website\ eXtractor
• Webster.Pro
• Wget
• Widow
• Xaldon\sWebSpider
• Xenu's
• Zeus
• [Ww]eb[Bb]andit
• asterias
• botALot
• cosmos
• eCatch
• hloader
• httplibb
• humanlinks
• ia_archiver
• larbin
• libWeb/clsHTTP
• lwp-trivial
• moget
• pavuk
• pcBrowser
• psbot
• sitecheck.internetseer.com
• sohu-search
• spanner
• suzuran
• tAkeOut
• toCrawl
• turingos
• webcollage

Please let me know if youl know any other malicious agents, ips or referrers