Management of Excessive Bot Crawling on Our WordPress Site

  • Resolved theginaddict

    (@theginaddict)


    3 weeks, 6 days ago

    Hello,

    We are currently experiencing server load issues due to an excessive number of crawler bots accessing our WordPress site. Despite our efforts to block some bots using the robots.txt file and the Black Hole plugin, we continue to experience an overload of unwanted traffic. Our hosting provider has also alerted us to this issue, noting that server monitoring had to be disabled due to recurring false alerts.

    Could you please advise on any additional actions I could take?

    Thank you in advance,
    Benjamin

    The page I need help with: [log in to see the link]

Viewing 8 replies - 1 through 8 (of 8 total)
  • akamarski

    (@akamarski)

    3 weeks, 6 days ago

    Hello Benjamin,

    You can put the domain behind the free plan of Cloudflare, which should handle a part of the bot traffic out of the box. https://www.cloudflare.com/en-gb/plans/free/

    Moderator James Huff

    (@macmanx)

    Volunteer Moderator

    3 weeks, 6 days ago

    Have you also tried blocking their user agents and/or IPs as mentioned earlier? https://wordpress.org/support/topic/surge-in-requests-and-wordpress-crash-2/#post-17758136

    rade1

    (@rade1)

    3 weeks, 6 days ago

    Hi,
    You can try using plugins to block Bad Bots requests and reduce load on server, I did use both of this in past, and you can test each to see what works best for you
    https://wordpress.org/plugins/blackhole-bad-bots/ (it worked for me on almost every case)
    https://wordpress.org/plugins/stopbadbots/

    Thread Starter theginaddict

    (@theginaddict)

    3 weeks, 6 days ago

    Hello,
    Thanks for that.
    I already have blackhole bad bots but i do not have stop bad bots.
    I see that i need a bit of conguration.
    Can you help me with that for stop bad bots plugin ?

    Thread Starter theginaddict

    (@theginaddict)

    3 weeks, 6 days ago

    Yes i already try this and i already block theses :

    Negative Robots Blocking

    User-agent: Abonti
    Disallow: /

    User-agent: BDCbot
    Disallow: /

    User-agent: BLEXBot
    Disallow: /

    User-agent: BuiltBotTough
    Disallow: /

    User-agent: Bullseye
    Disallow: /

    User-agent: BunnySlippers
    Disallow: /

    User-agent: CCBot
    Disallow: /

    User-agent: Cegbfeieh
    Disallow: /

    User-agent: CheeseBot
    Disallow: /

    User-agent: CherryPicker
    Disallow: /

    User-agent: CherryPickerElite/1.0
    Disallow: /

    User-agent: CherryPickerSE/1.0
    Disallow: /

    User-agent: CopyRightCheck
    Disallow: /

    User-agent: Cosmos
    Disallow: /

    User-agent: Crescent
    Disallow: /

    User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
    Disallow: /

    User-agent: DOC
    Disallow: /

    User-agent: DittoSpyder
    Disallow: /

    User-agent: DotBot
    Disallow: /

    User-agent: Download Ninja
    Disallow: /

    User-agent: EasouSpider
    Disallow: /

    User-agent: EmailCollector
    Disallow: /

    User-agent: EmailSiphon
    Disallow: /

    User-agent: EmailWolf
    Disallow: /

    User-agent: EroCrawler
    Disallow: /

    User-agent: Exabot
    Disallow: /

    User-agent: ExtractorPro
    Disallow: /

    User-agent: Fasterfox
    Disallow: /

    User-agent: FeedBooster
    Disallow: /

    User-agent: Foobot
    Disallow: /

    User-agent: Genieo
    Disallow: /

    User-agent: HTTrack
    Disallow: /

    User-agent: Harvest
    Disallow: /

    User-agent: Incutio
    Disallow: /

    User-agent: InfoNaviRobot
    Disallow: /

    User-agent: IstellaBot
    Disallow: /

    User-agent: Java/1.
    Disallow: /

    User-agent: JennyBot
    Disallow: /

    User-agent: Kenjin Spider
    Disallow: /

    User-agent: Keyword Density/0.9
    Disallow: /

    User-agent: LNSpiderguy
    Disallow: /

    User-agent: LexiBot
    Disallow: /

    User-agent: LinkScan/8.1a Unix
    Disallow: /

    User-agent: LinkWalker
    Disallow: /

    User-agent: LinkextractorPro
    Disallow: /

    User-agent: MIIxpc
    Disallow: /

    User-agent: Mata Hari
    Disallow: /

    User-agent: MaxPointCrawler
    Disallow: /

    User-agent: MegaIndex
    Disallow: /

    User-agent: MJ12bot
    Disallow:/

    User-agent: MSIE Crawler
    Disallow: /

    User-agent: mozilla/4
    Disallow: /

    User-agent: Mozilla/4.0 (compatible; BullsEye; Windows 95)
    Disallow: /

    User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 95)
    Disallow: /

    User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 98)
    Disallow: /

    User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows NT)
    Disallow: /

    User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows XP)
    Disallow: /

    User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 2000)
    Disallow: /

    User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows ME)
    Disallow: /

    User-agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; SV1)
    Disallow: /

    User-agent: mozilla/5
    Disallow: /

    User-agent: Mozilla/5.0
    Disallow: /

    User-agent: Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)
    Disallow: /

    User-agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
    Disallow: /

    User-agent: Mozilla/5.0 (compatible; YandexAccessibilityBot/3.0; +http://yandex.com/bots)
    Disallow: /

    User-agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1
    Disallow: /

    User-agent: Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0)
    Disallow: /

    User-agent: Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
    Disallow: /

    User-agent: Microsoft URL Control
    Disallow: /

    User-agent: Mippin
    Disallow: /

    User-agent: Missigua Locator
    Disallow: /

    User-agent: Mister PiX
    Disallow: /

    User-agent: NICErsPRO
    Disallow: /

    User-agent: NPBot
    Disallow: /

    User-agent: NetAnts
    Disallow: /

    User-agent: Niki-Bot
    Disallow: /

    User-agent: Nutch
    Disallow: /

    User-agent: Offline Explorer
    Disallow: /

    User-agent: Openfind
    Disallow: /

    User-agent: ProPowerBot/2.14
    Disallow: /

    User-agent: ProWebWalker
    Disallow: /

    User-agent: Python-urllib
    Disallow: /

    User-agent: QueryN Metasearch
    Disallow: /

    User-agent: RMA
    Disallow: /

    User-agent: RepoMonkey
    Disallow: /

    User-agent: SISTRIX
    Disallow: /

    User-agent: SemrushBot-BM
    Disallow: /

    User-agent: SemrushBot-CT
    Disallow: /

    User-agent: SemrushBot-SWA
    Disallow: /

    User-agent: SemrushBot-SI
    Disallow: /

    User-agent: SemrushBot-BA
    Disallow: /

    User-agent: SeznamBot
    Disallow: /

    User-agent: SiteAuditBot
    Disallow: /

    User-agent: SiteSnagger
    Disallow: /

    User-agent: SnapPreviewBot
    Disallow: /

    User-agent: Sogou
    Disallow: /

    User-agent: SpankBot
    Disallow: /

    User-agent: Spinn3r
    Disallow: /

    User-agent: Szukacz/1.4
    Disallow: /

    User-agent: Teleport
    Disallow: /

    User-agent: Telesoft
    Disallow: /

    User-agent: The Intraformant
    Disallow: /

    User-agent: TheNomad
    Disallow: /

    User-agent: TightTwatBot
    Disallow: /

    User-agent: Titan
    Disallow: /

    User-agent: True_Robot
    Disallow: /

    User-agent: TurnitinBot
    Disallow: /

    User-agent: URLy Warning
    Disallow: /

    User-agent: UbiCrawler
    Disallow: /

    User-agent: UnisterBot
    Disallow: /

    User-agent: Unknown
    Disallow: /

    User-agent: VCI
    Disallow: /

    User-agent: WBSearchBot
    Disallow: /

    User-agent: WWW-Collector-E
    Disallow: /

    User-agent: Web Downloader/6.9
    Disallow: /

    User-agent: Web Image Collector
    Disallow: /

    User-agent: WebAuto
    Disallow: /

    User-agent: WebBandit
    Disallow: /

    User-agent: WebCopier
    Disallow: /

    User-agent: WebEnhancer
    Disallow: /

    User-agent: WebReaper
    Disallow: /

    User-agent: WebSauger
    Disallow: /

    User-agent: WebStripper
    Disallow: /

    User-agent: WebZip
    Disallow: /

    User-agent: WebmasterWorldForumBot
    Disallow: /

    User-agent: Website Quester
    Disallow: /

    User-agent: Webster Pro
    Disallow: /

    User-agent: Wotbox
    Disallow: /

    User-agent: Xenu
    Disallow: /

    User-agent: YandexAccessibilityBot
    Disallow: /

    User-agent: YandexAdNet
    Disallow: /

    User-agent: YandexBlogs
    Disallow: /

    User-agent: YandexBot
    Disallow: /

    User-agent: YandexCalendar
    Disallow: /

    User-agent: YandexDirect
    Disallow: /

    User-agent: YandexDirectDyn
    Disallow: /

    User-agent: YandexFavicons
    Disallow: /

    User-agent: YaDirectFetcher
    Disallow: /

    User-agent: YandexForDomain
    Disallow: /

    User-agent: YandexImages
    Disallow: /

    User-agent: YandexImageResizer
    Disallow: /

    User-agent: YandexMarket
    Disallow: /

    User-agent: YandexMedia
    Disallow: /

    User-agent: YandexMetrika
    Disallow: /

    User-agent: YandexMobileBot
    Disallow: /

    User-agent: YandexMobileScreenShotBot
    Disallow: /

    User-agent: YandexNews
    Disallow: /

    User-agent: YandexOntoDB
    Disallow: /

    User-agent: YandexPagechecker
    Disallow: /

    User-agent: YandexRCA
    Disallow: /

    User-agent: YandexRenderResourcesBot
    Disallow: /

    User-agent: YandexSitelinks
    Disallow: /

    User-agent: YandexSpravBot
    Disallow: /

    User-agent: YandexTracker
    Disallow: /

    User-agent: YandexTurbo
    Disallow: /

    User-agent: YandexUserproxy
    Disallow: /

    User-agent: YandexVertis
    Disallow: /

    User-agent: YandexVerticals
    Disallow: /

    User-agent: YandexVideo
    Disallow: /

    User-agent: YandexVideoParser
    Disallow: /

    User-agent: YandexWebmaster
    Disallow: /

    User-agent: YandexScreenshotBot
    Disallow: /

    User-agent: Zao
    Disallow: /

    User-agent: Zeus
    Disallow: /

    User-agent: ZyBORG
    Disallow: /
    Bill Minozzi

    (@sminozzi)

    3 weeks, 6 days ago

    Hi theginaddict

    After installing the Stop Bad Bots plugin, go to the Settings Page and open the General Settings Tab. Check Yes for all options. Then, visit the plugin’s Dashboard and monitor the results. The plugin also features a visitor table and an analytics page, showing your real visits separate from bots.

    Regards,

    Bill, Plugin Developer, Stop Bad Bots.

    Moderator James Huff

    (@macmanx)

    Volunteer Moderator

    3 weeks, 5 days ago

    As I mentioned in the other thread:

    First, keep in mind that robots.txt is more like guidelines than actual rules. The file is important, because good bots (like spiders from reputable search engines) will respect it, but there is no requirement for bots to respect it.

    https://wordpress.org/support/topic/surge-in-requests-and-wordpress-crash-2/#post-17758136

    If you really want to block user agents, you need to do it in .htaccess: https://www.siteground.com/kb/htaccess-block-user-agent/

    Thread Starter theginaddict

    (@theginaddict)

    3 weeks, 5 days ago

    Hello ,

    Thanks for both answers.
    I did it and i will see the results.

Viewing 8 replies - 1 through 8 (of 8 total)
  • You must be logged in to reply to this topic.