Bots account for a large share of web traffic. Recent studies put automated traffic at nearly 50% of all internet requests. Some bots are useful, such as search engine crawlers that index your site. Some are clearly harmful, such as scrapers and sneaker bots. Others sit in a grey area, including backlink and marketing bots from services such as Ahrefs and SEMrush. Even useful bots can create problems when they crawl too hard. This article looks at the main bot types and how to manage them with robots.txt and bot management tools.
Understanding the Different Types of Bots
'Good Bots'
Good bots perform legitimate work. Search engine crawlers like Googlebot and Bingbot index webpages so search results can stay current and relevant. Other examples include uptime and performance monitoring bots.
'Bad Bots'
Bad bots harm websites, users, or both. Common examples include:
- Scraping content, copying and repurposing data from websites.
- Sneaker bots, automatically purchasing limited-edition products (like sneakers) before human users can.
- Spam bots, posting unsolicited messages and advertisements in comment sections or forums.
- Vulnerability Scanners, trying thousands of website URLs to find security vulnerabilities.
- Account Takeover, attempting to gain access to existing user/admin accounts using either credential stuffing or brute-force attacks.
'Grey Bots'
Grey bots sit between good and bad. They often serve a useful purpose and may follow crawling directives in robots.txt, but they can still cause problems when they crawl too aggressively. Common examples include:
- AhrefsBot: A backlink analysis bot used by Ahrefs, an SEO tool.
- SEMrushBot: A bot used by SEMrush, another popular SEO and digital marketing tool.
- MJ12bot: A bot used by Majestic, a service that provides backlink data and analysis.
- ScreamingFrog: An SEO analyser run from a local desktop.
When Grey bots (and even Good Bots) go bad.
Left unattended, grey bots can create practical problems:
- Slow page loading times, which affect user experience.
- Strain on server resources, potentially causing crashes, downtime, and higher costs.
- Distorted website analytics, when bot traffic is mistaken for human traffic.
Managing Grey Bots with Robots.txt
The robots.txt file is a simple text file that tells web crawlers which parts of your site they can or cannot access. You can use it to manage bot behaviour and protect your website from aggressive crawling. Useful controls include:
Disallowing specific bots: You can block specific bots from accessing your site by adding a "User-agent" and "Disallow" directive to your robots.txt file. For example:
User-agent: AhrefsBot
Disallow: /
Limiting crawl rate: You can ask bots to slow down their crawling by adding a "Crawl-delay" directive:
User-agent: SEMrushBot
Crawl-delay: 10
Not all bots will follow robots.txt. ScreamingFrog, for example, can be instructed to ignore robots.txt and crawl a site as quickly as possible. You would not want a competitor doing this to your site.
Bot Management Tools
In addition to robots.txt, bot management tools (like those provided by Peakhour) can protect your website from abusive bots. Good bot management tools automatically block most unwanted traffic using a combination of Threat Intelligence, Fingerprinting techniques, Reverse DNS verification, and Header Inspection.
Advanced techniques like rate limiting and machine learning can help identify more sophisticated bad bots.
Search Bots and Double Crawling
Search bots like Bingbot can sometimes blindly follow links and crawl the same page multiple times due to different URL parameters. This double, triple, or worse crawling can increase server load and make indexing less efficient. eCommerce sites are especially exposed because product catalogues often have several filtering paths. We've seen Bing go haywire on a number of sites. Most recently, it was issuing around 50,000 requests per day to the search function of a Magento 2 store while cycling through parameters. This dropped to 2-3k requests per day when fixed. On another store, Bing was responsible for nearly half of all page requests (40k page requests) on a busy OpenCart store. Configuring it to ignore parameters dropped this to around 4k per day.
Control the crawl path that exists now
Google retired Search Console's URL Parameters tool in 2022. Bing removed its equivalent control too. Do not follow old instructions that send you hunting through either webmaster console.
Google says its crawler now learns how URL parameters behave automatically. When that is not enough, use robots.txt rules, canonical URLs and clean internal links to make the intended crawl path clear. Our worked example shows how to exclude query-string combinations with robots.txt. Google's own notice explains why it retired the URL Parameters tool.
Check the result in access logs. The useful measure is not whether a rule exists; it is whether duplicate parameter combinations stop consuming requests while the pages you want indexed remain crawlable.
Watch the request volume
When good or grey bots crawl too aggressively, they can cause the same operational problems as malicious bots: overloaded servers, slower pages, and worse user experience. Monitor crawler request volume and server load, set narrow robots.txt rules, then verify the change in logs rather than assuming the crawler behaved as intended.
