Good bots vs bad bots: how to manage automated traffic

The World of Automated Web Traffic

Bots are programs that visit websites or call APIs. A useful bot does a wanted job, such as indexing pages or checking uptime. A harmful bot can steal account access, place fake orders or overload a service. Judge each bot by what it does on your site.

What is a Good Bot?

Good bots are automated programs that perform useful or helpful tasks. They typically respect the rules set out in a website's robots.txt file and are transparent about their identity and purpose.

Examples of Good Bots:

• Search Engine Crawlers: These are perhaps the most well-known good bots. Bots like Googlebot, Bingbot, and DuckDuckBot crawl the web to discover and index content, making it discoverable through search engines. Without them, search engines wouldn't work. They identify themselves clearly in their User-Agent string.

  • Example User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

• SEO and Site Monitoring Tools: Services like Ahrefs, SEMrush, and UptimeRobot use bots to crawl websites to analyze SEO performance, check for broken links, or monitor for downtime. These bots help website owners improve their sites.

• Copyright Bots: These bots scan the internet for copyrighted content (e.g., images, music) that is being used without permission, helping creators protect their intellectual property.

• Data Feed Bots: Bots used by aggregators, such as flight or hotel comparison websites, to pull publicly available pricing and availability data via approved APIs or feeds.

Characteristics of a Good Bot:

• Transparency: They have a clear identity and provide a way for website owners to get more information (e.g., a URL in the User-Agent string).
• Respectful: They follow the directives in the robots.txt file.
• Controlled Crawl Rate: They are designed to crawl websites at a reasonable rate to avoid overwhelming the server.
• Provides Value: Their function is beneficial to either the website owner or the broader internet ecosystem.

What is a Bad Bot?

Bad bots are designed to perform malicious, disruptive, or illegal activities. They often disguise their identity, ignore robots.txt, and use sophisticated techniques to evade detection.

Examples of Bad Bots:

• Web Scrapers / Content Scrapers: These bots are programmed to steal content, pricing information, product listings, or other proprietary data from websites. This data can be used to undercut competitors, republish content without permission, or for market analysis.

• Credential Stuffing Bots: These bots take lists of stolen usernames and passwords from data breaches and systematically try them on other websites to perform account takeovers.

• Spam Bots: These bots automatically post spam comments on blogs, forums, and social media, or create fake accounts to send phishing messages.

• Ad Fraud Bots: These bots generate fake clicks or impressions on online advertisements to defraud advertisers and publishers.

• Scalper / Hoarder Bots: Commonly seen in e-commerce and ticketing, these bots are used to automatically buy up limited-stock items (e.g., concert tickets, sneakers) faster than any human could, with the intent of reselling them at a higher price.

• DDoS Bots: These bots are part of a botnet and are used to flood a target server with traffic, causing a Distributed Denial of Service (DDoS) attack that can take a website offline.

Characteristics of a Bad Bot:

• Deceptive: They often spoof their User-Agent string to mimic a legitimate browser or a search engine crawler.
• Disrespectful: They completely ignore robots.txt directives.
• Aggressive: They can make a huge number of requests in a short period, consuming server resources.
• Distributed: They often operate from large, distributed proxy networks (residential or datacenter) to hide their origin and bypass IP-based blocking.
• Harmful: Their actions are detrimental to the website, its users, or its business.

Check the crawler before giving it access

Check the claimed bot name, request rate, routes and results together. For Google crawlers, follow Google's verification steps before granting special access. A user-agent name is easy to copy.

Choose a rule around the task

Allow approved crawlers to reach the public pages they need. Set route and rate controls for busy crawlers. Review repeated login failures, unwanted scraping and requests that make the site slow.

A bot security check can help your team review automated traffic and choose which bots to allow, limit, challenge or block. Track successful user visits as each rule changes.