Peakhour.IO - DNS

Cloudflare outage proves Plan B depends on controlling DNS

2025-11-18T21:00:00+11:00

On Tuesday, 18 November 2025, Cloudflare’s own status page marked every major service—CDN, Firewall, WARP, Workers, and the dashboard—as degraded for most of the day while engineers worked through an internal control-plane failure. The timeline moved from “Investigating” at 11:48 UTC to “Monitoring” after 14:42 UTC, and the incident wasn’t officially resolved until 19:28 UTC. During the worst of it, Cloudflare disabled WARP in London, bot scores seesawed, and customers were told to wait while remediation continued.

Waiting was the only option for many teams because their Plan B lived behind the same dashboard that was timing out. The top comment on the Hacker News thread was a set of curl commands for moving domains off Cloudflare’s proxy edge. Admins were stuck in 2FA flows trying to fetch an API token, or searching for Terraform credentials so they could toggle a proxied flag. That is not a resilience strategy.

We learned this lesson the hard way—and wrote about it after the 2021 Fastly outage in How to have a Plan B. The rule still stands: the platform you are trying to leave cannot be the only place that can change where your DNS points.

Detect: understand what’s actually broken

Incidents like Tuesday’s change shape quickly. Cloudflare’s own feed showed different failure domains every 30 minutes: bot management, dashboard auth, Access, WARP. The first mile is impartial telemetry that tells you what your users feel, not what the provider thinks. At Peakhour we stream real user monitoring, synthetic checks, and control-plane health from multiple CDNs and DNS partners. That lets us distinguish “cache errors in Hong Kong” from “global auth outage” and choose the right lever.

Decide: keep DNS authority in neutral territory

When your domain delegation lives with agnostic providers—Route 53, NS1, Azure DNS, or the enterprise registrar your legal team already approved—you can make failover decisions without pleading with a failing control plane. Peakhour doesn’t replace those vendors; we orchestrate them. We set short-but-safe TTLs, keep secondary answers staged, and continuously audit API access so we can flip traffic with one signed request. The minute you outsource DNS authority to a proxy CDN, you have given up the control that makes Plan B possible.

Divert: run the playbook in minutes, not hours

A workable Plan B has three moves:

Pre-stage alternate edges. Your secondary CDN, origin, or transit provider must be in sync with the active one—certificates, cache rules, WAF policies, everything. We keep them hot by replaying production configs across vendors.
Wire DNS automation. We integrate with multiple third-party DNS APIs at once so we can update apex A/AAAA, flattened CNAMEs, and geo/latency rules in a single workflow. Because the automation lives off the impacted platform, we can execute even while Cloudflare’s dashboard is returning 500s.
Drill humans on the handoff. Our SOC sits in Sydney and Melbourne, but we cover global hours. During an incident we line up Slack/Teams bridges with your SREs, confirm business impact, and keep execs in the loop while traffic drains to the healthy provider.

With that in place we routinely hit sub-five-minute diversion times, including DNS propagation, because the decision, the tooling, and the people are ready before the outage hits.

What Peakhour brings to your Plan B

Independent authority, familiar vendors. We leverage multiple established DNS providers instead of locking you into ours. You keep your contracts; we bring the automation and guardrails.
Unified multi-CDN config. Cache rules, image optimisation, WAF, and routing policies stay aligned across providers so you don’t lose capabilities when you switch.
Real drills, not just runbooks. Quarterly failover exercises prove that certificates, APIs, and humans are ready. We share the post-mortems so your execs see clear RTO/RPO numbers.
People you can phone. 24×7 Australian-based engineers who know your stack and can execute the play while your own team communicates with customers.

Book a resilience review

If Tuesday exposed that your failover path still depends on your primary provider’s dashboard, book a 30-minute Resilience Review with Peakhour and we’ll:

Map who really controls your DNS today.
Identify the gaps between your primary and standby CDNs.
Outline the automations we can layer on top of your existing DNS and hosting vendors.

The output is a concrete Plan B, a drill schedule, and a team that can execute it the next time a global provider blinks.

Anatomy of a Credential Stuffing Attack

2025-09-01T00:00:00+10:00

In early 2024, major Australian retailer The Iconic was hit by a widespread account takeover attack. Fraudsters used stolen credentials to log into customer accounts, place orders with stored credit cards, and ship goods to different locations. The incident caused significant reputational damage and financial loss, forcing the company to issue refunds and publicly address the security breach.

This attack wasn't the result of a direct hack on The Iconic's systems. It was a classic case of credential stuffing: an automated attack that works because people reuse passwords across services. This article breaks down how credential stuffing works, the attacker's toolkit, the business impact, and the controls that make it harder to run at scale.

What is Credential Stuffing?

Credential stuffing is an automated attack where malicious actors use lists of stolen usernames and passwords—often obtained from third-party data breaches—to gain unauthorised access to user accounts on other websites. The attack works because many users recycle the same password across multiple online services. If a password for a user's social media account is leaked, attackers will "stuff" that same email and password combination into the login forms of e-commerce sites, banking portals, and other high-value targets.

Because attackers submit valid credentials, even though they are stolen, these login attempts can be difficult to distinguish from genuine user activity. That makes credential stuffing harder for traditional security controls to spot.

The Attacker's Toolkit

Modern credential stuffing is not a manual process. Attackers use a mature set of tools and resources to automate and scale their campaigns:

Automation Software: Tools like OpenBullet are central to these attacks. OpenBullet is a powerful, open-source web testing suite that allows even non-programmers to create complex attack scripts. Attackers can find or create "configs" that tell the software exactly how to interact with a target website's login form.
Breached Credential Lists: Dark web markets carry massive databases of usernames and passwords harvested from data breaches. These "combo lists" are the raw material for credential stuffing attacks and can be purchased for very little cost.
Proxy Networks: To avoid being blocked, attackers distribute their login attempts across thousands or even millions of IP addresses. They often use residential proxy networks, which route traffic through the internet connections of real home users. This can make malicious traffic appear to come from legitimate customers, weakening IP-based blocking and rate limiting.

The Business Impact

The consequences of a successful credential stuffing attack extend beyond the login event:

Direct Financial Loss: As seen with The Iconic, attackers can make fraudulent purchases, drain loyalty points, or transfer funds, leading to direct financial losses and the cost of refunding customers.
Damage to Brand Reputation: Publicly reported breaches erode customer trust. Users who have been defrauded may share their negative experiences on social media, leading to lasting reputational harm.
Loss of Customer Trust: When customers believe their accounts are not secure, they may abandon the platform altogether, leading to customer churn and a decline in lifetime value.
Operational Costs: Responding to an attack involves significant operational overhead, including customer support time, fraud investigation, and new security measures.

Building a Multi-Layered Defense

Stopping automated attacks requires a defence strategy that goes beyond simple password policies. A modern, multi-layered approach should include:

Advanced Bot Protection: The first step is to distinguish bots from humans. Modern bot management uses techniques like network and browser fingerprinting, proxy context, route behaviour, and behavioural analysis to detect automated login attempts, even when they mimic human behaviour.
Check Credentials Against Breach Databases: Proactively check usernames and passwords used in login attempts against comprehensive databases of known breached credentials. If a credential pair is known to be compromised, you can flag the login for additional verification or alert the user to change their password.
Advanced Rate Limiting: Traditional IP-based rate limiting struggles against distributed attacks. Advanced rate limiting groups requests by more stable identifiers, such as a TLS fingerprint, which can remain consistent even as an attacker rotates through thousands of IP addresses. This helps track and block a single malicious actor launching a distributed attack.
Enforce Multi-Factor Authentication (MFA): MFA is not a silver bullet, but it provides a critical layer of security by requiring a second form of verification. Websites should strongly encourage or enforce MFA, especially for sensitive actions like changing account details or making purchases.

For production teams, the useful bot-management view ties those controls together on the login path. Credential exposure, residential proxy evidence, fingerprints, request rate, response codes, and session changes should feed one decision about whether to allow, challenge, rate limit, block, or review the attempt.

By combining these controls, organisations can make credential stuffing harder to scale, protect user accounts, and reduce the business risk when attackers test stolen credentials.

Why We Can't Trust IP Addresses

2025-03-11T14:00:00+11:00

Blocking bad traffic by checking an IP address used to be a reasonable starting point. It is not enough anymore. The rise of residential proxies, especially mobile proxies like those from Proxidize, has weakened one of the simpler assumptions in web security: that an IP address tells you much about who is behind a request.

Why is this a problem now?

Residential proxies route traffic through real household IP addresses, so requests look as if they come from normal homes rather than data centres. Companies like Proxidize have made mobile proxy setups accessible using Android phones or USB modems.

In my presentations at AISA and other security conferences, I've described these proxies as systems that "masquerade internet usage as originating from residential and office networks," because they sit outside the assumptions used by many security controls.

What has changed recently is access. Proxidize offers kits that let anyone set up a proxy farm - from 5-modem kits at $499 to 80-modem setups for around $6,000. They have turned proxy farming into a plug-and-play system where you can be up and running "in less than 60 seconds."

The scale is large. Proxidize users process an estimated 80 billion records combined every single day: 80B+ Records Scraped Daily.

The model is also being sold as a "passive income opportunity," where people can earn money by setting up proxy farms and selling access to others. In their recent webinar, they announced plans for a "Proxidize Grid" marketplace where users can sell their proxies with "a single click through an automated Marketplace."

The BYOD mobile proxy revolution

Companies like iProxy.online have taken this further with a Bring Your Own Device (BYOD) approach. Rather than requiring specialised hardware, they let customers turn any Android device into a mobile proxy.

As Sabir, the cofounder of iProxy.online, explained in a recent interview, "You can install iProxy app here and in the dashboard you have proxy access like Socks5, HTTP accesses, and traffic goes through your device."

This means anyone with an old Android phone and a SIM card can create their own mobile proxy, lowering the barrier to entry. For around $59 per month (based on Proxidize's pricing), users get access to what Sabir calls "precious" mobile IP addresses.

Why are mobile IPs so valuable? As Sabir explains: "If you have Barcelona, we are here in Barcelona and you have like 2 million people living there and you have like several thousands of IP addresses from your mobile providers. And one IP address is shared by many. By thousands of people... And if you have mobile IP address, this cannot be blocked by Facebook or Instagram or any other services because in this case, like innocent people, like thousands of them will be blocked."

This carrier-grade NAT (CGNAT) technology means mobile IP addresses are shared across thousands of users, making broad IP blocks difficult without affecting legitimate users.

What this enables attackers to do

With residential proxies, attackers can:

Hide behind legitimate IP addresses that security systems trust
Bypass geo-restrictions to attack from what appears to be a local source
Distribute attacks across thousands of residential IPs to avoid detection
Make malicious traffic look like it comes from normal users

In my work at Peakhour.IO, we've seen a rise in attacks originating from these residential proxies. The Chinese state-sponsored group Camaro Dragon showed the potential of the model when they developed custom firmware for TP-Link routers, turning them into residential proxies for their operations. This method let them bypass traditional defences like GeoIP blocking because the traffic appeared to come from normal homes.

The broader trend is commoditisation. You no longer need to be a nation-state actor to use them. Anyone with a few hundred dollars can set up a residential proxy farm or use services like iProxy.online to route their traffic through mobile networks.

How it enables data exfiltration

Data exfiltration is harder to detect when residential proxies are involved. State-sponsored actors like Volt Typhoon have used compromised network devices to "proxy all network traffic to targets through compromised SOHO network edge devices."

This means stolen data travels through home routers or office equipment before reaching the attacker, making it harder to trace. Since the traffic appears to come from thousands of different legitimate sources, traditional data loss prevention tools struggle to identify and block the exfiltration.

I've worked with organisations that have suffered breaches where data was exfiltrated through residential proxies. In these cases, the traffic blended in with normal home user traffic, making it difficult to detect. These weren't sophisticated nation-state attacks - they were conducted by ordinary cybercriminals using commercially available residential proxy services.

How it enables credential stuffing and other attacks

Credential stuffing attacks have hit Australian businesses hard, with companies like The Iconic, Guzman y Gomez, Dan Murphy's, and others falling victim. Residential proxies help these attacks work because attackers can distribute their login attempts across thousands of residential IP addresses.

When an attack comes through residential proxies, each login attempt appears to come from a different legitimate user. IP-based rate limiting fails because no single IP shows suspicious volume. Even when security teams try to block suspicious regions, proxies let attackers appear to be local customers.

According to our research at Peakhour.IO, traditional IP intelligence services are failing to detect these proxies. Tests we conducted showed that top providers like Maxmind detected 0% of residential proxies, while even the best performer, IP Quality Score, only identified 24%.

The traffic share can be significant. We've seen cases where up to 40% of traffic to Australian e-commerce sites consists of bots using residential proxies for credential stuffing, price scraping, and inventory checking. This puts customer accounts at risk, distorts analytics, and wastes marketing budgets on fake traffic.

The TCP/IP fingerprinting challenge

One aspect of mobile proxies that makes them even more effective is the ability to match TCP/IP fingerprints with the purported device. As Sabir from iProxy.online explains:

"In some cases, your fingerprint, TCP fingerprint should match to your user agent. For example, if you like pretending to be a Mac user or iOS user or Windows user, your TCP fingerprint should be matched with your browser fingerprint."

This means detection mechanisms that look for mismatches between TCP/IP fingerprints and browser types can also be bypassed.

Anybody can now set them up

The barrier to entry for setting up residential proxies has fallen sharply. Companies like Proxidize market their products as simple to use, with statements like "Start using Proxidize in less than 60 seconds."

There are YouTube videos showing how to earn "passive income" by setting up proxy farms. One video explains how hosts can earn "$200 a month minimum" by hosting Proxidize hardware in their homes.

With iProxy.online, it's even simpler—just install an app on an Android phone, and you have a mobile proxy. As Sabir explains, "Actually your expenses are like you pay like for the SIM card, you pay a small subscription fee to the service and you just... That's it. It requires like one minute of work just to download an app."

This accessibility means residential proxy use is no longer limited to nation-states and sophisticated cybercriminal organisations. It is now within reach of anyone with basic technical skills.

The solution: per-connection detection

The rise of residential proxies means IP reputation databases are not enough on their own. As I've been explaining in my talks, "Residential proxies pose a significant challenge to traditional defense mechanisms... making malicious traffic appear legitimate."

The practical answer is per-connection detection that looks at network behaviour patterns rather than just IP addresses. At Peakhour.IO, we stack detections across layers to identify and mitigate proxy traffic.

A useful technique is analysing protocol behaviour. When traffic passes through a residential proxy, there are often detectable differences between network signatures (which come from the proxy) and the application behaviour (which comes from the third-party application).

These techniques can identify proxy connections even when they come from legitimate residential IP addresses, giving defenders a way to respond without blocking whole residential or mobile networks.

A call to action for businesses

If you're a business, especially in e-commerce, financial services, or any industry that relies on user accounts, residential proxy traffic needs to be part of your security model.

Traditional security approaches based on IP reputation, geolocation, and rate limiting are no longer sufficient. You need to implement per-connection detection that can identify residential proxy usage regardless of the source IP address.

At Peakhour.IO, we've seen organisations fall victim to attacks that could have been prevented with the right detection mechanisms. Waiting until credential stuffing or data exfiltration becomes visible is the expensive way to learn this lesson.

IP addresses alone can no longer tell us who to trust. We need to look deeper at each connection to protect systems and data now that proxy networks are easy to rent or build.

Your Anti-Fraud Residential Proxy Detection Sucks

2024-10-04T13:00:00+10:00

Online fraud is big business: account takeovers, chargebacks, scams, even romance scams. It costs businesses billions of dollars every year.

A common way websites fight it is to use an anti-fraud service to calculate the risk of a transaction. Most teams get this intelligence from a third-party service, either through an API or a plugin.

For online stores, ecommerce fraud prevention has to protect checkout and account flows without punishing real customers.

One of the major signals these services use is IP reputation. IP reputation tries to answer questions like:

Is the order coming from a datacentre?
Is it coming from a country other than your target audience?
Is the IP address a known VPN?
Is it a known TOR exit node?
Have lots of fraudulent orders come from this IP address in the past?

Until recently, these services gave teams a useful way to calculate fraud risk from an IP address.

Not anymore.

Fraud traffic has shifted in recent years, away from VPNs and TOR and toward residential proxies. These same anti-fraud services claim they can detect residential proxies, but what if the services many businesses rely on are falling well short?

The results are bad enough that they deserve a blunt look.

The Shocking Truth: Our Results

We took 25 IP addresses that had just been used as residential proxies in an attack on one of our clients, and within 5 minutes of detection ran them through some of the most popular IP intelligence services. The results are not going into anyone's marketing deck.

Here's a summary of our findings:

Service	Detected Proxies	Accuracy
Maxmind	0/25	0%
IP Quality Score	6/25	24%
Seon	1/25	4%
ProxyCheck.io	0/25	0%
ip2proxy	1/25	4%

The best performer in our test, IP Quality Score, detected only 24% of the proxies. The others ranged from 0% to 4%.

Why Your Residential Proxy Detection Service is Failing You

So why are these services performing so poorly? To understand it, we need to look at how proxy usage and detection have changed.

The Good Old Days of Proxy Detection

In the recent past, detecting proxies was much easier. Fraudsters primarily used:

TOR networks
VPN services
Data center proxies

These were relatively static targets. They were tied to a single, stationary IP, or IP ranges. Listing them in IP block lists was straightforward.

The Rise of Residential Proxies: A New Breed of Threat

Now we need to talk about residential proxies, the new go-to tool of fraudsters and scammers. These are not just a new label for old proxies. They behave differently.

What Are Residential Proxies?

Residential proxies come from IP addresses assigned to real residential services by Internet Service Providers (ISPs). These can be:

Home computers
Mobile phones
Tablets
IoT devices

Unlike data center proxies, which use IP addresses from hosting companies, residential proxies use IPs that look just like any other home or mobile user. They have become the tool for avoiding security controls on websites in the last 2-3 years, and they are causing all sorts of headaches for website owners.

How Are Residential Proxy Networks Formed?

This is where the problem starts:

Compromised Devices: Malware can turn innocent devices into proxy endpoints without the owner's knowledge.
Incentivised Programs: Some companies offer users benefits (like free VPN services) in exchange for using their device as a proxy endpoint. Hola VPN and Brightdata are prominent examples.
APP SDKs Quite often, proxy providers will incentivise app developers to include their proxy toolkit in their apps. The user is totally unaware that their device's internet connection is now being resold.

So your personal device, be it a computer or phone, could have its internet connection used to carry out a crime without you knowing. The police could come knocking on YOUR door one day.

Why Are They So Dynamic?

Since the proxy is formed by reusing the internet connection of a device, it is inherently much more dynamic than a proxy formed on a server.

Device Mobility: A mobile phone can connect from home Wi-Fi, then a coffee shop, then a cellular network – all in one day.
ISP IP Rotation: Many ISPs dynamically assign IP addresses, changing them periodically.

Depending on the type of fraud being carried out, the attacker might also rotate the device being used, popping out of a different location. Also, due to the way these proxies are formed, i.e. via an app on a computer or phone, that particular exit point on the proxy network might depend on that app being open.

This dynamic nature is what makes residential proxies so hard to detect using traditional methods.

Shared IPs: The Needle in the Haystack Problem

Residential proxy IPs are not just dynamic. They are typically shared. This means that a single IP address could be used by both legitimate users and proxy traffic:

ISP IP Pools: Internet Service Providers often use large pools of IPs that are dynamically assigned to users. This means that an IP used by a proxy one minute could be assigned to your grandmother's iPad the next.
Carrier-Grade NAT (CGN): Mobile carriers frequently use CGN, which can make hundreds or thousands of users appear to come from the same IP address.
Compromised Routers: A single compromised home router could serve both the legitimate traffic of the homeowner and proxy traffic from the attacker.

If you simply blocked any IP that shows proxy behavior, you would end up blocking legitimate users too.

Why Traditional Methods Are Failing (Revisited)

Now that we understand residential proxies better, let's revisit why old-school detection methods are not enough.

1. Port Scanning

Traditional proxy detection often relies on scanning for open proxy ports. Here's a simple port scanner:

import socket

def port_scan(ip, port):
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    result = sock.connect_ex((ip, port))
    sock.close()
    return result == 0

# Example usage
ip = "123.45.67.89"
proxy_ports = [80, 8080, 3128]  # Common proxy ports

for port in proxy_ports:
    if port_scan(ip, port):
        print(f"Port {port} is open - potential proxy detected")

Why it fails: Residential proxies don't typically have these ports open. They route traffic through standard web ports, making them indistinguishable from normal traffic.

2. Honeypots

Honeypots try to lure and identify proxy traffic.

Why it fails: Sophisticated residential proxy networks can identify and avoid known honeypots. Plus, since they're using real residential IPs, even if they do hit a honeypot, the IP itself isn't a reliable indicator of proxy usage.

3. Client-Side Detection

Detection services may also try to detect proxies by executing Javascript in the browser and checking the result for inconsistencies. These are the common techniques.

3.1 WebRTC Leak

WebRTC can sometimes reveal a user's true IP address:

function detectRealIP(callback) {
    var RTCPeerConnection = window.RTCPeerConnection || window.mozRTCPeerConnection || window.webkitRTCPeerConnection;
    var pc = new RTCPeerConnection({iceServers:[]}), noop = function(){};
    pc.createDataChannel("");
    pc.createOffer(pc.setLocalDescription.bind(pc), noop);
    pc.onicecandidate = function(ice) {
        if(!ice || !ice.candidate || !ice.candidate.candidate) return;
        var myIP = /([0-9]{1,3}(\.[0-9]{1,3}){3}|[a-f0-9]{1,4}(:[a-f0-9]{1,4}){7})/.exec(ice.candidate.candidate)[1];
        pc.onicecandidate = noop;
        callback(myIP);
    };
}

detectRealIP(function(ip) {
    console.log("Your real IP address is: " + ip);
});

3.2 Geolocation Inconsistencies

Comparing IP-based geolocation with browser-reported location.

navigator.geolocation.getCurrentPosition((position) => {
  const browserLat = position.coords.latitude;
  const browserLong = position.coords.longitude;
  // Compare with IP-based geolocation from server
});

3.3 DNS Leaks

Check whether DNS requests are routed through the proxy or are leaking:

const image = new Image();
const uniqueDomain = `test-${Date.now()}.example.com`;
image.src = `http://${uniqueDomain}/pixel.gif`;
// Monitor DNS requests server-side to detect leaks

3.4 Browser Fingerprinting

Check whether there are inconsistencies with the browser, e.g. timezone, and the geolocation of the IP address

const fingerprint = {
userAgent: navigator.userAgent,
screenResolution: `${screen.width}x${screen.height}`,
colorDepth: screen.colorDepth,
timezone: Intl.DateTimeFormat().resolvedOptions().timeZone,
plugins: Array.from(navigator.plugins).map(p => p.name),
// ... other characteristics
};
// Analyze fingerprint for proxy indicators

Why these techniques fail

Proxy services can work around all of these methods. Many browsers now allow users to disable WebRTC or use extensions that prevent this leak. Some residential proxy services are sophisticated enough to handle WebRTC requests without leaking the real IP.

Finally, relying on client-side detection means: * Your detection can be reverse engineered and bypassed. * You've already served the content the attacker wants. * It requires Javascript execution, something that won't always be available, for instance on an API.

4. Threat Intelligence

Threat intelligence involves maintaining databases of known proxy IP addresses:

import requests

def check_ip_threat_intel(ip):
    api_key = "your_api_key_here"
    url = f"https://api.threatintelligence.com/v1/ip/{ip}?key={api_key}"
    response = requests.get(url)
    if response.status_code == 200:
        data = response.json()
        return data.get('is_proxy', False)
    return False

# Example usage
ip = "123.45.67.89"
if check_ip_threat_intel(ip):
    print(f"{ip} is a known proxy according to threat intelligence")

Why it fails: As our results show, threat intelligence databases are struggling to keep up with the dynamic nature of residential proxies. By the time an IP is identified and added to a database, it may no longer be in use as a proxy.

Why IP-Based Blocking Is No Longer Enough

Given the shared nature of IPs in the age of residential proxies, simply identifying and blocking "bad" IPs is too blunt. Here's why:

False Positives: Blocking an IP used by a proxy might also block legitimate users sharing that IP.
Ineffectiveness: Proxies can quickly switch to new IPs, so IP-based blocking turns into a chase.
Collateral Damage: You might end up blocking entire ISPs or mobile carriers, cutting off large swaths of legitimate users.

The practical failure is customer friction. A checkout, login, or account recovery flow cannot treat every residential or mobile network as hostile. Teams need enough request context to choose a proportionate response: log the signal on a low-risk page, challenge a suspicious account change, rate limit repeated failures, or block only when proxy evidence lines up with stronger abuse indicators.

The same problem shows up on APIs and partner integrations. A partner batch job, mobile carrier, or shared office network can look noisy without being hostile. A compromised key can look legitimate until it starts hitting expensive routes. Good review paths keep the route, account or API key, request rate, fingerprint, and recent outcomes together so the answer is not just "block the IP" or "allow everything."

This is where proxy detection stops being a lookup problem and becomes an operating problem. The team needs to know which route was hit, what account or session was involved, whether the request matched normal behaviour, and what the enforcement cost would be if the signal was wrong. That is the difference between reducing abuse and quietly pushing good customers into support. For a deeper treatment of that tradeoff, see the guide to customer friction and false-positive measurement.

The Need for Connection-Level Detection

Instead of focusing only on IPs, we need to look at the connections themselves. Here's what this means:

Deep packet inspection: Analyses traffic patterns and characteristics beyond surface-level indicators.
Protocol behaviour analysis: Identifies subtle anomalies in how network protocols are implemented across the proxy chain.
TLS/TCP fingerprinting: Examines characteristics of TLS handshakes to detect proxy usage.
Timing analysis: Measures minute differences in network latency that can indicate the presence of a proxy.

Final Thoughts

Proxy usage has evolved, and detection methods need to keep up. Simple IP-based blocking and static lists of "bad" addresses are no longer enough. They still have a place, but they cannot be the whole answer.

Modern residential proxy detection has to evaluate the live request: IP context, connection behaviour, fingerprints, route, account state, request rate, and recent outcomes. On a low-risk page that may only justify logging. On login, signup, checkout, password reset, API key creation, or account recovery, the same signal may justify a challenge, hold, rate limit, or block.

Peakhour's residential proxy detection is built for that request-path decision. The point is not to label every residential IP as bad. The point is to give operators enough evidence to act when proxy use lines up with account abuse, fraud, scraping, or automated traffic.

If you're still treating IP reputation as the main answer, you're already behind. It's time to stop blocking IPs and start understanding the request.

Want a demo of our residential proxy detection? Contact us for a live demo of our service.

The Australian epidemic of Account Takeover attacks

2024-07-29T10:00:00+10:00

In recent months, credential stuffing attacks have hit a number of Australian businesses, leading to compromised accounts, fraudulent purchases, and customer complaints. The pattern is a reminder that account protection cannot stop at password policy or MFA alone.

A Case Study in Credential Stuffing

Security researcher Jacob Larsen has documented a credential stuffing operation targeting Australian businesses. Larsen's research, detailed in his blog post, describes the activity of a threat actor known as "Crabby," who has sold compromised Australian accounts since July 2023.

Larsen's findings show:

The operation began with a threat actor called "Based" selling compromised accounts via Discord and dedicated websites.
In November 2023, the operation was acquired by "Juicy," a notorious account vendor, and rebranded as "Crabby."
As of May 2024, over 19,000 compromised accounts from various Australian brands were offered for sale.
Low-level fraudsters purchasing these accounts have used them to make unauthorised purchases, often targeting high-value items for resale.

The Crabby operation shows how credential stuffing has moved beyond isolated login attempts. It now includes account marketplaces, low-level fraud buyers, and the challenges businesses face once compromised accounts are monetised.

The Difficulty of Defense

Credential stuffing defence is harder when attacks are spread across residential proxies and kept to single attempts per account.

Residential Proxies: The Invisible Threat

Residential proxies weaken traditional IP-based controls. These proxies use IP addresses assigned to real residential internet connections, so malicious traffic can look like normal customer traffic. That helps attackers bypass simple rate limiting and geolocation checks.

That distribution makes login traffic harder to classify. Signals such as a high volume of attempts from one IP address become less useful when attackers can spread requests across a pool of residential IPs.

Single-Hit Attacks: Precision Strikes

Single-hit attacks are another way attackers avoid noisy patterns. In this approach, each stolen credential is used only once per target site, reducing the chance of detection by traditional rate-limiting or anomaly detection systems.

By limiting each credential to one attempt, attackers avoid controls tuned to repeated login failures. A business can have rate limiting in place and still miss credential stuffing that never crosses those thresholds.

The Mobile API Conundrum

As mobile applications become a primary user interface, credential stuffing also moves into mobile API traffic. Traditional bot protection often relies on JavaScript challenges or browser fingerprinting, which does not apply cleanly to attacks against mobile APIs.

Mobile applications typically communicate with backend services via APIs, bypassing the browser environment where many bot detection techniques run. This creates several challenges:

Lack of JavaScript Execution: Mobile APIs don't execute JavaScript, making it impossible to use browser-based bot detection techniques.
Limited Fingerprinting Capabilities: Standardised mobile API requests make it difficult to distinguish between legitimate user activity and automated attacks based on request characteristics.
Increased Attack Surface: More mobile apps means more potential entry points for attackers, making comprehensive protection more complex.
Authentication Simplification: To improve user experience, mobile apps often use simplified authentication flows, which can create weaker controls against automation.

This gap needs API-centred controls that can assess mobile login behaviour without relying on browser-only signals.

Framing Credential Stuffing as a Business Risk

Credential stuffing should be treated as a business risk, not just an authentication issue. The impact can include refunds, chargebacks, customer support load, reputational damage, and regulatory disclosure work.

Risk Quantification and Disclosure

Risk quantification gives security teams a way to explain credential stuffing in business terms. By applying frameworks like FAIR (Factor Analysis of Information Risk), businesses can:

Quantify the potential financial impact of credential stuffing attacks.
Prioritise security investments based on risk reduction potential.
Communicate the importance of cybersecurity measures to non-technical stakeholders.

CPS 234 in Australia adds a disclosure dimension for regulated entities. Businesses need to protect against credential stuffing and be able to explain their exposure, controls, and mitigation strategy.

The State of Credential Stuffing Defense in Australia

Our recent survey of Australian businesses shows uneven adoption of credential stuffing defences:

While 77% of respondents use Multi-Factor Authentication (MFA), only 40% have implemented bot protection measures.
15% of companies chose not to respond to questions about their security measures, suggesting potential gaps in protection.
Just 29% of businesses check credentials against known breaches, leaving a large window of opportunity for attackers using stolen credentials.
Only 15% of organisations use residential proxy detection, a critical component in identifying and mitigating modern credential stuffing attacks.

These results suggest a gap between how credential stuffing is run now and the controls many Australian businesses have in place.

Recommendations for Enhanced Protection

Based on our analysis and survey results, businesses should review the following controls:

Implement Advanced Bot Protection: Deploy controls that detect and mitigate bot attacks, including attacks using residential proxies.
Enhance Mobile API Security: Use mobile API controls that focus on anomaly detection and behavioural analysis rather than browser-based techniques.
Adopt Risk-Based Authentication: Implement dynamic authentication mechanisms that adjust based on the assessed risk of each session or transaction.
Utilise Breached Credential Databases: Check user credentials against known breach databases and enforce password changes for compromised accounts.
Implement Residential Proxy Detection: Use technology that identifies and mitigates traffic from residential proxy networks. This is a key control for modern credential stuffing attacks.
Apply Advanced Rate Limiting: Utilise device fingerprinting and other identifiers beyond IP addresses to implement more effective rate limiting, particularly for single-hit attacks.
Employ Contextual Security: Use signals such as user behaviour patterns, device characteristics, and historical usage to identify anomalies that may indicate credential stuffing attempts.
Quantify and Communicate Risk: Use frameworks like FAIR to quantify the potential impact of credential stuffing attacks and communicate this risk to stakeholders.
Implement Continuous Monitoring: Deploy real-time monitoring that detects patterns indicative of credential stuffing attacks, and update defences as attack methods change.

These controls address the specific problems created by residential proxies, single-hit attempts, mobile API traffic, and weak credential hygiene. They also reflect the limits of IP-only rate limiting and browser-only bot detection.

Credential stuffing defence works best as a layered programme: bot detection, residential proxy detection, breached credential checks, mobile API coverage, and risk reporting. The practical goal is to stop account takeover attempts earlier, reduce fraud exposure, and give security teams evidence they can act on.

The Cost of Credential Stuffing

2024-07-17T00:00:00+10:00

In recent months, Australian businesses have faced a wave of credential stuffing attacks. These attacks do not require the affected website itself to be breached. They target customer accounts, leading to fraudulent transactions. The damage is practical as well as reputational: disputed purchases, refunds, locked accounts, and customers asking how someone else was able to use their account.

What is Credential Stuffing?

Credential stuffing occurs when attackers use login details obtained from a data breach to access accounts on other sites. Criminals test millions of credentials against a target website to identify working combinations. This attack affects users who reuse passwords across multiple services [1].

The Scale of the Problem

Tens of thousands of Australian online accounts are reported to have been accessed since late November 2023 [2]. The attacks affected major retailers and service providers, including:

The Iconic
Guzman y Gomez
Dan Murphy's
Event Cinemas
Stan

The Impact

While reusing passwords between sites has long been considered poor security practice, users still do it. Blaming the customer, as 23andMe did in its response to an attack, is not a serious account protection strategy. Over 70% of Americans believe that websites have a responsibility to prevent account takeovers via stuffing attacks. Not doing so can negatively impact a business in several ways.

Financial Impact

The cost can fall on either the affected business or the affected customer. Fraudsters made significant purchases using compromised accounts. One scammer claimed to have spent over $800 on high-end alcohol at Dan Murphy's [2]. Others bought iPhones and clothing. Either the customer will be out of pocket, or the business when the customer issues a chargeback on the purchase.

Reputation Damage

The attacks leave businesses dealing with customer complaints, refunds, and visible questions about account security. The Iconic pledged to refund affected customers [1]. Dan Murphy's confirmed that a "small number of user accounts were subject to fraudulent transactions" [3].

Customer Trust

These incidents erode customer trust. Users expect businesses to make account abuse difficult, even when the original password leak happened somewhere else. When accounts are taken over, customers question the security practices of the affected companies.

Business Response

Companies responded by:

Locking compromised accounts
Issuing refunds
Encouraging customers to change passwords
Implementing stronger security measures

Dan Murphy's advised customers to "practise good password hygiene, using a strong password and changing it periodically" [3].

Prevention Strategies

To protect against credential stuffing, businesses should:

Implement multi-factor authentication
Educate customers about password security
Monitor login behaviour on their website
Implement, and regularly update, security measures, including bot management and advanced rate limiting.

Credential stuffing is not just a password reuse problem. It is an account protection problem, and businesses that sell online need controls that make stolen credentials harder to turn into purchases.

Sources:

[^1^] ABC News: "The Iconic was hit by criminals taking money by 'credential stuffing'. How can you stay safe?" [^2^] Cyber Daily: "Guzman y Gomez, Dan Murphy's customers affected in credential stuffing campaign" [^3^] The Sydney Morning Herald: "Thousands of Australians hacked in 'credential stuffing' credit card scam"

How To Exclude Query String Parameters from Search Engine crawling

2024-05-21T13:00:00+10:00

Last year we wrote about the problem of excessive crawling from search engine spiders. Search engines such as Google and Bing aim to index as much content as possible. For ecommerce sites, this often means indexing pages with query string parameters used for sorting, filtering, or pagination. Those parameters help users navigate the site, but they can cause a few predictable crawler problems:

Over-Crawling: Search engines may spend too much time crawling similar pages with different parameters, wasting crawl budget.
Duplicate Content: Pages with different parameters can be treated as duplicate content, weakening SEO performance.
Server Load: Excessive crawling can increase server load, slow down your site, and affect user experience. Search engines typically account for 30-50% of page requests to an ecommerce store. Managing their crawling effectively can have a material effect on site speed and server spend.

Another common cause of over crawling is internal searches being indexed.

In our previous article we mentioned using the webmaster tools provided by Google and Microsoft to manage crawler behaviour by adding ignored parameters. Since then, both tools have been updated and no longer allow you to add parameters to ignore during a crawl.

Differences in Crawling and Indexing

Search engines maintain an 'index' of web pages. Pages in this index are what appear in search results. To maintain the index, the search engine crawls a website to 'discover' new content and keep existing entries up to date. Webmasters can control what gets indexed with tags or headers in their web pages. These include:

Canonical Tags can be used to indicate the preferred version of a page. This helps consolidate link 'juice' and tell the search engine which URL to index.
Noindex tags can be used to prevent specific pages from being indexed. This is useful for thank you pages, admin pages or any content you don't want to appear in search results.
Nofollow links can be used to indicate to a search engine not to pass on SEO value to the linked page.

However, controlling what does or does not get indexed does not prevent content from being crawled. The only way to do that is via the robots.txt file. You may be familiar with the Disallow directive in the robots.txt file, but you can also use wildcards to prevent crawling of url parameters.

An example...

Consider an ecommerce store that has a category page which can then be customised with the following parameters:

    orderBy
    colors
    brands
    page
    results

These may appear in any order, and the combinations can result in 100s or even 1000s of variations of essentially the same page. Google is fairly smart when presented with this scenario, but Bing.... Bing can crawl very aggressively and it likes to try everything. In our example above, we may want to stop crawling everything except the page number, in which case an effective way to control crawler behaviour would be:

    User-agent: *
    Disallow: /*?*orderBy=*
    Disallow: /*?*colors=*
    Disallow: /*?*brands=*
    Disallow: /*?*results=*

We can't really do this in a single Disallow because the parameters might be in any order. By including the ? in the url we're ensuring that the parameter names are only in the query string, not in the main url path. This prevents crawlers from wasting crawl budget and putting unnecessary load on server resources.

Final Thoughts

Search engines can often make up 30-50% of the overall page requests to a website. Managing their behaviour helps maximise useful crawling and minimise server utilisation. Keep an eye on your access logs for unwanted behaviour, and use robots.txt where it gives you the right level of control.

HTTP/2 Rapid Reset Attack Deepdive

2023-10-12T00:00:00+11:00

Distributed Denial of Service (DDoS) attack vectors keep changing. The recent spike in HTTP/2-based DDoS attacks has been notable for its volume, with some attacks surpassing 398 million requests per second. Peakhour observed these attacks and worked through how to understand and mitigate them. This article explains how they work and what operators can do to reduce exposure.

The Rise of HTTP/2 in DDoS Attacks

HTTP/2 was designed to make web traffic more efficient. The same features that improve performance for legitimate users can also be abused in DDoS traffic.

Much of HTTP/2's efficiency lies in "stream multiplexing." It allows multiple messages to be sent over a single TCP connection. While HTTP/1.1 processes each request serially, HTTP/2 can manage multiple concurrent streams on a single connection. This means a client can send multiple requests in a single round trip, increasing how much work each connection can drive.

The 'Rapid Reset' Attack Explained

The "Rapid Reset" attack is a specific DDoS technique built around HTTP/2. The attacker starts by opening multiple streams, much like in a standard HTTP/2 attack. However, instead of waiting for responses, they cancel each request immediately.

The client does this by sending a RST_STREAM frame, indicating that a previous stream should be cancelled. The rapid request-and-reset sequence means the server spends resources processing the request, only for it to be cancelled before a response is generated. This tactic amplifies the server's workload without the attacker needing to wait for responses, which increases the pressure each connection can place on the server.

Variants of the Rapid Reset Attack

Attackers also used variations of the Rapid Reset attack:

One variant involves delaying the reset action. The attacker opens multiple streams, waits, then cancels the streams and instantly opens new ones. This method can evade some rate-based defences.
Another variant avoids stream cancellations. Instead, the attacker tries to open more streams than the server allows. This aims to keep the server continually busy, processing a near-constant flow of requests.

Effective Mitigation Techniques

Mitigation is not as simple as blocking individual malicious requests. A more effective approach is to close the entire TCP connection when malicious activity is detected. The HTTP/2 protocol supports connection termination through the GOAWAY frame. This feature needs to be used aggressively to prevent Rapid Reset attacks, rather than relying on the more passive, standard implementation.

Deciding which connections to treat as malicious is a challenge. One potential strategy is to monitor connection statistics. If a connection exceeds a set threshold of cancelled requests, it might be deemed malicious. Responses to suspect activity could range from sending a GOAWAY frame to terminating the TCP connection.

For the non-cancelling variant, the best approach is to shut down connections that breach the concurrent stream limit, either immediately or after a few violations.

Broader Protocol Implications

These attack techniques are specific to HTTP/2, but the wider protocol lesson still matters. The HTTP/3 (QUIC) protocol isn't directly vulnerable in the same way. As a precaution, server implementations should consider limiting the work done by a single connection.

The Importance of Industry Collaboration

When the threat of the Rapid Reset attack became apparent, the industry collaborated to address the issue. The vulnerability was disclosed to key HTTP/2 implementers, helping to devise and distribute effective countermeasures. The vulnerability is logged against CVE-2023-44487.

HTTP/2 'Rapid Reset' DDoS attacks pose a serious risk to services using the protocol. To reduce exposure, service providers should promptly apply available software patches and updates.

Learn how Peakhour's Application Security Platform helps protect against Layer 7 DDoS attacks, including the HTTP/2 Rapid Reset vulnerability. Contact our team to secure your infrastructure.

Understanding the HTTP/2 Rapid Reset Attack

2023-10-11T00:00:00+11:00

The discovery of the HTTP/2 Rapid Reset flaw exposed a serious weakness in a widely used version of the HTTP protocol. When exploited, it can be used to generate large Distributed Denial of Service (DDoS) attacks against HTTP/2 services. This post explains how the attack works and what operators can do to strengthen their defences.

A Deep Dive into the HTTP/2 Rapid Reset Flaw

HTTP/2 is widely deployed, so a flaw in how implementations handle rapid stream resets can have a large operational impact. To take advantage of the issue, a malicious actor sends a request and immediately cancels it, then repeats that pattern over the same HTTP/2 connection. By scaling this "request, cancel" behaviour thousands of times, an attacker can overwhelm vulnerable HTTP/2 implementations. The result is DDoS attacks at the application layer, with potential downtime and disruption.

Major companies including Cloudflare and Google have dealt with this issue. Google, for example, mitigated a DDoS attack reaching a peak of 398 million requests per second that relied on this technique. For scale, this two-minute-long attack generated more requests than the total number of article views reported by Wikipedia in September 2023.

Mitigating the Threat

Large infrastructure providers have led much of the work to understand the attack mechanics and develop mitigations:

Patching Systems: Prompt patching is the primary control for the HTTP/2 Rapid Reset attack. Companies including Peakhour, Microsoft, and others have tested and patched their systems against this threat.
Rate Limiting: Advanced rate limiting has been a recommended action. It provides an extra layer of protection, minimising the risk of massive request inflows.
Collaborative Efforts: Google and Microsoft have both shared intelligence and collaborated with other cloud providers and software maintainers implementing the HTTP/2 protocol stack. This has resulted in patches and mitigation techniques now employed by numerous large infrastructure providers.

What's Next for Users and Enterprises?

If you serve an HTTP-based workload online, understand whether this attack affects your environment. Verify that servers supporting HTTP/2 are either not vulnerable or have applied the necessary patches. Stay informed and consider reaching out to your service providers or account representatives for configuration assistance and guidance.

The HTTP/2 Rapid Reset flaw is a serious application-layer DDoS risk, but it is manageable with the right mitigations in place. Apply the recommended patches and keep HTTP/2-facing services under active review.

Discover how Peakhour's Application Security Platform protects against Layer 7 DDoS attacks, including the HTTP/2 Rapid Reset vulnerability. Contact our team to secure your infrastructure.

ZDNS - scan the entire internet

2023-06-20T13:00:00+10:00

The lack of a free Reverse DNS (rDNS) lookup database has made large-scale DNS research harder. To address this, we used ZDNS, an open-source, high-performance DNS toolkit developed by Stanford University, to create our own rDNS database. To reduce UDP timeout issues during rDNS operations, we devised a scan-ordering approach that randomised the IP space and improved the efficiency of the scanning process.

Leveraging ZDNS for rDNS Lookups Across the Internet

Understanding rDNS is useful for internet operations and research. Active DNS measurement helps us inspect how providers advertise the use of their IP address space. One of the components of this ecosystem is Reverse DNS (rDNS), which serves an important role in IP database categorisation and ASN (Autonomous System Number) classification. However, running rDNS across the entire internet is not a trivial task.

Previously, Rapid7 provided a free database for rDNS lookups, but it has discontinued the offering. This situation has prompted the need to create our own database, calling for a robust, efficient, and scalable tool to accomplish the task. ZDNS was the right fit.

Introducing ZDNS

ZDNS, a part of the ZMap.io project, is a capable tool developed by Stanford University to support scalable and reproducible DNS research. ZDNS is an open-source DNS measurement framework specifically optimised for large-scale DNS research on the public internet. It can resolve 50 million domains in 10 minutes and query the PTR records of the complete public IPv4 address space in approximately 12 hours.

This high-performance toolkit offers a modular interface, enabling researchers to safely implement new functionalities. Its architecture is designed to expose DNS lookup chains by performing recursive resolution. ZDNS supports a command-line interface and outputs results in JSON, a machine-parsable format.

Enhancements by ZDNS

ZDNS's architecture and feature set are tailored to the challenges of extensive DNS research. Its guiding principles are that the DNS lookup chain is exposed, and that the tool is safe, easy to use, and extensible.

ZDNS's performance optimisations make it a suitable tool for DNS experiments that require querying a large number of names. Parallelism, UDP socket reuse, and selective caching are some of the critical performance optimisations that enable ZDNS to efficiently handle large volumes of DNS queries.

ZDNS's scalability, execution time, and success rate have been evaluated against several existing tools, showcasing its performance. For instance, when it comes to exposing the DNS lookup chain, ZDNS is 85 times faster than Dig. ZDNS also outperforms other higher-performance tools, achieving 2.6 to 3.6 times more successful queries per second and experiencing about 30% less packet drop than MassDNS.

Our rDNS Journey

When we started scanning the whole internet with rDNS, we hit a practical roadblock: UDP timeouts made the scans slow. The system spent too much time waiting for responses from parts of the internet that were either empty or broken.

We used two changes. Firstly, instead of scanning the internet's addresses in order, we mixed them up and scanned randomly. This spread out our requests and stopped the system from getting stuck on troublesome ranges. Secondly, we checked smaller sections of the internet first, so we did not waste time waiting for big chunks of the internet that weren't responding.

With these changes, we scanned the whole internet in 13 days, finding over a billion addresses. The main lesson was straightforward: scan order matters when timeout behaviour dominates runtime.

Wrapping Up

ZDNS has proven to be a valuable tool for DNS research, especially for substantial tasks like performing a reverse DNS scan of the entire internet. Our experience underscores the value of practical adjustments when dealing with large-scale challenges, like randomising the IP space to avoid delays caused by UDP timeouts.

As an open-source tool, ZDNS is available on Github. For more detail, read the award-winning paper presented at IMC 2022.

Our work with ZDNS shows its value in DNS research and the operational detail involved in large-scale DNS work. By randomising the scan order, we mitigated timeout issues and improved the efficiency of our scanning process.

Izhikevich, L., Akiwate, G., Berger, B., Drakontaidis, S., Ascheman, A., Pearce, P., Adrian, D., & Durumeric, Z. (2022). ZDNS: a fast DNS toolkit for internet measurement. In Proceedings of the 22nd ACM Internet Measurement Conference (pp. 33-43). https://doi.org/10.1145/3517745.3561434 ↩
ZMap Project. (n.d.). ZDNS. GitHub. Retrieved 2023-05-15 13:00, from https://github.com/zmap/zdns. ↩

When Bots Break Bad

2023-05-16T13:00:00+10:00

Bots account for a large share of web traffic. Recent studies put automated traffic at nearly 50% of all internet requests. Some bots are useful, such as search engine crawlers that index your site. Some are clearly harmful, such as scrapers and sneaker bots. Others sit in a grey area, including backlink and marketing bots from services such as Ahrefs and SEMrush. Even useful bots can create problems when they crawl too hard. This article looks at the main bot types and how to manage them with robots.txt and bot management tools.

Understanding the Different Types of Bots

'Good Bots'

Good bots perform legitimate work. Search engine crawlers like Googlebot and Bingbot index webpages so search results can stay current and relevant. Other examples include uptime and performance monitoring bots.

'Bad Bots'

Bad bots harm websites, users, or both. Common examples include:

Scraping content, copying and repurposing data from websites.
Sneaker bots, automatically purchasing limited-edition products (like sneakers) before human users can.
Spam bots, posting unsolicited messages and advertisements in comment sections or forums.
Vulnerability Scanners, trying thousands of website URLs to find security vulnerabilities.
Account Takeover, attempting to gain access to existing user/admin accounts using either credential stuffing or brute-force attacks.

'Grey Bots'

Grey bots sit between good and bad. They often serve a useful purpose and may follow crawling directives in robots.txt, but they can still cause problems when they crawl too aggressively. Common examples include:

AhrefsBot: A backlink analysis bot used by Ahrefs, an SEO tool.
SEMrushBot: A bot used by SEMrush, another popular SEO and digital marketing tool.
MJ12bot: A bot used by Majestic, a service that provides backlink data and analysis.
ScreamingFrog: An SEO analyser run from a local desktop.

When Grey bots (and even Good Bots) go bad.

Left unattended, grey bots can create practical problems:

Slow page loading times, which affect user experience.
Strain on server resources, potentially causing crashes, downtime, and higher costs.
Distorted website analytics, when bot traffic is mistaken for human traffic.

Managing Grey Bots with Robots.txt

The robots.txt file is a simple text file that tells web crawlers which parts of your site they can or cannot access. You can use it to manage bot behaviour and protect your website from aggressive crawling. Useful controls include:

Disallowing specific bots: You can block specific bots from accessing your site by adding a "User-agent" and "Disallow" directive to your robots.txt file. For example:

User-agent: AhrefsBot
Disallow: /

Limiting crawl rate: You can ask bots to slow down their crawling by adding a "Crawl-delay" directive:

User-agent: SEMrushBot
Crawl-delay: 10

Not all bots will follow robots.txt. ScreamingFrog, for example, can be instructed to ignore robots.txt and crawl a site as quickly as possible. You would not want a competitor doing this to your site.

Bot Management Tools

In addition to robots.txt, bot management tools (like those provided by Peakhour) can protect your website from abusive bots. Good bot management tools automatically block most unwanted traffic using a combination of Threat Intelligence, Fingerprinting techniques, Reverse DNS verification, and Header Inspection.

Advanced techniques like rate limiting and machine learning can help identify more sophisticated bad bots.

Search Bots and Double Crawling

Search bots like Bingbot can sometimes blindly follow links and crawl the same page multiple times due to different URL parameters. This double, triple, or worse crawling can increase server load and make indexing less efficient. eCommerce sites are especially exposed because product catalogues often have several filtering paths. We've seen Bing go haywire on a number of sites. Most recently, it was issuing around 50,000 requests per day to the search function of a Magento 2 store while cycling through parameters. This dropped to 2-3k requests per day when fixed. On another store, Bing was responsible for nearly half of all page requests (40k page requests) on a busy OpenCart store. Configuring it to ignore parameters dropped this to around 4k per day.

Configuring Search Bots to Ignore Query Parameters

Note: Since publishing both Google and Bing have removed the ability to ignore parameters when crawling via their webmaster/search console tools. See using robots.txt to instruct search engines to ignore query string parameters

To help search bots crawl your site efficiently, you can configure them to ignore specific query parameters. Use these methods:

Configuring Bing Webmaster Tools

Bing Webmaster Tools provides an option to specify URL parameters that should be ignored during the crawling process. To configure this setting, follow these steps:

Sign in to your Bing Webmaster Tools account and select the website you want to manage.
Navigate to the "Configure My Site" section and click on "URL Parameters."
Click on "Add Parameter" and enter the parameter name you want Bingbot to ignore.
Select "Ignore this parameter" from the dropdown menu and click on "Save."
Configuring Bing Webmaster Tools this way helps stop Bingbot double crawling pages with specific URL parameters, reducing server load and improving indexing efficiency.

Managing Other Search Bots

For other search engines like Google, use the relevant webmaster tools to manage URL parameters. In Google Search Console, follow these steps:

Sign in to your Google Search Console account and select the property you want to manage.
Navigate to the "Crawl" section and click on "URL Parameters."
Click on "Add Parameter" and enter the parameter name you want Googlebot to ignore.
Choose "No URLs" from the "Does this parameter change page content seen by the user?" dropdown menu.
Click on "Save."
Specifying the parameters you want search bots to ignore can prevent double crawling and make indexing more efficient.

Final Thoughts

When good or grey bots crawl too aggressively, they can cause the same operational problems as malicious bots: overloaded servers, slower pages, and worse user experience. Monitor website traffic and server load, set clear robots.txt rules, and use the major search engines' webmaster tools to control inefficient crawling. Done properly, this improves website performance and can lower infrastructure costs.

Layer 7 DDoS Protection

2023-02-07T13:00:00+11:00

In previous blog posts we've covered the benefits that full page caching has on page load performance. We also covered how caching pages lowers origin server utilisation, so the site can handle more customers. A lesser-known side benefit is protection against Layer 7 denial of service (DoS) attacks.

What is a Denial of Service attack?

The goal of a DoS attack is to make a network resource, such as a website or networked service, unavailable to users by overwhelming it with excessive traffic or requests. DoS attacks can be launched from a single source or from multiple sources (known as a distributed denial of service, or DDoS attack).

Typically, an attacker floods the web server with HTTP or API requests to try to overwhelm it. An attacker might also launch a 'slow attack', especially if they find a weak point in the application that consumes a lot of server resources for a single request. One example is repeatedly using a site search function. By slowing down the rate at which requests are sent, the attacker can bypass common rate-limiting and traffic-shaping mechanisms that are designed to block high-volume traffic spikes.

DoS conditions can also be inadvertent. A CMS like Magento or WordPress on an underpowered server can be overwhelmed, or slowed to a crawl, by so-called 'grey' bots. Examples include Semrush, Ahrefs, dotbot, and MJ12Bot. These spiders can crawl a site aggressively enough to bring it to its knees. Even Bing or Google can negatively impact a site.

What is Layer 7?

In networking parlance, Layer 7 refers to the actual application running on a web server. It is part of the OSI (Open Systems Interconnection) model used to describe the various functions in transmitting data over a network. The model is as follows:

Layer 1 -> Physical Layer, responsible for transmitting raw bits of data, e.g., a wire or wireless link
Layer 2 -> Data Link, responsible for transmitting data frames between network devices and detecting transmission errors.
Layer 3 -> Network Layer, responsible for routing data packets between networks and determining the best path for transmission.
Layer 4 -> Transport Layer, responsible for end-to-end communication between applications, providing reliable data transmission and flow control.
Layer 5 -> Session Layer, responsible for establishing, maintaining, and terminating communication sessions between applications.
Layer 6 -> Presentation Layer, responsible for formatting data to be presented to the application layer and for converting data from the application layer into a standardised format for transmission.
Layer 7 -> Application Layer, responsible for providing services to the user, such as file transfer, email, and web access.

How Full Page Caching helps

Full page caching helps by reducing the number of dynamic requests to the server. Caching the entire page allows the server to serve pre-generated content to visitors, rather than generating it dynamically each time a page is requested. This reduces the server's processing load and helps prevent excessive requests from overloading the server or making it unavailable.

A real world example

As part of a series of Peakhour recommendations to 'harden' its website against layer 7 attacks, a Peakhour client (a government site) applied limited full page caching with a time to live of 10 minutes. This means that Peakhour will serve a page from its Edge cache for 10 minutes before hitting the origin again for a new version.

Not long after implementation, that change was tested: a DDoS originating from hundreds of IPs across multiple countries hammered a set of 5 pages. The attack was spread over several days with bursts of activity lasting about 15 minutes. With the first hit, the page was cached and every subsequent hit was served from our high-performance Edge cache. No slowdown was observed and the attackers gave up.

Real attack on a Peakhour client; the spikes formed in 15 minute bursts.

Real page load times measured in the client browser

But my site is dynamic...

Many websites have a small dynamic component on the page. For ecommerce sites this might be the mini cart in the top right showing the number of items in the cart, or it might be some personalisation for a user. Often these dynamic parts of the page can be rendered in the browser using Ajax or local storage, rather than rendered on the server. By moving the dynamic components to the browser, the full page becomes cacheable. Another option is to use Edge Side Includes (ESI), which enables the majority of the page to be cached in the CDN while the dynamic parts are fetched separately before serving the full page to the user.

Peakhour can help move dynamic page components from the server to the browser and cache more at the edge. We also have a range of CMS plugins that do just that. Contact us if you want some help.

Final Thoughts

Layer 7 DoS attacks have become more common than traditional DoS attacks, as they typically require far fewer resources from the attacker. Reducing dynamic requests to the origin using full page caching is a useful but underappreciated way to mitigate them. If you're concerned about website security, resilience, and performance, Peakhour helps you cache more so you can protect, accelerate, and scale your website.

Origin shield

2022-06-10T13:00:00+10:00

CDN providers often promote the size of their network, and how many Points of Presence (POPs) they have. Higher capacity, more resilient networks are useful from a security point of view (think DDoS attacks), but more POPs can also work against what the CDN was designed to do: take load off an origin and improve performance for end users.

The POP Problem

Modern CDNs are what's called 'Pull' CDNs. That means the CDN won't store content/resources until a user requests it. The first time a user requests a resource, it goes to the CDN POP, checks its local cache, gets a miss, and then passes the request through to origin. As the resource is returned, the CDN stores a copy for the next time someone wants it. If your CDN has 100 POPs, then this process has to be repeated 100 times to fully 'warm' the CDN for that specific resource. That's 100 requests to origin. The more POPs your CDN has, the more likely you are to get a miss and hit the origin.

When the caches at POPs are fully populated, the effect on your application can be minimal. During a cache MISS event, typically either due to resource expiration or a manual purge, many requests can be sent to the origin server concurrently while the individual POPs rebuild their caches. The more POPs, the longer the process takes.

This can be a problem, especially when caching dynamic pages that need to be server side rendered, large resources, or transformed resources. For example, take a busy ecommerce store running Magento during a sale, Magento will purge content when sales are made, forcing the cache to rebuild each time. During a busy period it can reduce your cache hit rate and degrade site performance.

Enter Origin Shield

CDN Origin Shield is a feature that lets you nominate the CDN Point of Presence closest to your server as a shield. All requests that hit other POPs and receive a cache miss will then go to the nominated shield before hitting the origin. The shield becomes a 'super cache' and can reduce the amount of requests to your origin in a cache miss.

Peakhour.IO implements origin shield as a simple dropdown on an origin pool where you can select the geographic location that should be used as a shield. Requests to your origin are now routed through this geographic location before reaching your origin in a cache miss scenario.

Clients who use multiple geographic origins can also benefit from Origin shield. Peakhour.IO allows the specification of an origin shield per origin. For geographic load balancing, you will need to contact support for setup.

Seeing is believing

The Peakhour.IO summary now includes your edge CHR, your shield CHR and your overall CHR so that you can see the effect in action.

Some of our clients have seen typical increases of 10-20% of their overall Cache Hit Rate, and greater than 40% when specifically looking at often flushed dynamic content.
Quicker cache convergence
Fewer hits to origin
Better end-user experience
Higher conversions

Final Thoughts

Origin Shield is an important feature for certain types of site, or when you're looking to maximise your cache hit rate. CMSs that offer built in full page caching, like Magento and Drupal, flush content often, and are susceptible to performance degradation as load increases. Minimising hits to the origin in these cases is vital.

If you are interested in getting more out of your CDN, need a bespoke CDN solution, or need a provider that offers performance, optimisation and security services, reach out to discuss the right setup.

Origin shield with request collapsing helps minimise origin hits, improve CHR and maintain user experience for your web application.

Fastly Outage

2021-06-09T13:00:00+10:00

You may have heard that Fastly, one of the world’s largest providers of CDN services, had an outage of about 1 hour on the 8th July. Some of the world's largest websites and services were down, including reddit, CNN, The Guardian, Shopify Stores, Stripe and Spotify, to name a few.

According to Fastly themselves, the outage was caused by a 'service misconfiguration' (Update: Bug triggered by a client changing their configuration), which propagated globally and took websites offline. When users tried to access a website using the Fastly service, they were presented with a Varnish 503 Guru Meditation error (for those of us old enough to remember, Guru Meditation is a geek reference to the Commodore Amiga computer of the late 80s!). This generally occurs when there is an issue contacting the server that the website is actually hosted on. There were also some reports on twitter saying 'unknown domain'.

Essentially, Fastly took down its own network with a bad software update. Similar problems have affected other online platforms in the recent past, including Google, Amazon, and Cloudflare.

Why wasn’t there a Plan B?

Fastly is an excellent service, with an enviable reliability record. There is a reason why they're trusted by some of the world's largest websites to improve reliability and load times. However, the vast majority of Fastly clients still had to sit tight and wait for Fastly to fix the issue. Luckily this was only an hour. It could have been much longer.

Just like death and taxes, software outages are a certainty. The real story is not that Fastly had an outage. It is why didn't these large websites have a contingency plan for a single point of failure. For sites at that scale, this is a major oversight in infrastructure planning.

How to handle a CDN failure

The simple solution is to have a backup CDN provider already configured and tested, ready to switch over to if your primary provider fails. You can then utilise short expiry of DNS records to redirect users when the failure happens. This needn't be very expensive or complicated, although individual circumstances vary.

A Quick Introduction To DNS (Domain Name System)

Modern CDNs, like Fastly, Cloudflare, and Peakhour, operate as ‘reverse proxies’. This means they sit between a website's end users and the website server itself. They achieve this through DNS configuration.

When someone types a domain url into a browser, eg fastly.com, a request is sent to a DNS server with the host name (eg fastly.com) to find the IP address of the server to retrieve the content from. CDNs, like Fastly, get website admins to list the address of the CDN on the DNS server. That means requests for a website go through the CDN first. The process is analogous to listing someone else’s number in the phone book so they take calls for you.

The DNS server has a TTL (Time To Live) associated with its records. This TTL tells whoever asked for an IP address, for a given hostname, to remember the answer and not ask again until after the TTL has passed. Typically DNS record TTLs will be 1 hour, but they can be shorter, eg 1 minute.

Switching providers in case of an outage

By keeping a short TTL in DNS, webmasters can switch the answer for a DNS request to that of another provider, meaning users can quickly be directed to an alternative Cloud Provider. Once service has resumed on the primary provider, DNS can be switched again so normal traffic is resumed. The key is that the alternative provider is configured, tested, and ready to go.

This switch can even be automated to minimise outages. Premium DNS services, like Amazon’s Route 53, have optional health checking of DNS answers. This allows a switch to happen nearly instantly. The only downtime would be for people already on the site who have to wait for the TTL to expire before being directed to the backup Cloud Provider. In fact this is exactly what Peakhour.io does. In the event of a catastrophic outage we use DNS to switch to backup infrastructure so our clients are minimally affected.

Backup provider options

Now we've shown how switching CDN providers can be done, let's compare the major players and how they might serve as a backup CDN for Fastly. The three things we'll look at are Cost, Features, Integration.

Simply route traffic to the origin

This would be the simplest and most cost effective option, Assuming your origin server can handle the increased load that removing its CDN would entail. It also assumes that it's ok to lose any features that you may have been relying on, eg load balancing, WAF, edge scripting, image optimisation etc.

Cloudflare

Many people use Fastly because it uses Varnish, a richly featured, programmable cache with several advanced features. If you rely on those features, eg cache tags, cache on cookie value, custom cache tags, then you have to be on Cloudflare's top plan, which is not cheap.

The other major drawback of Cloudflare is that, unless you are on the most expensive plans, you have to cede control of DNS to them by delegating your domain. Cloudflare DNS is a great service, however it has the major drawback of caching negative DNS requests for an hour. If you were switching from an A record to a CNAME record or vice versa, you could be down for an hour regardless. Not ideal.

Akamai

Akamai has a highly respected, fully featured, and very expensive product. Maintaining a backup option with them will run into the $1000s a month. Only you can decide whether it’s worth it.

Cloudfront

Amazon's CDN offering is the third of the big three alternatives. Since it uses volume based billing, it could be an attractive CDN option as a standby, as long as you don't mind missing out on cache by tag (sorry Magento and Drupal). It is also complicated to configure for dynamic content and could miss features that you need. In fact most people use Cloudfront for static content, eg images, CSS, etc and run a Varnish instance within AWS to provide easier to configure full page caching.

This is what the BBC did with the Fastly outage. They had their backup infrastructure on Cloudfront and, as of time of writing, hadn't switched back to Fastly.

Peakhour.io

Peakhour is also volume based billing with a minimum monthly charge of $20. We provide all the advanced caching features that Fastly does, as well as WAF and image optimisation as standard, all in the one service fee. We don't require you to cede control of DNS to us and we're Australian owned and based.

Final Thoughts

CDNs, no matter how big, can fail. If your website is important then it needs a Plan B. This is how that Plan B works, and it doesn't have to be expensive when using a provider like Peakhour.io.

The important part is having it configured and tested before you need it.

Instant Alerts

2019-05-31T13:00:00+10:00

We've introduced Instant Alerts, an optional service that notifies you when there is a problem with your site. You can specify the email addresses and/or mobile numbers that should receive the alerts.

At the moment you can receive alerts when your origin server:

Times out
Cannot be reached at all
Returns an error

To avoid repeated messages, each alert has a cooldown limit you can set from 30 minutes to 24 hours.

If you're using a service like Pingdom to check that your site is up, you probably will want to enable Instant Alerts. Services like Pingdom might be served pages from our cache while your origin server is actually down.