Peakhour.IO - Bot Management

Using Network Fingerprints in Bot and Rate-Limit Decisions

2026-08-23T09:00:00+10:00

A network fingerprint is a useful rate-limit key. It is a poor verdict.

That sounds like a small distinction, but it changes how a control behaves. A limit keyed only by source IP can miss a distributed scraper moving through thousands of addresses. Add a TLS or HTTP fingerprint and the requests may fall into a recognisable cohort. The operator can count and compare them even while the IPs rotate.

The same key can also group thousands of ordinary users running the same browser release. If the control reads “this fingerprint is a bot” and blocks it everywhere, the grouping feature has become a false-positive multiplier.

The practical job is to use the fingerprint where grouping helps, then make the decision from the route, behaviour and consequence. This article sets out one way to do that.

Start with the route, not the fingerprint

A request for a cached image and a request to submit a password do not deserve the same policy. Neither do a search request and an export that starts an expensive database job.

Before adding fingerprint data, classify the routes that matter:

what the request can change or disclose;
whether it consumes scarce application or origin capacity;
whether a legitimate user can retry safely;
whether the client can complete a challenge;
which identity is available, such as an account, session, API key or no identity at all;
what an incorrect block would cost.

This route map sets the response. An unfamiliar cohort fetching public content might only be worth observing. The same cohort attempting passwords across many accounts may justify a tight shared limit. A fingerprint mismatch on an authenticated payment action may call for step-up verification rather than a network block.

Route sensitivity also prevents a common emergency mistake: applying a site-wide rule because an attack was visible on one endpoint. Narrow rules are easier to explain, test and remove.

Use the fingerprint as a cohort key

JA3, JA4 and Cisco Mercury do not identify a person or device. Each method selects and normalises protocol fields, then groups connections that look equivalent under those rules. Our same-ClientHello lab shows how three methods produce three different representations from one handshake.

That makes the fingerprint useful for questions such as:

Are login failures arriving from many IP addresses but one narrow set of client stacks?
Did an expensive API route suddenly attract a new protocol cohort?
Does traffic claiming to be a browser have a consistent TLS, HTTP and header shape?
Did a cohort disappear after an abusive tool changed version, or did the fingerprinting rules change underneath us?

The key should be composite. A workable rate counter might resemble:

route_class + fingerprint_method + fingerprint_version + fingerprint_value

Depending on the abuse case, add an account, tenant, API credential, ASN or country. Do not throw every field into every key. An over-specific key divides a distributed campaign into buckets too small to see; an over-broad key combines unrelated users.

Run a few candidate keys in observation mode and measure their bucket sizes. Look at both tails. A key that places half the site's users in one bucket is too blunt for direct enforcement, however suspicious that bucket looked during one incident.

Record where the value came from

Fingerprint provenance is operational data, not documentation trivia.

At an origin behind a CDN or reverse proxy, the visible TLS connection may belong to that intermediary. A fingerprint forwarded in a header might describe the original client-facing connection, or it might describe a later hop. The receiving service cannot tell from the hash alone.

For each event, keep at least:

method:       ja4
version:      pinned implementation or rule version
value:        t13d..._..._...
capture:      client-facing edge
source:       named collector or trusted forwarding hop
observed_at:  timestamp
policy:       policy identifier and revision
action:       observe, challenge, rate-limit or block

Where the format permits it, retaining the selected raw fields or unhashed representation makes later comparison much easier. Cisco Mercury's full Network Protocol Fingerprint is structured and versioned; its optional hash is a compact nickname that discards that structure. JA4 publishes both a canonical hashed form and a raw JA4_r form. The details are in the Mercury NPF specification and JA4 technical specification.

Trust the forwarding path as deliberately as any other security metadata. Strip client-supplied copies at the boundary, set the authoritative value there, and record the hop that produced it. Otherwise an attacker may be able to choose the value used by the policy.

Use an action ladder

Bot controls work better with space between “allow” and “block”. A simple ladder is observe, challenge, rate limit, then block. It is not a mandatory sequence for every request; it is a way to match disruption to confidence.

Observe

Observation is the right first action for a new cohort, a changed implementation or a weak hypothesis. Record volume, routes, account outcomes, response codes, request cost and how the cohort overlaps with known legitimate clients.

Set an end date before starting. “Log this for seven days and review on Monday” produces a decision. “Log this” often produces another permanent stream nobody owns.

Challenge

A challenge asks the client for more evidence. It can separate some interactive browsers from simple automation, but it is not a universal test of humanity. Browser automation can complete challenges; privacy software, accessibility tools and broken JavaScript can make legitimate users fail them.

Use challenges where the client can reasonably complete one and where failure has a safe recovery path. They are a poor fit for machine-to-machine APIs unless the protocol already defines an authentication or proof step.

Rate limit

Rate limiting fits repeated or costly behaviour. Count a cohort against the route it is affecting, then layer in the identities that make sense for that route. For example:

login:  fingerprint + account + failed outcome
search: fingerprint + route + request cost
API:    fingerprint + tenant or credential + operation

A limit should say what it protects. Requests per minute is sometimes enough. Expensive operations may need a cost-weighted budget. Login protection may count failed attempts while allowing successful account recovery to proceed. Distributed abuse may need a cohort-wide ceiling plus per-account and per-IP limits so that no single key carries the whole policy.

Return an explicit response and keep the retry behaviour predictable. The HTTP 429 Too Many Requests status was defined for rate limiting in RFC 6585, including the option to send Retry-After. Clients still need sensible backoff, and operators need to watch whether retries are making origin pressure worse.

Block

Block when the evidence is strong, the harm is current, and a less disruptive action cannot protect the route. Good candidates include a confirmed exploit tool hitting the vulnerable route, a cohort causing active availability loss, or repeated abuse that has failed narrower controls.

Keep the scope bounded: fingerprint method and version, affected route, relevant behaviour, start time, owner and expiry. A bare fingerprint deny-list with no incident context is hard to audit and tends to outlive its evidence.

For a broader mapping of actions to evidence, see Network Fingerprint Signals and Security Decisions and Network Fingerprinting for Rate Limiting.

Combine evidence before increasing friction

The most useful policy does not ask whether a fingerprint is good or bad. It asks whether several observations agree.

Consider a login campaign. The source addresses rotate through residential networks. The TLS cohort is common because the tool drives a real browser. The fingerprint alone is weak. But the same traffic may attempt many accounts, reuse a route sequence, fail at a high rate, omit normal session history and arrive at a cadence unlike ordinary users. Together those observations justify a shared limit or challenge.

Now reverse the example. A rare TLS cohort signs in successfully to one account from its usual region, follows a normal route sequence and maintains an established session. Rarity does not add much. Blocking it would mostly punish an unusual client.

This is also the line between a format and a detection system. JA4 defines a fingerprint. Mercury defines fingerprint representations and includes separate analysis components. Any claim such as “this is Chrome”, “this is a scraper” or “this is malicious” comes from surrounding data and inference, not from the fingerprint characters themselves. A Network Fingerprint Is a Cohort, Not a Client explains that boundary in more detail. Fingerprints Are Evidence, Not Identity covers the same problem across browser, network and behavioural fingerprints.

Expect versions and populations to drift

Browsers ship, TLS libraries change defaults, mobile applications update unevenly and attack tools copy popular handshakes. Fingerprint implementations also change their parsing rules. Store the method and implementation version so client drift can be separated from detector drift.

Before changing an implementation, calculate old and new values side by side on a sample. Compare cohort sizes and known-good traffic by route. A partner API, mobile application and public website have different expected populations; a global label may be irrelevant to the local control.

Put expiry and rollback in the policy

Every enforcement rule needs an owner, reason, expiry and removal condition. Before enabling it, record request volume, successful transactions, challenges, 429 responses and origin health. Compare the same measures afterwards. A falling attack rate is not success if legitimate completion falls with it.

When a false positive appears, retain the fingerprint version, capture point, contributing evidence, route, identity, policy revision and rollback result. That record shows whether to change a threshold, split a route class, add an exception, demote the action or remove the fingerprint from the key.

A deployable first policy

For a first production use, choose one abused route. Generate a versioned fingerprint at a trusted capture point, log it beside the route and outcome, and observe cohort sizes long enough to include normal variation. Then choose one reversible action with success measures, rollback and an expiry.

The fingerprint has done its job when it helps a team see and control a group of requests that an IP-only rule would miss. It has exceeded its job when the group is treated as proof of a client, person or intent.

A Network Fingerprint Is a Cohort, Not a Client

2026-08-09T09:00:00+10:00

A TLS fingerprint usually tells you that two connections look alike under a particular set of rules. That is useful. It is not the same as showing that they came from the same client.

The distinction becomes obvious when a common TLS library sits underneath many programs. Those programs can offer the same protocol version, cipher suites, extensions and signature algorithms. A fingerprint built from those fields groups them together even though their purpose, owner and risk are different.

The reverse also happens. One application can generate several fingerprints after a browser update, operating-system change, feature rollout or configuration difference. A fixed application name does not imply a fixed ClientHello.

The safest mental model is a cohort: traffic that looks the same after a fingerprint method has selected and normalised its inputs.

The method defines the cohort

JA3 preserves the order of its selected ClientHello feature lists. Change the order and the MD5 value changes. JA4 sorts selected identifiers before calculating two of its components, so permutations that split a JA3 cohort may remain grouped under JA4.

Cisco Mercury can preserve more packet-derived structure in its full Network Protocol Fingerprint. Different Mercury rule versions also make different normalisation choices. A tls/2 value is therefore not just a longer spelling of JA4. It is the result of another feature-selection contract.

This means there is no format-independent "real fingerprint" hiding underneath the tools. Each method answers its own equivalence question: which differences count, and which should be ignored?

Our same-PCAP lab demonstrates that point without relying on a hypothetical browser. The three tools inspect the same ClientHello and produce representations with different retained detail.

Common does not mean safe

A popular browser fingerprint will naturally appear in a great deal of legitimate traffic. An attacker can also use a browser, drive one through automation, or imitate its TLS stack. Matching a common browser value therefore says little about intent by itself.

The opposite shortcut is just as risky. An uncommon fingerprint is not proof of malware. Internal tools, older mobile applications, embedded devices, accessibility software and regional client variants can all be rare in one dataset.

Rarity is relative to the observation point. A fingerprint that is common across a public content site may be unusual on an administrative API. A value common in one country, network or month may be rare in another.

Capture point changes what you see

TLS fingerprinting only works where the relevant handshake is visible. At an origin behind a CDN or reverse proxy, the TLS connection may have been terminated and replaced upstream. The origin can then observe the proxy's connection rather than the end user's ClientHello unless the edge explicitly forwards a derived fingerprint.

That forwarded value also needs provenance. Operators should know which implementation and format version produced it, whether it came from the client-facing connection, and whether middleware transformed or sampled the traffic.

Without that information, two values that look compatible may have been generated under different rules. Fastly has documented how implementation differences can undermine the portability promised by a shared hash in The State of TLS Fingerprinting.

Context can improve an inference

Cisco's Mercury research is helpful because it does not hide the ambiguity. The 2020 destination-context paper reports that one TLS fingerprint often maps to tens or hundreds of process names. Its classifier adds destination IP address, port and server name, backed by a labelled knowledge base, to rank the possible processes.

That is stronger than treating a bare fingerprint as an application name. It is still conditional. Change the environment, the age of the knowledge base, the available destination fields or the software population and the probabilities can change. The paper's reported accuracy belongs to its datasets and experimental design, not to every network that runs Mercury.

JA4 deployments often add context too. A security platform might combine the JA4 value with request rate, geography, path, account state or observations from other customers. Those extra fields are not secretly part of JA4. They are features in the surrounding detection system.

Use fingerprints where grouping helps

Fingerprints earn their place when grouping similar connections improves an investigation or control. Examples include:

counting login failures across rotating IP addresses;
finding a TLS stack that appeared at the start of an incident;
comparing a claimed browser with HTTP and browser-side evidence;
monitoring drift after a client or library release;
selecting traffic for review before writing a narrower rule.

For the decision and enforcement consequences, see Fingerprints are evidence, not identity. The protocol-specific conclusion here is narrower: a matching TLS fingerprint places connections in a cohort defined by one method. It cannot tell you, on its own, who is on the other end.

The practitioner follow-up, Using network fingerprints in bot and rate-limit decisions, turns that boundary into a route-scoped policy and rollback model.

How to fix your Adobe Commerce Overages with Fastly

2026-07-01T08:13:00+10:00

Adobe Commerce merchants do not usually notice Fastly overages straight away.

Traffic looks normal. Revenue looks normal. The site feels fine. Then the bill lands and CDN usage is well above contract.

In our experience, the drivers of CDN overages are due to Adobe Commerce and Fastly default behaviour.

The biggest culprits are:

shielding
ESI forcing uncompressed HTML from origin
PNG-heavy catalogues and storefront assets
bot traffic pouring fuel on all of the above

We recently worked with Dymocks and helped them reduce their CDN usage from 55 TB of monthly traffic down to 6 TB after we fixed the waste in the delivery path and filtered unwanted traffic, case study coming soon.

1. Shielding helps performance, but it can hurt your bill

Fastly shielding exists for a good reason. It reduces origin load and can improve cache efficiency by funnelling misses through a shield POP before they go back to origin.

The catch is billing.

Fastly is clear that traffic from one POP to another counts towards both request count and billable bandwidth. In the most extreme case, if everything is effectively a pass, request count and delivery bandwidth can almost double because the request is handled by two Fastly POPs instead of one.

That means a cache miss is no longer a simple edge-to-origin event. It becomes:

edge → shield → origin → shield → edge

If your cache hit rate is strong, that trade-off may still be worth it. If page caching is weak, purges are frequent, or large parts of the site are missing cache, shielding turns into a billing multiplier.

2. ESI (Edge Side Includes)

ESI is an old technology that allows a proxy (like Fastly) to 'assemble' pages using content fetched from another source.

Adobe Commerce/Magento 2 will use ESI out of the box for its navigation bar. The reasoning is to allow the navigation to be purged from the CDN independent of the page content. In theory this makes Full Page Caching more effective. In practise the difference is negligible as the Full Page Cache is cleared daily anyway.

Fastly’s ESI documentation says ESI cannot be performed on content compressed at origin. Fastly needs to inspect the response body for <esi:...> tags, and compressed content hides those tags from the parser.

The default Fastly configuration for Adobe Commerce will strip the client Accept-Encoding header before the request goes to origin.

The origin then returns uncompressed HTML, Fastly scans it for ESI, and only later is the response compressed again at the EDGE before it is delivered to the browser.

From the shopper’s point of view, nothing looks wrong. The page still arrives compressed at the edge. Performance may still look fine.

From a billing point of view, it is a very different story.

Magento’s default layout gives the top navigation block a ttl="3600" in default.xml, which makes it an ESI candidate out of the box. The Fastly Magento VCL snippets also show the downstream effect: Fastly sets x-compress-hint because “varnish doesn't compress ESIs”, so compression is pushed later in the chain.

Now combine that with shielding.

Say your page is:

50 KB compressed
600 KB uncompressed

If the request misses cache and shielding is enabled, you are not paying for a tidy 50 KB response. You are moving the 600 KB object through the shield path, and that larger object is counted across both the shield and edge legs.

That is effectively a 600 KB × 2 event.

That is why broken page caching hurts twice:

you get more misses
each miss is much larger than most teams realise

If enough HTML is missing cache, overages rise very fast.

3. A simple way to spot the problem in New Relic

An easy way to check for ESI driven egress is to open New Relic and look at CDN usage by content type.

If text/html is driving more usage than images, that is a strong sign you are paying for uncompressed page responses on cache misses.

text/html content type being the top CDN usage content type is a red flag.

That pattern usually points back to the same combination:

ESI
shielding
weak cache hit rates

On a healthy eCommerce site, images are usually the major share of bandwidth. When HTML starts dominating, something is off.

4. PNGs are another quiet cost driver

Fastly can absolutely turn a PNG into WebP, but that does not automatically mean you get a small lossy image.

Fastly’s own Image Optimizer reference says that if the source image is a lossless format, the output defaults to lossless when the chosen output format supports it. Fastly is explicit about the PNG case: with a PNG source and format=webp, format=jxl, or format=auto, the output image is lossless by default. In other words, Fastly can transform a PNG into a lossless WebP unless you explicitly force a lossy path.

That is why PNG-heavy catalogues can still burn a lot of bandwidth even when teams think Fastly is “optimising” them. The image format may change, but the delivery path can still stay lossless, which means the file is often much larger than expected. Fastly also notes that lossless output preserves all image data and therefore produces larger file sizes, basically the same as the source image.

This is where many Adobe Commerce stores get caught out. Most merchants rely on the default Fastly image setup and do not add per-image parameters or rewrites. Fastly’s documentation says that most image transformations require query string parameters on individual requests, and its default behaviour notes that WebP quality is 85 for lossy WebP images, not that every PNG source is automatically pushed into lossy Webp.

If you want smaller files, you need to force the outcome. Fastly says you can do that by using format=auto with quality<100, or by explicitly using a lossy WebP path. Adobe Commerce’s Fastly image optimisation settings also expose a Force lossy conversion option for this reason.

What to do about it

You do not need a full replatform to reduce overages. Most of the wins come from cleaning up the delivery path.

Review shielding

Start by measuring shielding, not assuming it is free.

Disabling shielding can be a fast way to cut billable traffic. The trade-off is lower cache efficiency and more origin load, so this needs testing, not guesswork.

Remove ESI where it is not earning its keep

On many Adobe Commerce builds, ESI is there mainly so the navigation block can be flushed independently.

That is rarely a good reason to pay for inflated HTML misses.

The usual fix is:

change the Magento layout or template so the block no longer needs ESI
strip the X-ESI path in VCL
restore Accept-Encoding to origin requests so HTML can travel compressed again

This is often the biggest bandwidth win because it reduces the size of every miss, not just edge delivery.

The immediate effect of disabling ESI and enabling compression from origin.

Force lossy png conversions

Fortunately forcing lossy conversion of PNG images is relatively straightforward.

Use VCL to rewrite image urls to include a query string, eg

    /someimage.png

becomes

    /someimage.png?format=auto&quality=85

Expect a 400kb source image to go to 100kb for no discernable difference. Play with quality for bigger gains.

Do not ignore bots

Then there is the traffic you never wanted in the first place.

Bad bots amplify every weakness above. They hit uncached pages, search pages, category pages, and dynamic endpoints. That creates more misses, more uncompressed HTML fetches, more shield traffic, and more wasted bandwidth.

This is why CDN cost control is not just a caching conversation. It is also a traffic quality conversation.

If you can stop junk traffic before it burns CDN bandwidth, the savings stack on top of the caching fixes.

That is where Peakhour fits. We help stores cut unwanted bot traffic before it turns into Fastly overages, and the upside can be dramatic. For example, we've helped one store go from 55 TB per month down to 6 TB.

Final thought

If your Fastly bill feels out of proportion to real shopper demand, start with two questions:

Why are we missing cache so often?
Why is text/html using so much bandwidth?

If those numbers look wrong, the overage usually stops being mysterious.

API Bot Abuse Does Not Stay in One Endpoint

2026-06-19T00:00:00+10:00

API bot abuse rarely stays politely inside one endpoint.

A campaign may start at login, move through token refresh, test account recovery, check saved addresses, probe checkout, abuse coupons, scrape product availability, and then return to account actions once a working session is found. Looking at each route alone makes the activity seem smaller than it is.

The pattern matters more than the individual request.

That is why API bot protection has to follow journeys, not just endpoints.

Login Is the First Measurement Point

Login endpoints are the obvious place to look for bot abuse. Credential stuffing, password spraying, brute force attempts, and token abuse all show up there.

But the login endpoint is only the first measurement point. Attackers are not trying to create a failed-login graph. They are trying to find usable accounts and move to the next action.

A bot campaign may keep failed attempts low per IP address. It may rotate through residential proxies. It may slow down to avoid simple thresholds. It may mimic the browser request shape closely enough to get past a basic check. It may use an API route that exists for the mobile app rather than the public login form.

So the useful question is not just "how many login attempts did we see?" It is "which client, route, network, fingerprint, account, and response pattern suggests automation?"

That is where bot management belongs in the account journey. It should not be an isolated "human or bot" label. It should become part of the request evidence used for login, recovery, checkout, and account-change decisions.

Checkout Abuse Is Often API Abuse

Checkout abuse is not always a stolen-card problem. It can be a request-path problem.

Bots can test coupon codes, reserve inventory, create carts, check delivery combinations, retry payment flows, and exploit business logic at machine speed. Some of this happens through visible browser journeys. Much of it happens through APIs used by the front end or mobile app.

The damage is not always dramatic in a single request. A few extra cart creations may look normal. A small number of coupon checks may be expected. A payment retry can be legitimate. The problem is the campaign shape across routes.

If the same automation profile moves through login, cart, promo, shipping, and payment APIs with abnormal timing or sequencing, the response should not depend on one endpoint crossing a crude global limit.

It should be route-aware.

A checkout API can tolerate different behaviour from a catalogue API. A payment route deserves different thresholds from a product search route. A coupon route may need controls around account age, session state, rate, and client evidence. A cart route may be harmless in one context and abusive in another.

Account Journeys Need Sensitive-Action Controls

Account abuse becomes most damaging when a session moves into sensitive actions.

Changing an email address, resetting a password, adding a new delivery address, viewing stored payment details, redeeming loyalty value, or placing an order are different from normal browsing. They deserve stronger context.

The request may be technically valid. The token may pass validation. The password may be correct. The API schema may be satisfied. That does not mean the action is safe.

A strong control model looks at the full path into that action. Did the session begin with credential stuffing signals? Is the client first seen? Did the network change? Is there proxy or fingerprint drift? Has the account recently failed login attempts? Is the request cadence consistent with a human journey? Is the action unusually soon after authentication?

These are not abstract "zero trust" slogans. They are practical checks on the account request path.

Rate Limits Need Better Keys

API abuse prevention often starts with rate limiting, but IP-only limits struggle with distributed automation and shared networks. The hard part is deciding what to count.

For API bot abuse, useful rate keys can include route, method, account, token, API key, response code, ASN, country, TLS or HTTP fingerprint, verified bot state, and combinations of headers. The right key depends on the journey.

A login endpoint might count failed attempts by account and fingerprint. A token endpoint might count refresh patterns by client and session. A checkout route might count attempts by account, payment state, and client fingerprint. A partner API might count by API key and route.

Advanced rate limiting is valuable because it can model the abusive actor more precisely than a single IP address. It also gives teams response options short of blanket blocking: log, challenge, throttle, or deny depending on the route and risk.

That matters because real API traffic includes customers, mobile apps, partners, service clients, good bots, bad bots, and increasingly AI-driven agents.

Agents Will Make the Journey Problem Harder

The next wave of automated API use will not all look like simple scripts. As discussed in Agentic AI vs. Your API, reasoning agents can explore, adapt, and change their behaviour based on responses.

That does not mean every AI agent is malicious. It does mean endpoint-by-endpoint rules will age quickly.

A reasoning agent can try one path, observe the result, and adjust. It can move from documentation to browser-backed APIs to mobile-shaped requests. It can test which routes are protected, which errors reveal state, and which actions trigger stronger checks.

Defence needs the same journey view. The campaign should be visible as it moves, even when the exact request pattern changes.

Keep Evidence Attached to the Campaign

API bot abuse is easier to manage when the evidence stays attached.

The useful record is not just "blocked by rule 42". It is the route, account state, token or key context, fingerprint, proxy signal, rate key, response pattern, action taken, and protected business step. That evidence lets teams tune controls without guessing and investigate incidents without reconstructing the whole path from raw logs.

For broader background, API abuse prevention covers the categories. The operational point is narrower: login, checkout, and account APIs should not be defended as separate islands.

Attackers use the journey.

The defence should too.

API Protection and Account Protection Are One Request-Path Problem

2026-06-19T00:00:00+10:00

Account protection is often discussed as if it belongs to the login page. That made sense when most account abuse looked like someone submitting a username and password into a web form.

That is not how modern account journeys work.

A customer signs in through a browser, a mobile app, a partner integration, a password reset flow, a token refresh endpoint, a profile update request, a checkout API, and sometimes a service-to-service call that exists nowhere in the visible front end. The account is not protected by one screen. It is protected, or exposed, by the whole request path.

That is why API security and account protection should not be treated as separate operating problems. The API route, the identity context, the client evidence, the rate pattern, the token behaviour, the bot signal, and the account action all arrive together. Splitting those signals across disconnected tools makes the final decision weaker.

The Login Is Only the Start

Credential stuffing is the obvious example. Attackers test leaked credentials against login endpoints, but the useful outcome is rarely the login itself. The value comes after the session opens.

They try to change the email address. They add a shipping address. They reset a password. They check stored cards. They redeem loyalty value. They place an order. They call account APIs that were built for the real customer journey and then abuse them in a different sequence.

If the login defence is separate from the API defence, the organisation may see only fragments:

A spike in failed logins in one dashboard.
A suspicious token refresh pattern somewhere else.
A burst of profile-change requests in application logs.
A fraud case after checkout.

Each of those is useful. None is the full story.

The better model is to keep the decision close to the request. A request to /login is different from a request to /account/email, /checkout/payment, or /api/token/refresh. The route matters. So does the method, the session state, the previous failures, the network, the client evidence, and the account action being attempted.

APIs Carry Account Risk

APIs are not just developer plumbing. They are where many account journeys now happen.

Mobile apps use APIs for login, registration, password reset, saved addresses, payment methods, and checkout. Single-page applications call APIs behind browser journeys. Partner systems may call account or order APIs directly. Internal services may use API keys or service credentials that function like non-human accounts.

That creates a practical issue: account protection must cover both human and non-human identity paths.

OAuth, JWTs, API keys, refresh tokens, and service credentials all need lifecycle control, least-privilege access, rotation, validation, and monitoring. But those controls are still not enough if the protected API cannot see whether the request is behaving like abuse.

A valid token can be stolen. A valid API key can be overused. A real session can be driven by automation. A known customer can suddenly perform a high-risk action from a first-seen client through proxy infrastructure.

The request has to be judged in context.

Rate Limiting Has to Follow Business Logic

Basic rate limiting often starts with an IP address. That is an understandable first step, but it is not enough for account protection. Attackers rotate through proxy networks. Legitimate users may share a carrier or office IP. Some attacks are low and slow enough that no single IP looks exceptional.

For account journeys, rate limiting needs to be tied to the thing being protected.

Login attempts can be counted differently from password resets. Token refreshes can be counted differently from product searches. A failed authentication response can be treated differently from a normal read request. A route hit by a first-seen client can be treated differently from one used by a known browser session.

That is the point of API protection as an operating layer, not just an API inventory exercise. The route, schema, authentication state, bot signal, and rate key should be available to the same decision. Otherwise teams end up writing compensating rules in several systems and hoping the gaps line up.

The Edge Decision Needs Options

Not every suspicious request should be blocked. Some should be logged. Some should be rate limited. Some should be challenged. Some should be allowed because the business impact of a false positive is worse than the risk presented by that specific request.

Account protection is strongest when the action matches the journey.

A login request with weak risk signals might be allowed but watched. A password reset request with stronger signals might require step-up. A checkout attempt from a newly compromised session might be blocked or reviewed. A partner API key exceeding expected usage might be throttled without affecting normal customers.

Peakhour's position here is simple: API, bot, WAF, rate, and account controls work better when they share request evidence. That can run on Peakhour Edge, or it can sit beside the CDN and cloud edge already in place through bring your own edge. The important part is not the label on the component. It is whether the request path has enough context to make the right decision.

Account Protection Is a Journey Control

A useful account protection programme should be able to answer operational questions.

Which routes are involved in account takeover attempts?
Which sessions moved from suspicious login behaviour into account changes?
Which tokens or API keys are behaving outside their expected pattern?
Which controls created friction, and where?
Which blocked requests actually protected account, checkout, or recovery actions?

Those questions cannot be answered from a login form alone. They require API visibility, account-event context, rate data, bot signals, and reviewable evidence.

That is the thesis of this series: API protection and account protection are one request-path problem. The account is compromised through a sequence of requests. The defence needs to see that sequence early enough to act.

For teams already working on account takeover prevention, the next step is not simply adding another login prompt. It is connecting the account journey to the API routes that now carry it.

The Real Cost of Breached Credentials

2026-06-19T00:00:00+10:00

The cost of breached credentials is usually counted in the wrong place.

When an organisation suffers a data breach, the obvious costs are incident response, legal work, notification, customer support, remediation, and regulatory attention. Those costs matter. IBM's 2025 Cost of a Data Breach Report puts the global average breach cost at about USD 4.4 million. IBM's data breach explainer also says stolen or compromised credentials were one of the top five initial attack vectors in the 2025 report, accounting for 10% of breaches and taking up to 186 days to identify.

But that is only the first bill.

Once usernames and passwords leave the original system, they do not stay attached to the original incident. They are copied, sorted, bundled, tested, resold, and mixed with other personal data. Another company's breach becomes your login problem. A password reused somewhere else becomes your fraud queue, your support call, your chargeback, your locked account, your angry customer, and your next security review.

That is the real cost of breached credentials: not just the breach, but the long tail of account abuse that follows.

The Roundup: Breaches Are Feeding Account Abuse

The numbers are not subtle.

The Identity Theft Resource Center's 2025 Annual Data Breach Report tracked 3,322 data compromises in 2025, a record high and a 79% increase over five years. The same report found that 70% of breach notices did not include attack information, making it harder for consumers and downstream businesses to understand what risk they now carry.

The ITRC also introduced a category it calls Previously Compromised Data: old stolen data that is repackaged and recirculated. In the full report, the ITRC says there were four major PCD releases in 2025, including two incidents involving roughly 16 billion records with no known notices. Its warning is the important part: while this may not be "new" stolen data, aggregation makes it highly effective for credential stuffing and account takeover attacks.

That matches the operational pattern security teams see on login endpoints. OWASP describes credential stuffing as automated testing of stolen username and password pairs against login forms. The reason it works is boring and persistent: people reuse passwords. Attackers do not need to breach your site if a customer has already reused a working credential somewhere else.

For Australian organisations, the local signals are just as relevant. The OAIC received 532 Notifiable Data Breach notifications between January and June 2025, with malicious or criminal attacks remaining the largest source of notifications. ASD's Annual Cyber Threat Report 2024-25 notes that its credential exposure notification process proactively sent 9,587 credential exposure events to about 220 organisations between 19 November 2024 and 30 June 2025.

None of that means every fraud loss starts with a reused password. It does mean credential exposure is part of the operating environment. Attackers have supply, tooling, proxy infrastructure, and plenty of places to turn account access into money.

The FBI's 2025 IC3 report gives useful context for that monetisation path. Cyber-enabled fraud accounted for 452,868 complaints and USD 17.697 billion in reported losses. Those losses include many crime types, not just credential stuffing, but the transaction paths are familiar to anyone dealing with account abuse: wire and ACH transfers, cards, peer-to-peer transfers, prepaid and gift cards, and cryptocurrency.

Where the Cost Actually Lands

A breached credential is cheap for the attacker and expensive for everyone else.

The first cost is detection. A login using the right username and password does not automatically look malicious. If the attacker spreads attempts across residential proxy infrastructure, uses one attempt per account, or targets mobile API endpoints directly, simple IP-based rate limits may not see the pattern. Peakhour has written about this in The Australian epidemic of Account Takeover attacks and in Credential Stuffing Does Not Stop at the Login Form.

The second cost is fraud. Once a credential works, the attacker looks for value: stored cards, gift cards, loyalty points, refunds, store credit, subscription changes, delivery addresses, and saved payment flows. This is why account takeover is not just an authentication problem. The expensive moment may be checkout, not login.

The third cost is support. Customers do not usually know whether the original password leak happened somewhere else. They know their account was used, their card was charged, their loyalty balance disappeared, or their email address changed. The business still has to handle the support ticket, freeze the account, unwind the transaction, review the evidence, and explain what happened.

The fourth cost is trust. We have covered this before in The Cost of Credential Stuffing: the reputational damage is practical. Customers see refunds, account locks, suspicious messages, and public complaints. Even if the business was not the source of the original breach, it becomes the place where the harm is felt.

The fifth cost is friction. If the only response is to challenge everyone, the business pays through abandonment and customer frustration. If the response is too soft, the business pays through fraud. The work is to apply friction where the evidence justifies it.

You Do Not Need Surveillance to Secure Accounts

There is a bad version of account protection that tries to identify people everywhere they go. That is not necessary, and it is not the right model for this problem.

Credential abuse defence should be scoped to the account security decision in front of you. Is this login using a known exposed credential pair? Is the session coming from suspicious infrastructure? Is it a first-seen device for the account? Is it trying to change email, reset the password, add a payout method, redeem stored value, or check out with saved payment details? Did the same client pattern just test many accounts?

Those questions can be answered with security-specific signals, not advertising-style tracking. Hash the credential check. Treat fingerprints as evidence, not identity. Keep the evidence tied to the protected account and request path. Use network, device, route, behaviour, and credential-risk context to decide whether to allow, step up, throttle, block, or review. Do not build a cross-site identity graph when the job is to stop account abuse on your own service.

That distinction matters. Users should not have to trade privacy for basic account security. Businesses also do not need to choose between doing nothing and adding blanket friction. Contextual security is useful because it lets the response match the risk.

What Teams Should Measure

If breached credentials are a business cost, measure them like one.

Useful measures include:

How many login attempts match known breached credential pairs.
How many breached-credential attempts result in a successful login.
Which routes see the risk: login, password reset, email change, stored-card checkout, gift card redemption, account recovery, mobile API, partner API, or admin access.
How often high-risk sessions move from login into sensitive account actions.
Which signals appear together: breached credential, residential proxy, first-seen device, unusual geography, repeated failure, rapid checkout, or recovery-flow pressure.
How many support tickets, refunds, chargebacks, account locks, and fraud reviews are linked to account takeover.
How many controls create customer friction, and whether that friction is landing on risky sessions or ordinary customers.

This does not need to be perfect on day one. The important step is to stop treating credential stuffing as a vague security category and start treating it as an observable account-risk workflow.

The Control Pattern

The control pattern is layered.

Start with breached credential scanning so reused or exposed credentials are visible at login. Feed that signal into account takeover prevention rather than treating it as a standalone report. Add bot management and advanced rate limiting so automation and distributed testing are harder to run at scale. Use residential proxy detection as a risk signal, especially where attackers are trying to make automated traffic look like normal customer traffic.

Then carry the risk forward after login.

A low-risk page view and a saved-card checkout should not inherit the same level of trust just because the password worked. A session that begins with a breached credential match, comes through suspicious infrastructure, and immediately changes the email address or redeems stored value deserves a different response from a known customer browsing order history.

The response can be graduated:

Log low-risk activity for visibility.
Tighten rate limits on suspicious automation.
Require step-up verification before sensitive account changes.
Hold or review risky transactions.
Notify the customer when high-risk account changes are attempted.
Block sessions when the evidence is strong enough.

That is how breached credential data becomes useful. It is not a panic button. It is a signal that helps decide when trust should be earned again.

The Practical Takeaway

Breached credentials are not only a breach-response issue. They are an account protection issue, a fraud issue, a support issue, and a customer trust issue.

The original breach may have happened somewhere else. The cost can still land on your login form, your checkout, your API, and your support team.

The goal is not to make every login difficult. The goal is to make stolen credentials harder to turn into account control, money movement, stored-value abuse, or customer harm.

That starts by making credential risk visible, connecting it to session and route context, and applying proportionate controls where the cost would otherwise show up.

Credential Stuffing Does Not Stop at the Login Form

2026-06-19T00:00:00+10:00

Credential stuffing is usually described as a login problem. An attacker takes breached username and password pairs, tests them against a site, and tries to find accounts where people reused passwords.

That description is accurate, but incomplete. The login is only the first gate. The real damage often happens in the flows that follow a successful login: password reset, email change, saved cards, loyalty balances, gift cards, account recovery, address changes, and checkout.

If account protection only watches the login form, it can miss the part of the attack that matters most to the business.

A successful credential stuffing attempt does not always look dramatic. The attacker may have valid credentials. They may come through residential proxy infrastructure. They may spread attempts across many IP addresses. They may slow the attack down to stay below simple thresholds. If the site treats a valid username and password as the end of the risk decision, the attacker inherits whatever the account can do.

That is why account takeover prevention needs to cover account actions, not just authentication.

The Attack Continues After the Password Works

Once an attacker is inside an account, they usually want persistence, value, or both.

A password reset or password change can lock the real customer out. An email change can move alerts, receipts, and recovery messages away from the owner. A phone number change can weaken later verification. A new shipping address can redirect physical goods. Stored payment methods can turn account access into immediate fraud. Gift cards, store credits, loyalty points, and refunds can be easier to monetise than a card transaction.

Checkout is often where the compromise becomes visible, but the risk builds earlier. A login from unfamiliar infrastructure followed by a profile change, then a saved-card purchase, is different from a returning customer browsing previous orders. A dormant account that suddenly redeems gift cards, changes email, and ships to a new address deserves more scrutiny than an ordinary login.

These flows are also common in APIs. Mobile apps, single-page applications, partner integrations, and checkout backends expose account actions through endpoints that may not share the same controls as the web login page. Attackers do not care whether the valuable step is behind /login, /api/account/email, or /checkout/payment. They follow the path that works.

Breached Credentials Are a Risk Signal

Breached credential checks are useful because they add context before the account is fully trusted. If a credential pair is known to have appeared in a breach, the site can treat the session differently from the start.

That does not mean every breached credential attempt should be handled the same way. A user may be genuinely logging in with a reused password. An attacker may be testing a combo list. A customer may be returning after a long period away. The point is to make the risk visible and carry it through the session.

Peakhour has written before about managing breached credential usage. The practical lesson is that credential risk should feed the wider account protection decision. A breached credential signal should be considered alongside client evidence, network context, request rate, route sensitivity, behaviour, and the action being attempted.

A low-risk page view and a stored-card checkout should not inherit the same confidence just because both follow a successful login.

MFA Helps, But It Does Not Close Every Path

Multi-factor authentication remains useful. It can stop many direct account takeover attempts and raise the cost of abuse. But MFA is not a complete defence when attackers use social engineering, session theft, weak recovery flows, trusted devices, or post-login actions that do not require step-up verification.

The better pattern is adaptive control. Let the login succeed when the evidence is low risk. Step up when the action matters. Ask for stronger verification before changing the email address, adding a new payout method, redeeming a stored balance, or checking out with saved payment details from an unfamiliar context.

This is not about adding friction everywhere. It is about reserving friction for the points where compromise turns into loss.

What to Monitor After Login

The useful signals are operational and specific:

Credential risk at login, including known breached username and password pairs.
New client, browser, or network evidence on an existing account.
Password reset, password change, email change, and phone change attempts.
New shipping addresses, payment method changes, stored-card use, gift card redemption, and loyalty balance activity.
Sudden changes in behaviour, such as rapid checkout after login or repeated account recovery attempts.
API routes that perform sensitive account actions without the same scrutiny as browser flows.

The response should match the confidence and consequence. Some events only need logging. Some need tighter rate limits. Some need a browser challenge, MFA step-up, temporary hold, customer notification, or review.

Credential stuffing defence is not finished when a password works or fails. The more useful question is: what does this session try to do next, and does the evidence justify trusting it?

For account protection, that is the line that matters.

Fingerprints Are Evidence, Not Identity

2026-06-19T00:00:00+10:00

The word "fingerprint" can create the wrong expectation.

In security, a browser or network fingerprint is not the same as a human fingerprint. It does not prove who a person is. It does not remove uncertainty. It should not be treated as a permanent identity for a customer.

A fingerprint is evidence. Sometimes it is strong evidence. Sometimes it is weak, common, stale, or deliberately manipulated. Its value comes from how it is combined with route, behaviour, account state, network context, credential risk, and the action being requested.

That distinction is more than wording. It affects how security teams design controls, explain decisions, and avoid overblocking legitimate users.

What Network Fingerprints Can Tell You

Network fingerprinting compares connection and protocol evidence. TCP behaviour, TLS handshakes, JA3 or JA4-style representations, HTTP/2 settings, header shape, MTU, proxy indicators, ASN, and path characteristics can all help classify the client or infrastructure behind a request.

That can be useful during credential stuffing, scraping, scanning, API abuse, or Layer 7 attack pressure. Attackers may rotate IP addresses, but parts of the client stack or automation framework can remain consistent. Grouping requests by network evidence can make rate limiting, bot detection, and investigation more precise than IP-only rules.

But the fingerprint is still not identity.

Common browsers can share similar network shapes. Mobile networks and carrier-grade NAT can make unrelated users appear close together. VPNs and residential proxies can distort source context. Browser and library updates can change fingerprints overnight. Hashing can make signals portable while hiding useful detail. Attack tools can also try to imitate normal clients.

The right conclusion from a suspicious network fingerprint is not "we know who this is". It is "this request deserves a different level of confidence".

What Browser Fingerprints Can Add

Browser fingerprinting adds evidence from the application layer and, where appropriate, browser-side checks. Headers, client hints, JavaScript-visible properties, rendering behaviour, storage behaviour, timezone, language, permissions, and API availability can help decide whether a request looks like the browser it claims to be.

This matters because many attacks try to borrow the appearance of ordinary browser traffic. Automation frameworks, emulators, headless browsers, anti-detect browsers, and scripted API clients can all present a user-agent string that looks plausible. Browser evidence helps compare the claim with the rest of the request.

Again, the useful output is confidence, not identity. A browser fingerprint might support a challenge. It might support a lower rate limit. It might explain why a session changing an email address needs step-up verification. It might help bot management separate obvious automation from normal traffic.

It should not become a claim that one technical pattern equals one person.

The Comparison Matters

Peakhour's page on browser fingerprinting vs network fingerprinting makes the operational split clear. Network fingerprints usually come from passive connection and protocol evidence. Browser fingerprints often involve request and browser-side evidence. They answer related but different questions.

A strong decision often needs both.

A request claiming to be a normal browser should look broadly consistent across TLS, HTTP/2, headers, JavaScript-visible browser properties, proxy context, route behaviour, and account history. If the browser looks normal but the network path resembles a known automation cluster, that is useful. If the network path looks ordinary but the browser evidence is inconsistent or missing on a sensitive route, that is useful too.

The mismatch is the signal. The response still depends on consequence.

A suspicious request to a public asset route might only need logging. The same evidence on login, password reset, stored-card checkout, account email change, admin access, or an expensive API route may justify a challenge, tighter limit, temporary hold, or review.

How to Use Fingerprints Responsibly

Fingerprints work best when they are attached to an explainable decision. A security event should show the route, account or token context where relevant, source network evidence, browser evidence, policy action, response code, and review outcome. That gives operators a way to understand and correct decisions.

Peakhour's guide to network fingerprint signals and security decisions frames the choices properly: allow, log, challenge, rate limit, block, or review. A fingerprint should help choose among those actions. It should not replace judgement.

Responsible use also means accepting uncertainty. Fingerprints collide. They drift. They can be spoofed. Some privacy tools intentionally reduce or alter browser signals. Some legitimate users have unusual configurations. Some high-risk requests have only partial evidence.

That uncertainty does not make fingerprints useless. It means they should be one layer in a wider control set.

For account and API security, the practical question is not "can this fingerprint identify a person?" It is "does this evidence change the confidence we should place in this request?"

If the answer is yes, use it carefully. Increase scrutiny on sensitive actions. Reduce friction where evidence is clean. Preserve enough context for review. Avoid pretending that a technical fingerprint is a human identity.

That is the more accurate model, and it leads to better security decisions.

For a protocol-level example, see A network fingerprint is a cohort, not a client. It compares the information retained by JA3, JA4 and Cisco Mercury and explains why a matching TLS value can still cover many applications and users.

How Residential Proxies Changed API and Account Abuse

2026-06-19T00:00:00+10:00

Residential proxies have changed the shape of API and account abuse. The old picture was easier to reason about: too many failed logins from one IP, a known hosting provider range, an obvious bot user agent, or a burst that crossed a threshold quickly enough to trip a rule.

That still happens, but it is not the harder problem.

The harder problem is the attempt that arrives through ordinary consumer networks, spreads itself across many addresses, and behaves just slowly enough to avoid looking like an incident. One login attempt here. A password reset probe there. A token refresh pattern that is unusual only when it is seen beside the route, the client, the ASN, the credential history, and the account event.

That is why residential proxy detection should be treated as part of the account and API decision path, not as a standalone allow/block list.

The Account Workflow Is Now a Distributed Target

Attackers do not need to break the whole application at once. They can work through the account surface in pieces:

Login attempts against known usernames.
Password reset initiation and verification.
New account registration.
Token issue and refresh routes.
Payment, address, profile, and email changes.
Loyalty, wallet, checkout, or stored-value workflows.
API calls that reveal whether an account or credential is valid.

Each route may look acceptable in isolation. The risk appears when the pattern is joined together.

A residential proxy network helps the attacker keep that pattern quiet. Requests rotate through many residential-looking exits. IP-based rate limits see different sources. A reputation feed may not have labelled a fresh or private proxy network yet. Geo checks can look plausible enough. The traffic does not necessarily arrive as a clean burst.

This is where static thinking breaks down. If the only question is "is this IP bad?", the answer will often arrive too late or be too blunt to use safely.

Fresh and Private Proxy Networks Create a Timing Problem

Many teams think about proxy detection as a database problem: look up the IP, see whether it is a proxy, then block it. That works for some traffic, especially known data centre proxies and commodity infrastructure.

Residential proxy abuse is less tidy. Fresh networks can appear before public datasets have a confident label. Private networks may not show up in broad feeds at all. Some exit points are shared with legitimate users. Some sit behind carrier-grade NAT or normal household connections. Blocking the address outright can create customer pain, while allowing it without context leaves the account flow exposed.

This is the practical reason Peakhour talks about residential proxy use as a signal. The signal matters, but it has to sit beside IP intelligence, connection characteristics, client history, request behaviour, account state, and route sensitivity.

A residential proxy on a marketing page may only need logging. The same proxy signal on a login route with recent failures may justify a challenge. On a password reset or high-value account change, it may justify step-up authentication, throttling, or blocking depending on the rest of the evidence.

The control should match the risk of the action.

Low-and-Slow Behaviour Is Still Automation

Low-and-slow abuse is uncomfortable because it avoids the easy operational story. There is no dramatic spike. There may be no single IP worth banning. The application may not be overloaded. Support may only see a few confused users, a few locked accounts, or a gradual rise in reset attempts.

For API and account workflows, this is still automation. It just looks less like a flood and more like a background process.

Useful signals include:

Repeated failed authentication across a shared fingerprint or client pattern.
Many accounts touched by similar request timing.
Token or reset routes used out of sequence.
Browser characteristics that do not match the claimed client.
Residential proxy use on sensitive account routes.
Fresh IP or ASN patterns appearing around account events.
Similar request shapes distributed across unrelated accounts.

None of these signals has to prove abuse by itself. The point is to combine them early enough that the application does not have to make the decision alone.

Peakhour's view is that proxy detection belongs in the same operating model as bot management, rate limiting, account risk scoring, and event evidence. The useful question is not "can we block every residential proxy?" It is "what should this route do when proxy use appears with this account, this client, this credential pattern, and this recent behaviour?"

API Routes Need the Same Treatment as Browser Flows

A common gap is protecting the visible login page while leaving API routes with weaker controls. Browser-side checks can help on web flows, but many account actions now happen through mobile apps, single-page applications, partner integrations, and backend APIs.

Those routes still need context. They need request-level validation, route-aware thresholds, proxy and IP signals, token checks, and evidence that can be reviewed later. A login API, a reset API, and a profile-change API should not all receive the same action just because the source address has the same reputation.

This is also why rate limiting has to move beyond source IP. A rule can key on a token, header, fingerprint, account identifier, route, response code, or a combination of signals. That makes it possible to slow failed login behaviour without punishing every legitimate user behind the same network.

The background reading on proxy detection challenges and quantifying residential proxy risk covers the broader detection problem. For API and account teams, the immediate step is more operational: find the routes where a residential proxy signal should change the action.

The Right Outcome Is Controlled Friction

Residential proxy detection is not a magic verdict. It is a way to make the account decision more honest.

Some traffic should pass. Some should be logged. Some should be rate limited. Some should be challenged. Some should be blocked. The difference should come from route sensitivity, request context, and observed behaviour, not from a single IP label.

A practical policy might look like this:

Monitor proxy use across all account and API routes.
Apply tighter thresholds on login, reset, token, and account-change routes.
Combine proxy use with credential, client, rate, and behaviour signals.
Preserve decision records so security and support can explain what happened.
Move from monitor to enforce only after reviewing false positives and customer impact.

That model gives teams a way to respond to residential proxy abuse without turning every shared residential network into a casualty.

For a grounding definition, see What is Residential Proxy Detection?. For the product control, see Residential Proxy Detection.

The important shift is simple: residential proxies are not just a network category. In account and API protection, they are context for deciding how much trust a request deserves.

Shadow APIs Are Account-Abuse Paths

2026-06-19T00:00:00+10:00

A shadow API is not dangerous because it has a mysterious name. It is dangerous because it still accepts requests.

If an endpoint can reset a password, refresh a token, check an account, change a delivery address, apply a discount, validate a stored payment method, or expose customer data, it is part of the account-abuse surface. Whether it appears in the current OpenAPI file is secondary.

Attackers do not need your API catalogue to be tidy. They need one working route that your normal controls do not understand.

The Forgotten Paths Are Often Real Paths

Most organisations have more API surface than they think.

Mobile apps leave behind old versions. Partner integrations get built for a campaign and then stay online. Browser-backed APIs are treated as internal because they are called by the front end, even though anyone can inspect and replay the requests. Legacy account endpoints remain active because turning them off might break an unknown client.

None of this is unusual. It is how real systems evolve.

The risk appears when those routes keep accepting production traffic without the same security treatment as the visible application. A current login page may have bot detection, adaptive prompts, and tuned rate limits. An older mobile endpoint may only check whether the credentials are valid. A partner route may trust an API key that has not been rotated. A browser API may return more account state than the front end displays.

That gap is where account abuse gets practical.

Unknown Does Not Mean Unused

Security teams sometimes talk about discovery as if the main outcome is a cleaner inventory. Inventory matters, but the more useful question is: what can this route do?

A shadow API that serves public catalogue data has one risk profile. A shadow API that changes account details has another. A forgotten token endpoint is different again. A mobile route that accepts username and password combinations is a credential stuffing target, even if the public login page has already been hardened.

This is why API security has to stay connected to account context. Route discovery is only the start. The protection model needs to know method, schema, authentication state, response pattern, user journey, and business sensitivity.

A POST request to an account recovery endpoint deserves different treatment from a GET request to a static content API. A password reset route used by a first-seen client through rotating proxy infrastructure is not the same as the same route used by a known customer session.

The route matters because the account outcome matters.

Browser-Backed APIs Are Still APIs

A common blind spot is the API behind the web application.

The front end might make a neat request to /api/account/profile, /api/cart/apply-coupon, or /api/session/refresh. Because the route was designed for the browser, teams may assume the browser is the control. It is not.

Requests can be replayed outside the page. Headers can be copied. Tokens can be stolen. User agents can be faked. Automation can follow the same sequence as the application, only faster and at scale.

The right response is not to treat every browser-backed API as hostile. The right response is to attach evidence. Is this a known browser session? Is the TLS and HTTP behaviour consistent? Is the request sequence normal for the journey? Is the session suddenly moving from login to sensitive account changes? Does the schema match what the route expects?

Those questions sit between basic definitions of what API security is and the operational work of stopping abuse.

Mobile and Partner APIs Need Ownership

Mobile and partner APIs create a slightly different problem. They often have legitimate non-browser clients, so crude controls can break real use.

A mobile app may have older versions in the field. A partner may send traffic from fixed infrastructure, or from changing cloud infrastructure. A service client may authenticate with an API key rather than an interactive user session. Some requests will look less browser-like because they are not meant to be browsers.

That makes ownership important.

Each route should have an owner, expected clients, authentication model, rate expectation, schema expectation, and deprecation plan. API keys should be treated as credentials, not configuration strings. OAuth and JWT use should include short-lived access, appropriate scopes, and validation at the endpoint. Legacy flows should not survive indefinitely just because nobody is sure what they support.

For REST services, that discipline includes the basics covered in REST API security: method control, status-code handling, input validation, token handling, rate limiting, and useful logging. The account-abuse angle is narrower and more operational: which of these controls tells us whether this request can harm a customer account?

Discovery Has to Feed Enforcement

A report listing unknown endpoints is useful for a week. A discovery process that feeds policy is useful every day.

When a new route appears, the security question should be concrete:

Who owns it?
Is it authenticated?
Does it match a known schema?
Does it touch account state?
Can it reset trust, change value, or expose customer data?
What rate and behaviour patterns are normal?
Which action should apply when it is abused?

That is where API protection use cases become more than documentation. The goal is not to produce a perfect catalogue for its own sake. The goal is to reduce the number of unknown request paths that can be used for account abuse.

Shadow APIs are not a separate class of attack. They are normal APIs without enough operational visibility.

And when they sit on account journeys, they become a direct path from unknown surface to customer harm.

Price Transparency Is Now a Data Access Problem

2026-05-18T00:00:00+10:00

Australia's supermarket pricing debate can look like a consumer pricing story.

For digital teams, it is also a bot and API protection story.

The ACCC's final supermarket inquiry recommended that ALDI, Coles, and Woolworths publish prices on their websites. It also recommended that Coles and Woolworths make dynamic price APIs available for third-party comparison tools.

Then, on 18 May 2026, Guardian Australia reported on CW Scanner data about Coles and Woolworths promotion patterns. For digital teams, the operational detail is this: the report said CW Scanner's operator stated the service was not scraping, and instead used the supermarkets' website application programming interfaces.

That does not settle questions about permission, terms of use, supermarket approval, or the status of any specific API. It does make the practical problem sharper.

Price transparency increasingly depends on automated access to current data.

Every automated request still needs a decision.

APIs do not remove the bot problem

It is tempting to treat an API as the clean alternative to scraping. Sometimes it is cleaner. A documented API can make access more predictable, auditable, and easier to govern than repeated extraction from product pages.

But an API is still an automation surface.

The same retailer or marketplace may need to support:

public product pages;
price, promotion, search, listing, catalogue, and inventory routes;
browser-backed application calls;
documented APIs and partner feeds;
comparison tools and public-interest services;
search engines, monitoring systems, and accessibility tooling;
unknown collectors rebuilding price, inventory, availability, or ticketing datasets outside the intended access model.

Some of that traffic is useful. Some is commercially necessary. Some is abusive. Much of it will not identify itself honestly.

So the question is not "should bots be blocked?"

The question is: can you tell intended automated access from uncontrolled extraction?

The decision needs evidence

A blanket "block all automation" position can break comparison services, partner integrations, search visibility, monitoring, accessibility tooling, and APIs that were built to be automated.

A blanket "allow everything" position can expose pricing, product, inventory, account, checkout, ticketing, and API paths to extraction and abuse.

The practical middle ground is governed automation:

publish the access you want to support;
recognise the clients and behaviours you expect;
validate API route, schema, method, authentication, and client context;
detect traffic that has drifted from the intended use;
keep decision logs so security, legal, product, and commercial teams can review what happened;
respond proportionately with allow, log, rate-limit, challenge, block, or review decisions.

That is where bot protection becomes more than a yes-or-no control.

It supplies the evidence behind each access decision.

Where Peakhour fits

Scraping protection should identify repeated extraction across product, price, search, listing, catalogue, article, inventory, and availability routes. The goal is not to stop every non-human request. It is to separate expected access from collectors rebuilding data outside the site's control.

API bot protection applies the same discipline to automated API clients. APIs exist to be automated. The risk comes from unknown clients, unexpected route combinations, credential abuse, endpoint enumeration, excessive request volume, and business-logic abuse that generic perimeter controls cannot explain.

Bot management turns request evidence into a decision: allow trusted traffic, log expected automated access, rate-limit noisy collectors, challenge uncertain sessions, block confirmed abuse, or send edge cases to review.

Verified browser trust adds a useful signal when browser-backed journeys are being automated or replayed. Headers and cookies can be copied, proxy networks can rotate, and automation can mimic ordinary navigation. Peakhour can challenge the browser path, verify that the expected evidence came back, and attach that witness to the wider decision record.

That browser signal does not, by itself, prove the user, device, or account is trustworthy. It helps the risk engine decide what to do alongside route, proxy, device, behaviour, credential, and API context.

Why this matters beyond supermarkets

The ACCC's 2024 proceedings against Coles and Woolworths were about alleged false or misleading price statements, not supermarket price regulation, collusion, or anti-competitive conduct. The ACCC announced on 14 May 2026 that the Federal Court found Coles made false or misleading representations in 13 of 14 agreed sample "Down Down" tickets, with penalties and other orders still to be determined. For Woolworths, the separate "Prices Dropped" proceeding was awaiting judgment at publication.

Those legal details matter, but they are not the Peakhour point.

The Peakhour point is operational: when transparency, comparison, availability, or fairness depends on current digital data, organisations need a control plane that can support the access they intend and limit the extraction they do not.

That pattern shows up in retail, marketplaces, ticketing, travel, financial services, media, and any platform where public pages, browser-backed calls, and APIs expose commercially valuable data. It also shows up in adjacent problems like account abuse, checkout abuse, ticket scalping, product scraping, distorted analytics, and inventory harvesting.

The organisations that handle this well will not treat every automated request as the same.

They will be the ones that know what access they intend to allow, what behaviour they intend to stop, and why.

The BOTS Act and the War on Ticket Scalping

2025-09-17T00:00:00+10:00

For popular concerts and events, tickets can disappear in seconds. Often the competition is not just other fans, but ticket scalping bots built to buy faster than a human can, then push those tickets into secondary markets at heavily inflated prices. The United States introduced the Better Online Ticket Sales (BOTS) Act to address that conduct. Whether it is working is less clear. Recent investigations into Ticketmaster suggest enforcement is still catching up.

Understanding the BOTS Act

Enacted in 2016, the Better Online Ticket Sales (BOTS) Act is a U.S. federal law aimed at giving consumers a fairer chance when buying tickets online. The law makes it illegal to use software, or "bots," to bypass security measures on ticket-selling websites.

Specifically, the BOTS Act prohibits:

Circumventing a security measure, access control system, or other technological control used by a ticket issuer to enforce purchasing limits.
Using a bot to purchase tickets in excess of posted limits.
Selling tickets that were acquired in violation of these rules, if the seller knew or should have known they were obtained illegally.

The aim was to stop scalpers using automated tools to buy tickets in bulk, so fans had a fairer chance of purchasing tickets at face value.

Australia's Approach

Ticket scalping bots are not only a U.S. problem. Other countries have also used legislation to protect consumers. In Australia, the state of New South Wales (NSW) became the first to outlaw ticket bots in 2017.

The legislation introduced heavy penalties, with fines up to $110,000 for corporations using bots to snap up tickets. The law also went further than the US BOTS Act by capping ticket resale prices at no more than 10% above the original price. At the time, a representative from TEG, the owner of Australia's largest ticket seller Ticketek, stated that bots accounted for up to 70% of activity on their website, showing how widespread the problem had become.

Ticketmaster Under the Microscope

Despite the BOTS Act being in place for years, ticket scalping persists. U.S. regulators have now turned their attention to the platforms themselves. In September 2025, reports emerged that the U.S. Federal Trade Commission (FTC) was investigating whether Ticketmaster, a subsidiary of Live Nation, was doing enough to comply with the BOTS Act.

The probe is reportedly examining whether the ticket seller has a financial incentive to allow resellers to circumvent its own rules. Ticketmaster denies wrongdoing and claims to block hundreds of millions of bots daily, but the investigation raises a direct question: are ticket platforms motivated to eliminate a secondary market from which they can also profit?

If the FTC pursues a case and finds Ticketmaster in violation, the penalties could be substantial. The law allows for fines of up to $53,000 per violation, which could translate to billions of dollars in penalties and put the rest of the industry on notice.

Renewed Political Pressure

Ticket scalping has also drawn direct political attention. In March 2025, President Donald Trump signed an executive order aimed at cracking down on ticket scalpers and price-gouging practices. The order specifically calls on the FTC to increase its enforcement of the BOTS Act, a piece of legislation that, despite being law since 2016, has been used to prosecute offenders only once.

The executive order also pushes for greater price transparency throughout the ticket-buying process and calls for investigations into unfair and anti-competitive practices in the secondary market. This follows similar efforts from the Biden administration, which had focused on eliminating "junk fees" and promoting all-in pricing. The bipartisan pressure points to a growing consensus that the current state of the ticketing industry is failing consumers.

The order received broad support from across the industry, with Live Nation, StubHub, and the National Independent Venue Association (NIVA) all voicing their approval. That support reflects a clear industry position: stronger enforcement of existing laws like the BOTS Act is a necessary step in protecting fans from scalpers.

Fighting on Two Fronts

While regulators are questioning Ticketmaster's efforts, the company has also used legal channels to fight bot operators. A notable case involved Prestige Entertainment, a ticket broker that Ticketmaster accused of using sophisticated bots to buy large quantities of tickets for resale.

In that lawsuit, Ticketmaster alleged that Prestige used bots and dummy accounts to bypass security measures like CAPTCHAs and purchase limits. The case produced some interesting legal arguments, including a novel copyright claim. Ticketmaster argued that in order to develop the bots, the software creators must have illegally copied code from Ticketmaster's website and app.

The legal battle ended in a settlement in 2019, with Prestige Entertainment permanently banned from using bots on Ticketmaster's platforms. The case shows that while Ticketmaster is under scrutiny itself, it has also pursued those who exploit its system. That leaves Ticketmaster fighting on two fronts: against bot operators, and now against regulators questioning whether its own enforcement is strong enough.

An Endless Cat and Mouse Game

The ticketing industry is locked in a constant technical arms race. As platforms like Ticketmaster develop more sophisticated security measures, bot creators find new ways to circumvent them. They use tools like residential proxies to mimic human users and distribute their attacks across thousands of IP addresses, making them difficult to detect.

Ticketmaster states it blocks 200 million bots a day, a fivefold increase from 2019. That figure shows the scale of the problem. The financial incentive for scalpers is large, and they are willing to invest heavily in technology to stay ahead.

For consumers, this means sold-out events and high resale prices are likely to continue. The BOTS Act provides a legal framework to punish offenders, but its effectiveness depends on enforcement and cooperation from major players like Ticketmaster. The current FTC investigation may show how much practical force the law has.

Agentic AI vs. Your API

2025-09-01T00:00:00+10:00

For years, "bots" mostly meant simple, scripted programs. They followed rigid, predefined rules: if you see X, do Y. They were predictable. They could still do damage in attacks like credential stuffing, but their lack of intelligence made them relatively easy to detect. Their patterns were repetitive and clearly different from the complex, often messy, behaviour of human users.

That model is no longer reliable. The emergence of open and powerful reasoning models like DeepSeek has given rise to a new class of automation: agentic AI. These are not just scripts. They are autonomous agents that can reason, plan, and adapt their behaviour in real time. They don't need a human to write a script for every possibility. Give them a goal and they can work out the steps themselves. That changes the nature of automated threats, and security controls need to change with it.

The New API Consumer

Historically, APIs were consumed by two main groups: human users via a front-end application, and scripted bots following predictable patterns. Agentic AI introduces a third consumer, and one likely to become dominant. These AI agents are becoming primary users of web APIs, and they interact with them in materially different ways.

An AI agent can analyse an entire API surface in seconds, understand the relationships between different endpoints, and generate complex interaction patterns that a human developer would rarely attempt. They don't just follow a linear path; they can explore, learn, and optimise their interactions to achieve their goals, whether that's finding the best price on a product, gathering data, or probing for security weaknesses.

New Security Challenges: The Self-Hacking AI

The reasoning capabilities of these agents introduce security challenges that static, rule-based systems are poorly equipped to handle. An agentic AI doesn't just throw known exploits at a system; it can probe its defences and invent new attacks as it goes.

Consider a traditional Web Application Firewall (WAF) that relies on pattern-matching rules to block threats like SQL injection. An AI agent can send a series of carefully crafted requests, observe the WAF's responses, and systematically learn the structure of its rules. Once it understands the patterns the WAF is looking for, it can generate a custom exploit designed to bypass those rules while still achieving its malicious objective.

This isn't theoretical. Security teams are already reporting sophisticated attacks that adapt in real time, adjusting their tactics based on the system's defensive responses. These aren't simply pre-programmed behaviours; they are reasoning models at work.

A New Security Paradigm: From "Block Bots" to "Manage Agents"

The rise of agentic AI changes the security question. The old goal of "blocking all bots" is no longer viable or even desirable. AI agents will be used for both benign and malicious purposes. A customer's personal AI assistant booking a flight is useful automation; an attacker's AI agent trying to find vulnerabilities is not.

Bot management cannot stop at trying to keep automation out. It needs the intelligence to safely identify and manage AI agents. This requires moving away from static, signature-based detection and toward a more contextual, behavioural approach.

The key questions will no longer be "Is this a human or a bot?" but rather:

"What is the intent of this automated agent?"
"Is its behaviour consistent with a legitimate use case?"
"Can we trust this agent?"

This requires a new generation of security tools that can understand and adapt to agent behaviour, distinguishing between the legitimate AI assistants that will soon be a core part of our digital lives and the malicious ones that seek to exploit our systems. Organisations that fail to prepare for this shift risk having their defences systematically tested, mapped, and bypassed by the next wave of intelligent, automated threats.

Beyond the IP Address

2025-09-01T00:00:00+10:00

For years, rate limiting has been a standard control for protecting websites and APIs from abuse. The basic model is simple: limit the number of requests a single "user" can make in a given period. If a user exceeds the limit (e.g., 10 login attempts in a minute), they are temporarily blocked.

The hard part has always been identifying that "user". Traditionally, the answer was the IP address. The assumption was that one IP address equaled one user. In the early days of the internet, this was a reasonable approximation. Today, that assumption no longer holds, and it leaves systems exposed to modern attacks.

The IP address is no longer a reliable identifier for a single user or device. There are three common reasons:

Proxy Networks: Attackers don't use a single IP address. They use large residential proxy networks to rotate requests through thousands or even millions of different IP addresses, making each request look like it comes from a new user.
Shared IPs (CGNAT): At the same time, a single IP address can represent thousands of legitimate users. Mobile carriers use Carrier-Grade NAT (CGNAT) to make many mobile devices share the same public IP. Similarly, an entire office building or university campus might appear to the internet as a single IP.
Distributed Attacks: Modern automated attacks, like Layer 7 DDoS or credential stuffing, are inherently distributed. Attackers use botnets or proxy networks to spread their attack across a large number of IPs, so no single IP ever exceeds a traditional rate limit.

Blocking a shared IP because of one bad actor can cause collateral damage, denying access to thousands of legitimate users. On the other side, failing to see that thousands of IPs are part of a single coordinated attack means the attack succeeds. Traditional IP-based rate limiting is no longer enough.

The New Way: Advanced Rate Limiting

Advanced Rate Limiting addresses this by moving beyond the IP address. Instead of grouping requests by a single, unreliable identifier, it lets you count requests using more stable and meaningful characteristics of the connection or the software making it.

This approach groups requests using identifiers like:

TLS/HTTP2 Fingerprints: Every client application (like a browser or a script) has a unique "fingerprint" based on how it initiates a secure connection (TLS) or communicates over HTTP/2. This fingerprint remains consistent even as an attacker rotates through thousands of IP addresses. By rate limiting based on the TLS fingerprint, you can track and block the underlying automation tool itself, not just the IPs it uses.
Device Characteristics: A fingerprint can be constructed from a range of attributes, including the device's operating system, browser version, and more. This allows for the detection of repeated requests coming from the same class of device.
A Combination of Headers: For authenticated APIs, you can group requests by an Authorization header or API key, enforcing fair usage and preventing abuse by a single authenticated client.

Practical Use Cases

The value of advanced rate limiting is clearest when it is applied to real-world threats:

Mitigating Distributed Credential Stuffing: An attacker using a tool like OpenBullet launches a credential stuffing attack against your login page, rotating through thousands of residential proxy IPs. Traditional rate limiting is ineffective here. However, the OpenBullet software has a consistent TLS fingerprint. By setting a rule to limit failed login attempts per TLS fingerprint, you can detect and block the entire distributed attack, regardless of how many IPs are involved.
Protecting APIs from Abuse: A partner is abusing their API key, sending far too many requests and degrading service for other users. By rate limiting based on the Authorization header, you can enforce usage limits on a per-client basis, keeping access fair without affecting other users.
Stopping Content Scrapers: A scraper is hammering your e-commerce site to steal pricing data. They are using a botnet to distribute the requests across hundreds of IPs. However, the scraping script has a unique combination of a user-agent and a TLS fingerprint. Advanced rate limiting can count requests based on this combined signature and block the scraper, protecting your intellectual property.

When attackers are distributed, your defences need to see the single actor behind the many IPs. Advanced rate limiting provides that visibility and should be part of a modern application security strategy.

The Invisibility Cloak

2025-09-01T00:00:00+10:00

Every time you connect to a website, you leave behind a "digital fingerprint." This is not a physical fingerprint, but a set of signals from your device and browser. Security tools analyse this fingerprint—which includes your IP address, browser type, operating system, supported fonts, and even subtle characteristics of your network connection (TLS fingerprinting)—to distinguish legitimate users from malicious bots.

For years, this was a reliable way to spot automated threats. Bots often had clumsy, inconsistent fingerprints that made them easier to identify. Today, attackers can combine tools that mimic real users closely enough to weaken many traditional defences. The two most important components of this modern "invisibility cloak" are residential proxies and anti-detect browsers.

What Are Residential Proxies?

A residential proxy is an intermediary server that uses an IP address assigned by an Internet Service Provider (ISP) to a real home internet connection. When a bot routes its traffic through a residential proxy, its requests appear to originate from a genuine home user, not a data centre.

These proxy networks are large, often containing millions of IP addresses sourced from around the globe. How are these IPs obtained? Often through questionable means:

Malware and Botnets: Unsuspecting users' devices are infected with malware that turns them into proxy endpoints.
SDKs in Free Apps: Some free applications (often VPNs or mobile apps) include code that enrols the user's device into a proxy network in exchange for using the app, often without the user's full knowledge or consent.

By rotating through this large pool of legitimate-looking IPs, attackers can launch large-scale attacks that are difficult to separate from normal traffic. To a website's security system, a distributed attack from a residential proxy network looks like thousands of individual customers from different locations.

What Are Anti-Detect Browsers?

While residential proxies mask the attacker's network location, anti-detect browsers are designed to spoof the rest of the digital fingerprint. These specialised browsers allow an attacker to create and manage thousands of unique browser profiles, each with a customised and consistent fingerprint.

An anti-detect browser can control and randomise every detail a website uses for identification, including:

Browser type and version (e.g., Chrome, Firefox, Safari)
Operating system (Windows, macOS, iOS, Android)
Screen resolution, fonts, and plugins
Time zone and language settings
Subtle browser characteristics like Canvas and WebGL rendering

With a few clicks, an attacker can make a single machine in one country appear as thousands of unique users on different devices and operating systems from all over the world.

The Combined Threat: A Perfect Storm for Attacks

When attackers combine residential proxies with anti-detect browsers, they cover both the network and browser layers that many controls rely on. The residential proxy provides a legitimate IP address, and the anti-detect browser provides a consistent, human-looking browser fingerprint.

This combination makes attacks like large-scale credential stuffing, content scraping, and inventory scalping much harder to distinguish from legitimate user traffic. Each malicious request appears to be from a unique person on a standard device, using a normal home internet connection.

Why Traditional Defenses Fail and What to Do About It

This level of sophistication weakens traditional security measures:

IP Blocklists and Reputation Services: These struggle when attackers are using a constantly rotating pool of millions of legitimate residential IP addresses. Our own research shows that even the best IP intelligence services fail to detect the vast majority of residential proxy traffic.
Basic Browser Fingerprinting: Anti-detect browsers are specifically designed to defeat these checks by providing a consistent and realistic fingerprint.

To combat this combined threat, organisations need a modern approach to bot detection that looks beyond the surface:

Advanced Network Fingerprinting: Instead of just looking at the IP address, modern solutions analyse the underlying characteristics of the network connection itself (like the TLS/JA3 fingerprint). These signatures can often identify the underlying automation tool or proxy network, even when the IP address appears legitimate.
Behavioural Analysis: Advanced systems model normal user behaviour—such as mouse movements, typing speed, and page navigation—to identify the subtle, non-human patterns of automation that even sophisticated bots can't perfectly mimic.
Hardware and Rendering Fingerprinting: While anti-detect browsers can spoof software-level details, faking the underlying hardware is far more difficult. Advanced techniques, such as those used in Google's Picasso, analyse how a device renders graphics (e.g., Canvas and WebGL), processes audio, and performs CPU-intensive tasks. This creates a hardware fingerprint based on the unique characteristics of the GPU, audio stack, and CPU clock speed. This fingerprint can reveal inconsistencies between the claimed browser profile and the actual hardware being used. When combined with network fingerprinting and residential proxy detection, this becomes a strong signal for identifying a single machine attempting to impersonate many different users.
Dedicated Residential Proxy Detection: Specialised techniques are required to identify traffic coming from residential proxy networks. This is a critical signal, as very few legitimate users have a reason to route their traffic this way.

Attackers using residential proxies and anti-detect browsers are harder to identify, but they still leave signals. Network characteristics, hardware fingerprints, and the behavioural tells of automation give security teams a better chance of separating the bot from the user it is trying to resemble.

The CAPTCHA Conundrum

2025-09-01T00:00:00+10:00

Remember the distorted, wavy letters you had to decipher to prove you were not a robot? That was CAPTCHA, which stands for "Completely Automated Public Turing test to tell Computers and Humans Apart." First developed in the early 2000s, its purpose was simple: create a challenge that humans could solve, but automated scripts, or bots, would struggle with.

For years, CAPTCHA was used across common web actions, from creating an account to posting a comment. But bots have improved faster than the test, while users have become less tolerant of friction. CAPTCHA now often fails in both directions: it interrupts legitimate users and can still be solved by automated campaigns.

The User Experience Problem: Killing Conversions

The biggest issue with visible CAPTCHAs is the friction they add to the user journey. Customers expect checkout, login, and form flows to be quick. Asking them to stop and solve a puzzle creates a clear point for abandonment.

The data is hard to ignore:

A landmark Stanford University study found that adding a CAPTCHA can reduce form conversions by up to 40%.
Research from bot management firm HUMAN Security revealed that 40% of real shoppers have abandoned a purchase because of CAPTCHA frustration.
Other analyses have shown that simply adding a CAPTCHA can lead to a 3.2% higher bounce rate and an overall 3-5% drop in conversions.

For an e-commerce business, losing up to 40% of potential sales at the final checkout step is not a marginal UX issue. These numbers map directly to revenue from legitimate customers who were annoyed, delayed, or unable to solve the puzzle. The impact is even worse for users with disabilities, for whom many visual CAPTCHAs are nearly impossible to complete.

The Security Problem: A Speed Bump for Bots

While CAPTCHAs frustrate legitimate users, they are often only a small obstacle for modern bots. A market exists for bypassing them at scale.

Attackers now use automated CAPTCHA-solving services, often called "CAPTCHA farms." These services use a combination of machine learning algorithms and low-wage human workers to solve CAPTCHAs in real time for a fraction of a cent per puzzle.

An attacker using an automation tool like OpenBullet can integrate with these services via a simple API call. When the bot encounters a CAPTCHA, it sends the puzzle to the solving service and receives the solution seconds later. In many cases, these services have a higher success rate at solving CAPTCHAs than actual humans.

That breaks the original CAPTCHA model. The test designed to block bots can now create a false sense of security while actively harming the experience for real users.

The Modern Alternative: Invisible Challenges

If visible CAPTCHAs are broken, what's the alternative? Modern bot management uses invisible challenges to verify users without causing friction. Instead of actively testing the user, these systems analyse background data to distinguish humans from bots.

This is achieved through a multi-layered approach:

Behavioural Analysis: These systems track subtle indicators of human behaviour, like mouse movements, typing cadence, and touchscreen interactions. Bots, even sophisticated ones, struggle to mimic these patterns consistently.
Network and Browser Fingerprinting: By analysing hundreds of data points from the browser and network connection, these systems can identify the tell-tale signs of automation, such as the use of data centre IPs, proxy networks, or inconsistencies in the browser fingerprint.
Machine Learning: Machine learning models are trained on vast datasets of human and bot traffic. They can identify complex patterns and adapt in real time to new and evolving bot techniques.

With this approach, most legitimate users never see a challenge at all. Their journey remains uninterrupted. Only when the system detects highly suspicious activity is a challenge presented, so security controls are applied where the risk justifies the friction. For sites that depend on checkout completion, account protection, or lead capture, that is a better trade-off than showing every user another puzzle.

Key Considerations for Effective Bot Management

2025-09-01T00:00:00+10:00

Introduction

Bots account for a large share of web traffic. Recent studies estimate that nearly 50% of all internet traffic is generated by automated programs. Some bots are necessary for the web to function, such as search engine crawlers, but a significant portion are malicious. These "bad bots" are used for content scraping, credential stuffing, spam, and DDoS attacks.

As bot operators become more sophisticated, bot management needs to cover detection, classification, and response. This article outlines the main considerations for security teams protecting intellectual property, online revenue, and user accounts.

The Goal: Accurate Bot Detection and Classification

The first step in effective bot management is separating legitimate users from automated threats. Identification is not enough on its own. Security teams also need accurate classification across good, bad, and "grey" bots.

Good Bots: Support normal internet operations, such as search engine crawlers (Googlebot, Bingbot) and performance monitoring bots.
Bad Bots: Carry out malicious activity such as content scraping, account takeover, and spamming.
Grey Bots: Serve a legitimate purpose but can cause problems when they crawl too aggressively, such as SEO and marketing bots (Ahrefs, SEMrush).

Effective detection usually needs more than basic signatures. A layered approach commonly includes:

Basic Protection: Targets simple bots using user agent checks and IP reputation databases.
Intermediate Protection: Uses JavaScript-based challenges and basic network fingerprinting, such as JA3/JA4, to detect less sophisticated bots.
Advanced Protection: Combines comprehensive network fingerprinting, behavioural analysis, and machine learning to identify sophisticated bots that mimic human behaviour, use residential proxies, or rely on anti-detect browsers.

Machine learning models help in this context because they can learn from changing bot strategies and inspect incoming traffic for subtle signs of automation.

The Method: Continuously Adaptive Detection and Response

Bot behaviour changes quickly. Threat actors modify tooling, traffic patterns, and infrastructure to avoid detection, so static defence rules degrade over time. Organisations need detection and response that can adapt as the attack changes.

That means correlating metadata with behavioural factors in real time, then applying the right response for the risk. When a bot attempts account takeover or data scraping, an adaptive response can act immediately to reduce the impact.

Effective adaptive responses include:

Advanced Rate Limiting: Goes beyond simple IP-based limits by grouping requests with more stable identifiers, such as TLS/HTTP2 fingerprints or device characteristics. This helps stop distributed attacks from tools like OpenBullet that rotate through thousands of IP addresses.
Web Application Firewalls (WAF): Provide an important first line of defence by filtering harmful Layer 7 traffic based on predefined rules.
Tarpitting: Slows malicious connections to increase cost and resource consumption for attackers.
Challenges: Traditional visible CAPTCHAs can harm user experience and are often solvable by modern bots. Invisible challenges can verify a legitimate browser environment with less friction.
Alternate Content Serving: Misleads scraping bots by serving alternate or cached content with incorrect information (e.g., higher prices), making their scraped data useless.

The same response process should also feed learning loops, building a repository of bot attack patterns that can train machine learning models and improve accuracy over time.

The Expected Outcomes: A Resilient Security Posture

An adaptive bot management strategy should support several practical outcomes:

Risk Mitigation: Reduce potential financial losses, service disruption, and data breaches associated with malicious bot activity such as credential stuffing, ad fraud, and inventory hoarding.
Improved User Experience: Keep disruption low for genuine users by using invisible challenges and behavioural analysis instead of frustrating CAPTCHAs, which can reduce conversions by up to 40%.
Intellectual Property Protection: Protect valuable content, pricing data, and other intellectual property from unauthorised scraping.
Online Revenue Security: Protect online revenue streams by preventing fraud, inventory scalping, and other malicious activity that targets e-commerce platforms.
Regulatory Compliance: Help organisations meet data protection and privacy regulations with a proactive bot management approach.

Conclusion: Fortifying Against Sophisticated Bots

Modern bot defence depends on accurate detection, precise classification, and adaptive response. Machine learning, comprehensive network fingerprinting, and behavioural analysis all contribute, but they work best as part of a layered control set.

With that approach, security teams can better protect intellectual property, online revenue, and user accounts from sophisticated bot activity.

The Bot Spectrum

2025-09-01T00:00:00+10:00

The word "bot" is often used as shorthand for unwanted automation: scripts trying to break into accounts, scrape content, or overwhelm websites. A large share of internet traffic does come from bad bots, but automated traffic is not automatically harmful. Some bots are part of how the web is discovered, monitored, and kept usable.

Effective bot management is not about blocking every automated request. It depends on accurate classification: separating good bots from bad bots, and recognising the "grey" bots that sit between them. That classification lets you apply controls that reduce risk without cutting off traffic that helps your site operate.

Good Bots: The Essential Workers of the Web

Good bots are automated programs that perform useful or necessary tasks. They are usually clear about who they are and respect the rules you set in your robots.txt file. Blocking them can damage search visibility, monitoring, or other business workflows.

Examples of Good Bots:

Search Engine Crawlers: Bots like Googlebot and Bingbot are the best-known good bots. They crawl and index your website's content, which is how your pages appear in search engine results. Blocking them would make your site invisible on Google.
Performance Monitoring Bots: These bots are used by services to check your website's uptime and performance from different locations around the world, and to alert you if your site goes down.
Copyright Bots: These bots scan the web for plagiarised content, helping to protect your intellectual property.

Management Strategy: Good bots should be identified and allowed to access your site freely. Verification techniques, such as reverse DNS lookups, can be used to confirm that a bot claiming to be Googlebot is actually coming from Google.

Bad Bots: The Malicious Actors

Bad bots are designed for malicious activity. They are a major reason bot management exists as a security function. These bots are deceptive, often hiding their identity and purpose, and they can be responsible for a wide range of costly and damaging activity.

Examples of Bad Bots:

Credential Stuffers: These bots use stolen usernames and passwords to carry out account takeover attacks.
Content and Price Scrapers: These bots steal your valuable content, product listings, and pricing data, often for use by competitors.
Spam Bots: These bots flood comment sections, forums, and contact forms with unwanted ads or malicious links.
Denial of Service (DDoS) Bots: These bots are part of a botnet used to overwhelm a website with traffic, causing it to slow down or crash.
Inventory Hoarding Bots: Common in e-commerce, these bots automatically add limited-edition products to shopping carts to prevent legitimate customers from buying them, often for resale at a higher price (scalping).

Management Strategy: Bad bots need to be accurately identified and blocked as quickly as possible, ideally at the network edge before they consume your server resources.

Grey Bots: The Nuanced Category

Grey bots are not inherently malicious, but their behaviour can still cause problems. They often serve a legitimate purpose, but become an issue when they crawl too aggressively, consume excessive bandwidth or server resources, and slow the site down for real users.

Examples of Grey Bots:

Aggressive SEO Tools: Bots from marketing tools like Ahrefs, SEMrush, and Majestic crawl websites to gather data for backlink analysis and competitive research. They can be useful, but their crawling can also be heavy.
Partner and Aggregator Bots: These could be bots from partner companies or price comparison websites that need to access your data. The activity may be legitimate, but it still needs to be managed.
Feed Fetchers: Bots that collect data for news aggregators or other applications fall into this category.

Management Strategy: Grey bots require more than a simple allow or block rule. The best strategy is often to rate-limit or tarpit them.

Rate-Limiting: This allows the bot to continue accessing your site, but slows it to a manageable level so it does not overwhelm your servers.
Tarpitting: This intentionally slows the connection for a specific bot, increasing the cost and time required to crawl your site and discouraging overly aggressive behaviour.

By classifying incoming bot traffic and applying the right control for each category, organisations can block threats, manage resource consumption, and allow the useful automation the modern web depends on.

How to Use Bot Management for IAM Use Cases

2025-08-20T00:00:00+10:00

Automated attacks against identity and access management (IAM) systems are now a routine account protection problem. Malicious bots drive account takeovers (ATO), credential stuffing, brute-force login attempts, and fake account creation. As these attacks adapt, traditional IAM controls such as password policies and even multi-factor authentication (MFA) are not enough on their own.

Identity and access management leaders should treat bot management as part of the IAM control set, not a separate website security add-on. A dedicated capability helps reduce avoidable financial and reputational losses from account compromise. It also gives organisations a way to manage the risks created as AI agents become regular users of web applications and APIs.

Introduction

Some estimates suggest nearly half of all traffic is automated. That mix matters: useful crawlers and monitoring tools are part of normal internet traffic, but malicious automation is built to test web applications at scale. IAM systems, which control access to sensitive user accounts and data, are a primary target.

The most common bot attacks targeting IAM include:

Credential Stuffing: Attackers use lists of stolen usernames and passwords from third-party data breaches to gain unauthorised access to user accounts. This attack vector is effective because password reuse is still common.
Brute-Force Attacks: Automated scripts guess passwords for known usernames, often targeting login endpoints for platforms like WordPress and Magento.
Fake Account Creation: Bots create fraudulent accounts at scale, which can be used for spam, malware distribution, or to abuse promotional offers.

Recent attacks on major Australian retailers like The Iconic and Dan Murphy's show the practical impact. These incidents, driven by credential stuffing, resulted in reputational damage and financial loss, forcing the companies to issue refunds and publicly address security concerns.

Analysis

Defending IAM systems starts with why common controls fall short and where bot management adds useful signal.

Why Traditional IAM Defences Fail

Attackers have adapted their techniques to bypass legacy security controls. Simple IP-based rate limiting and reputation lists struggle against the combination of residential proxies and anti-detect browsers:

Residential Proxies: Attackers route their traffic through large networks of IP addresses belonging to real residential internet connections. This makes malicious traffic appear legitimate and allows attackers to bypass IP-based blocking and geolocation restrictions. Our own tests show that even leading IP intelligence services fail to detect the vast majority of residential proxy traffic.
Anti-Detect Browsers: These specialised browsers allow attackers to spoof their digital fingerprints, mimicking legitimate user devices and browser configurations. This weakens many JavaScript-based challenges and fingerprinting techniques.

Used with automation suites like OpenBullet, these tools let attackers run "low and slow" distributed attacks that blend into normal traffic. For more information on these tools, see our guide to enterprise bot management.

The Flawed Logic of CAPTCHA

For years, CAPTCHA has been the default way to distinguish humans from bots. It is now a weak control when used on its own. Our research shows that visible CAPTCHAs have a severe negative impact on user experience and conversions. Studies have found that CAPTCHAs can reduce form conversions by up to 40%, as frustrated users abandon purchases or sign-ups.

Modern bots can also solve CAPTCHAs with high accuracy, often more effectively than humans, by using CAPTCHA-solving farm services. Relying on CAPTCHA alone creates friction for legitimate users while providing a false sense of security. Modern bot management uses invisible challenges and behavioural analysis to validate users without disrupting their session.

Modern Bot Management Capabilities for IAM

An effective bot management solution provides a multi-layered defence that goes beyond simple signatures. Key capabilities include:

Advanced Rate Limiting: Instead of relying on IP addresses, modern solutions group requests using more stable identifiers like TLS/HTTP2 fingerprints, device characteristics, or a combination of headers. This helps detect distributed attacks from a single malicious tool, even as it rotates through thousands of IPs.
Network and Device Fingerprinting: By analysing the unique characteristics of a client's TCP and TLS implementation, it is possible to identify the underlying software making the request, regardless of the user-agent header. This helps distinguish between real browsers and automated scripts.
Behavioural Analysis: Systems can model normal user behaviour—such as mouse movements, typing speed, and page navigation—to identify anomalies that indicate automation.
Residential Proxy Detection: Specialised techniques are required to identify traffic coming from residential proxy networks, which is a strong indicator of malicious intent.
Breached Credential Integration: By checking login attempts against databases of known breached credentials, security teams can apply additional scrutiny to high-risk authentication events.

Together, these controls give IAM teams more useful decision points than an IP address, a password check, or a CAPTCHA challenge alone.

The Next Frontier

The next major change in automated traffic is agentic AI. As reasoning models like DeepSeek become more accessible, we are entering an era where AI agents are becoming primary consumers of APIs and web applications.

These are not just the rigid scripts of the past. AI agents can reason, plan, and adapt their behaviour in real-time based on a system's responses. They can analyse an entire API surface in seconds and generate complex interaction patterns that human developers would be unlikely to try manually.

This creates a harder IAM problem. Bot management has usually looked for patterns that differ from normal human behaviour. AI agents can make those patterns less reliable by imitating user behaviour while still operating at machine speed. The line between human and automated traffic blurs.

IAM leaders need bot management solutions that can adapt to this shift. The future of bot management will not only be about blocking bots; it will also be about deciding which automated agents are acceptable, under what conditions, and with which controls. This requires a shift from static, rule-based security to contextual analysis that understands and adapts to agent behaviour, distinguishing between legitimate AI assistants and malicious ones. Organisations that wait until agent traffic is common will have less time to distinguish useful automation from AI-driven attacks.

The Negative Impact of Visible CAPTCHAs on Bounce Rates and Conversions

2025-08-06T13:00:00+10:00

A client of Peakhour's recently migrated their site to us from a bot management vendor that used visible CAPTCHAs. After the migration they noticed double-digit year-on-year growth in revenue and conversions. We'd like to take the credit, but there is a simpler explanation: our bot management works differently. We use targeted invisible challenges to verify browser environments rather than visible CAPTCHAs. Could that account for such a large difference? I decided to check what the research says.

Visible CAPTCHAs (like image-selection or text puzzles) are still a common way to separate humans from automated traffic. They can block some bots, but they also add friction for legitimate users. Recent analyses across providers, including Human (formerly PerimeterX), Google’s reCAPTCHA, Arkose Labs, and others, point to measurable impacts on ecommerce performance and customer behaviour.

Conversion Rate Impacts of Visible CAPTCHAs

Multiple studies and vendor reports show that visible CAPTCHAs can substantially reduce conversion rates. A Stanford University study found that a CAPTCHA challenge can reduce form conversions by up to 40%. In practical terms, many users abandon signup or checkout forms when they hit a CAPTCHA. HUMAN Security researchers similarly found that 40% of real human shoppers have given up on a purchase due to CAPTCHA frustration. For an online retailer, losing up to 40% of potential sales at the final hurdle is a direct revenue problem. Forrester Research reported 19% of consumers have abandoned a website entirely because of encountering a CAPTCHA – showing how these challenges can drive users away before conversion.

Even smaller conversion drops matter. One bot mitigation firm (Datadome) observed that adding a CAPTCHA to a site led to a 3.2% higher bounce rate and an overall 3–5% drop in conversion. Given that average e-commerce conversion rates are often just 2–3%, losing even a few more percent of customers can materially affect revenue. In industries with thin margins and high customer acquisition costs, no business wants to sacrifice those would-be buyers.

Bounce Rates and User Abandonment

Every extra step in the user journey increases the risk of bounce (users leaving after a single page). CAPTCHAs are a common cause. Studies show about 30% of users will leave a site if a CAPTCHA is too complex or cumbersome. Long or indecipherable challenges cause users to give up, raising bounce rates. In one example, shoppers who faced repeated CAPTCHA puzzles during checkout simply exited, contributing to higher cart abandonment and bounce metrics. Even users who intend to buy may get frustrated by being treated like “bots” and decide the purchase is not worth the hassle.

Online patience is thin. Customers expect quick transactions, especially on mobile devices. CAPTCHAs slow things down: one analysis noted completing actions on mobile takes 30–40% more time with a CAPTCHA than without. That delay is enough to hurt conversion, as hurried mobile users are quick to drop off. As a result, visible CAPTCHAs often correlate with higher bounce rates and shorter time-on-site, indicating that challenged users are abandoning sessions. Some reports estimate 20% of users will leave if they encounter difficulties solving a CAPTCHA. This abandonment directly translates to lost sales or sign-ups.

User Experience Friction and Qualitative Insights

Qualitatively, visible CAPTCHAs introduce real user experience (UX) friction. Usability experts note that CAPTCHAs are often hard to read, carry no real meaning for the user, and feel like an unnecessary test, all of which irritate customers and “kill” conversions. They can be especially off-putting to certain user segments, such as older users or those with disabilities. For example, visually impaired users may find image CAPTCHAs nearly impossible to complete, leading to exclusion and site abandonment (W3C Accessibility Guidelines). Legitimate customers often feel inconvenienced or even insulted by being forced to “prove” they are human. As one industry commentator quipped, “Every CAPTCHA is a time tariff imposed on your customers” – it is a tax on their time and patience, which benefits nobody in terms of sales.

False positives (human users being mistaken for bots) make this worse. If a security system is too sensitive and throws frequent CAPTCHAs at real shoppers, it creates friction without benefit. Users confronted with multiple challenges may think twice about continuing. DataDome’s research notes that excessive CAPTCHA usage causes a suboptimal experience, and customers have “little patience” for such delays. The result can be reputational damage as well – annoyed users might complain publicly, hurting the brand. In short, traditional CAPTCHAs tend to “treat customers like criminals”, which pushes people away.

Industry-Specific Observations

The negative effects of visible CAPTCHAs show up across industries, with different tolerance for security friction:

Retail & E-Commerce: Online retail (from fashion to electronics) is highly sensitive to checkout friction. Shoppers have many alternatives, so a challenging CAPTCHA can send them to a competitor’s site in seconds. Even one extra step can hurt sales conversion. E-commerce case studies consistently show that removing CAPTCHAs boosts conversions. Bot solution vendors note that losses from CAPTCHA friction are “especially noticeable in areas such as e-commerce” where even a 3–5% conversion dip means thousands of customers lost.
Travel & Ticketing: Travel sites (flights, hotels) and ticketing platforms often deploy CAPTCHAs to thwart scalpers and bots (e.g. for popular concert tickets or holiday bookings). That can protect inventory, but it can also turn away real customers. Travellers shopping around for deals won’t hesitate to bounce if a booking site throws up hurdles – they’ll try another site. Travel bookings are often time-sensitive (flash sales, limited seats), so any slowdown from a puzzle challenge can cause users to miss out and blame the site. The challenge for this sector is to weed out bot traffic (which can be a huge share of ticketing traffic) without derailing genuine user transactions. Some travel companies use alternatives like virtual waiting rooms or invisible challenges to reduce user-facing friction. A smooth booking path matters: industry observers emphasise that travel and hospitality businesses that “remove booking friction” are rewarded with higher conversion and direct revenue, whereas those using blunt CAPTCHA challenges risk higher bounce rates and lost bookings.
Finance & Banking: Financial services websites (online banking, fintech apps, etc.) deal with sensitive transactions and may introduce verification steps (CAPTCHAs or multi-factor authentication) for security. Users in this sector can be slightly more tolerant of friction if it clearly signals security. However, if a bank’s CAPTCHA fails normal users or frequently interrupts login, customers will get frustrated or call support. Financial institutions must balance fraud prevention with a smooth experience – if there is too much friction, users may abandon opening an account or using the service. In fact, the same 40% conversion drop risk applies here: lost applications or completed transactions if security measures are overbearing. So even in finance, the trend is toward smarter, invisible verification methods to minimise extra steps.
Other Industries: Nearly any consumer-facing online service – from gaming and streaming to government portals – faces the CAPTCHA UX dilemma. Users expect minimal friction. In some niches (e.g. gaming), users are extremely averse to any interruptions in sign-up or sign-in. In others (like online voting or government forms), a CAPTCHA might be more accepted, but if it fails or confuses users, it can prevent task completion. The pattern is consistent: user expectations for convenience are high across the board, and visible CAPTCHAs risk alienating users in any vertical if not handled carefully.

Death of the CAPTCHA

Visible CAPTCHAs are still widely used, and they still add friction at critical moments. The research above shows users often abandon sites or carts rather than struggle with puzzles – a loss of sales that can range from a few percent to double digits. This effect is seen broadly, from fashion retail to travel bookings to financial services.

That is before we consider the actual effectiveness of visible CAPTCHAs. Other studies have shown bots can be MORE effective than humans at solving them.

That is why Peakhour uses invisible challenges: verify the browser environment without making legitimate customers solve a puzzle.

AI as the Translator Between Human and Machine

2025-07-19T00:00:00+10:00

Think about how we've talked to computers over the years. At first, it was rigid and unforgiving. The command line expected the exact words, in the exact order. One typo, and you were met with an error. It was powerful, but only once you learned to speak the computer's language.

Then came the graphical user interface, or GUI—the familiar world of windows, icons, and mouse pointers. That changed the relationship. You no longer had to memorise commands before you could do something useful. You could see your options, click on them, and drag things around. It made computers accessible to hundreds of millions of people because it was more intuitive. It was a visual conversation.

But both of these interfaces, the command line and the GUI, share the same basic bargain: we adapt ourselves to the computer. We still have to navigate menus, find the right button, or remember a specific command. We take a goal in our head and break it into steps the computer understands.

What if that translation was no longer mainly our job? What if the computer could understand our goal well enough to work out the steps?

This is the next shift I find interesting, and it is powered by Artificial Intelligence. AI is starting to look less like another application and more like the next major interface. It's not a visual one with buttons and menus, but an intelligent one built on understanding.

The idea is simple, even if the implementation is not: we state our intent, and the AI figures out the steps. Instead of clicking through five different menus to create a sales report, you could just say, "Show me last quarter's sales figures for the eastern region, and visualise it as a bar chart." The AI's job is to understand that request and then do the work: query the database, aggregate the data, select the right chart type, and present it to you. It acts as a translator between human language and the computer's machine language.

We're already seeing the early stages of this. When you ask a smart assistant to play a song, or when an AI co-pilot writes code for you, you're using an intent-driven interface. You're not telling it how to do the task; you're just telling it what you want done.

That shift matters because it moves some of the cognitive load from us to the machine. We no longer need to be experts in using a particular piece of software; we just need to be clear about what we want to achieve. This has the potential to democratise technology on a scale we've never seen before, making complex digital tools feel closer to a conversation than a training course.

The future of computing isn't about learning more complex systems. It's about building systems that can learn from us. The interface of tomorrow won't be something we click on, but something we talk to, correct, and steer. That is the real change: technology that doesn't just follow instructions, but understands our goals.

A Complete Guide to SMS Pumping Fraud

2025-03-13T14:00:00+11:00

The Growth of SMS Fraud

SMS pumping fraud is a costly online abuse pattern, with global losses reaching an estimated $6.7 billion in 2021 alone. It targets companies that rely on SMS for verification or customer communications, leaving them to pay for traffic they did not request.

The scheme relies on malicious actors and dishonest telecom operators working together to generate and monetise large volumes of fraudulent text messages. For businesses caught in these schemes, the financial impact can be severe. Twitter (now X) reportedly lost $60 million to this type of fraud.

This guide explains how SMS pumping works, which businesses face the highest risk, and the controls your organisation can use to reduce exposure.

Understanding SMS Pumping Fraud

SMS pumping (also called SMS toll fraud, SMS spamming, or Artificially Inflated Traffic) involves manipulating mobile networks to inflate charges for text messages. The term "pumping" describes fraudsters forcing high SMS volume through a target's systems.

This fraud exploits how SMS messages travel and get billed across phone networks. Attackers target companies that use SMS codes to verify users. Each time a business sends a verification code, it pays a fee. Fraudsters trigger these systems to send thousands of messages to numbers they control.

These attacks create direct costs for businesses and revenue for the attackers. The fraud works through coordination between criminals and corrupt telecom operators, who charge premium rates for message delivery and share the proceeds.

The fraud has changed as more businesses have adopted SMS verification. Attackers keep developing new methods, and the phone industry has not removed the risk. Many companies still carry the financial exposure.

How SMS Pumping Works

SMS pumping attacks usually exploit message systems through these steps:

Finding Targets: Attackers look for websites or apps that send SMS codes for account verification or password resets.
Creating Fake Requests: Fraudsters use automation to send thousands of code requests to phone numbers they own or control.
Hiding Their Tracks: Attackers change their IP addresses and device information so requests appear to come from real users.
Sharing Profits: Fraudsters work with dishonest phone companies that charge high fees when messages pass through their networks. These companies then share the money with the attackers.
Using Complex Routes: Messages travel through many networks before reaching their destination, making the source of the fraud harder to trace.
Targeting Expensive Routes: Attackers focus on international numbers where sending messages costs more or where rules are weaker.

These attacks look legitimate because each message contains a real code sent to what appears to be a normal phone number. Companies like Twilio or Bird must pay fees to deliver these messages. Most businesses only find out about the fraud when a large bill arrives from their SMS service.

SMS pumping differs from basic spam because the profit-sharing between attackers and phone companies creates a direct cost for the target business.

Businesses at Risk

SMS pumping is most likely to affect these types of businesses:

Financial Institutions

Banks, investment platforms, and cryptocurrency exchanges use SMS codes to protect accounts. These firms send thousands of codes each day, which makes it hard to spot fake requests mixed with real ones.

E-commerce Platforms

Online shops use SMS messages when users create accounts, reset passwords, or make purchases. These businesses often run on small profit margins, so extra SMS costs can hurt their earnings. High volumes of new users make it easier for attackers to hide their activity.

Social Media Companies

Social networks use text messages to check user identity and stop fake accounts. These companies send millions of codes each day to users around the world. Twitter lost $60 million from this type of fraud, showing the scale these bills can reach.

Software-as-a-Service (SaaS) Providers

These companies often offer free trials that require SMS verification. They plan for a set cost to acquire each new user, but fraud can push these costs much higher than expected.

Telecommunications Companies

Phone companies face two problems: their own systems can be attacked, and parts of their network might help fraudsters. They need strong monitoring tools to find unusual patterns in message traffic.

Small Businesses and Startups

While smaller firms send fewer messages, they often lack security teams and fraud detection tools. This makes them easier targets. The cost of an attack can put these businesses at risk of closing down because they have less money in reserve.

Advanced Attack Methods

Attackers now combine SMS pumping with other techniques to avoid detection.

Credential Stuffing

Fraudsters use passwords stolen in data breaches to break into accounts. Once inside, they change phone numbers to ones they control and trigger verification messages. This makes fraud appear to come from real users.

Peakhour's breach database detection identifies when stolen credentials are used to access accounts. The system flags these attempts before phone numbers can be changed, stopping the attack chain.

Residential Proxy Networks

Unlike data centre proxies that security systems can often spot, residential proxies hide attack traffic behind home internet connections. This makes fraud look like it comes from regular users in different locations.

Peakhour specialises in residential proxy detection. Its technology identifies these masked connections and blocks them before verification requests can pass through. The system maps known proxy networks and detects signs of traffic passing through residential IPs.

When combined with device fingerprinting, these protections create a stronger defence. Fingerprinting tracks device characteristics that remain consistent even when attackers change IP addresses or accounts. Peakhour's fingerprinting technology works without cookies, making it effective against attackers who clear browser data.

These methods focus on the techniques fraudsters use to hide their identity. With Peakhour's protection, businesses can detect and block these attacks before they trigger costly SMS verification messages.

Historical Incidents

Reported SMS pumping incidents show how quickly costs can build:

Twitter's $60 Million Loss

In January 2023, Twitter owner Elon Musk said the platform lost more than $60 million to SMS pumping fraud. He named over 390 phone companies that took part in the scheme. While Twitter later questioned some claims, the case brought public attention to this type of fraud.

Industry-Wide Financial Impact

The Communications Fraud Control Association reports that SMS pumping caused global losses of $6.7 billion in 2021. Many companies do not share their fraud losses with the public.

Costs to Individual Businesses

Companies hit by these attacks pay between tens of thousands and millions of dollars each month in fake charges. These costs grow fast because each fake message costs much more than normal text rates.

Verification Policy Changes

Because of these threats, many large platforms have moved away from SMS codes. Twitter removed SMS verification for most users in March 2023, stating fraud as the reason.

Operational Disruptions

Beyond the cost of messages, businesses can face service problems during attacks. Real users may not get their codes on time. This can cause users to abandon transactions, contact support more often, and lose confidence in the company.

Rules and Enforcement

Rules to stop these attacks differ around the world. Some telecoms authorities have strict rules and fines for networks that allow fraud, but enforcement remains hard. Fraudsters use complex message routes that cross many countries to avoid getting caught.

Understanding the Stakeholders

SMS pumping involves these key groups:

Businesses

Companies use SMS to check user identity and send updates. They hire SMS gateway providers to handle their messages. When fraud happens, these businesses pay for the fake messages. Most find out about the attack only when they receive an unexpected bill.

SMS Gateway Providers

Companies like Twilio and MessageBird connect businesses to phone carriers. They give businesses tools to send text messages without working with phone networks directly. When fraud passes through their systems, these providers may try to stop it, but still charge businesses for the messages sent.

Mobile Network Operators (MNOs)

These companies run the networks that deliver messages to phones. Most work honestly, but SMS pumping schemes often include corrupt operators who charge extra fees for messages to numbers they control. These operators then split the money with the attackers who started the fraud.

Content Aggregators

These middlemen combine message traffic and work with many carriers to find the best routes. Most run honest operations, but their position in the message chain creates routing and oversight gaps that attackers can use.

Regulatory Bodies

Groups like the GSM Association create rules and standards for the industry. These rules are hard to enforce because phone networks cross many countries with different laws.

Financial Flow

The payment flow starts when businesses pay gateway providers to send messages. The gateway providers then pay fees to network operators based on where messages go. In fraud schemes, inflated fees go to corrupt operators who share the money with attackers. This creates a system where sending more fake messages makes more money for criminals while costing honest businesses more.

Effective Protection Strategies

Protecting your organisation usually requires several controls:

Basic Protections

Rate Limits: Restrict how many verification attempts a user can make in a set time period.
Traffic Pattern Checks: Track normal SMS message patterns and watch for changes that might indicate attacks.
Provider Protection: Services like Prelude's SMS Pumping Protection find and block messages to fake numbers.
Other Ways to Verify Users: Use app-based verification or push alerts instead of SMS codes.
Control by Country: Limit SMS verification to countries where you do business and add more checks for countries with higher fraud risk.
Work with Trusted Partners: Choose SMS service providers that focus on security and can help stop fraud quickly.

Advanced Protection Methods

Residential Proxy Detection: Find and block users who hide their true location behind home networks used as proxies.
Device Fingerprinting: Collect device signals to track users across sessions and spot when many verification requests come from the same device.
User Behaviour Tracking: Learn how real users act on your site and flag unusual actions that might be bots.
Machine Learning Systems: Use systems that learn from data to find hidden fraud patterns and adapt to new attack types.
Phone Number Checks: Use lists of known bad numbers to decide which phone numbers need more verification steps.
Verify in Multiple Ways: Ask users to prove who they are in different ways, such as email plus SMS, to make attacks harder.
Work with Other Companies: Share information about new attack methods and bad phone numbers with other businesses.
Watch Transactions as They Happen: Use systems that can pause message sending when they spot unusual patterns and learn from both legitimate and abusive traffic.

Fighting SMS Pumping Fraud

SMS pumping fraud costs businesses $6.7 billion worldwide each year. Companies like Twitter lost $60 million to these attacks, showing that scale alone does not remove the risk.

SMS pumping works through a network of fraudsters, network operators, and service providers who exploit the payment system for text messages. Fraudsters target authentication systems to generate large volumes of SMS, then collect revenue shares from the process.

Peakhour and Prelude offer combined protection against these threats. Peakhour provides device fingerprinting to identify suspicious devices attempting verification. Its residential proxy detection stops fraudsters who hide behind legitimate IP addresses. These tools block attackers before they access verification systems.

Prelude complements this protection with their multi-routing SMS verification platform. Its system uses real-time fraud detection across five messaging channels in 230 countries. When Prelude detects a potential attack, it automatically redirects traffic through secure routes.

Businesses need to understand the SMS delivery chain to protect themselves. Gateway providers, network operators, and content aggregators each introduce possible points of exploitation.

Prevention requires multiple security layers:

Rate limiting to restrict message volume
Device fingerprinting to track suspicious patterns
Residential proxy detection to unmask hidden attackers
Behavioural analytics to spot unusual activity
Machine learning to adapt to new attack methods
Continuous learning based on real user interactions

The continuous learning systems from both Peakhour and Prelude build protection that improves with each user interaction. Their platforms analyse legitimate traffic patterns to differentiate them from attacks, helping protection adapt over time.

While SMS verification remains common, Peakhour and Prelude help businesses implement more secure authentication methods. Together, they provide protection that adapts to evolving threats and reduces the cost of fraudulent verification traffic.

See how Peakhour's Application Security Platform helps protect against SMS pumping and other automated threats. Contact our team to secure your applications.

How AI Agents Are Writing Custom Exploits

2025-02-17T14:00:00+11:00

The trend is clear enough: AI agents can now craft exploits by analysing security responses in real time. That puts static security rules and traditional Web Application Firewalls (WAFs) under direct pressure. Here is why.

Last week I examined an AI agent probing a test environment. It sent requests, observed the responses, then built bypasses for each security control in sequence. The agent identified pattern-based rules, learned their structure, and generated variations until it found gaps. It did this without human intervention.

This kind of automated exploit development changes the operating conditions for defenders. Traditional defences rely on known patterns: regex rules, signature matching, IP reputation. Those approaches assume threats follow recognisable templates. That assumption is becoming much weaker.

Consider a standard WAF rule blocking SQL injection through pattern matching. An AI agent examines the responses, determines the matching patterns, then generates unique variants designed to bypass those rules while maintaining the exploit's functionality. The variants evolve as the agent learns which approaches succeed.

The same pattern applies beyond SQL injection. AI agents can probe XSS filters, access controls, and input validation in the same systematic way. Each static rule becomes something the agent can test, infer, and work around.

By 2026, I estimate AI agents will drive over 50% of exploit attempts. The speed of this shift stems from three factors:

AI agents operate continuously, testing and learning 24/7
Successful exploits feed back into training data, improving future attempts
Agents share knowledge, building collective intelligence about bypass techniques

This is the practical limit of static security. Traditional WAFs that rely on fixed rules and signatures struggle to keep pace with AI-generated exploits. Each rule loses value as agents discover new bypasses.

The path forward requires a different security architecture. Organisations need context-aware systems that analyse intent, not just patterns. These systems use behavioural AI to distinguish between legitimate requests and exploit attempts, even when the request structure changes.

Key elements of this new approach include:

Intent analysis through deep inspection of request sequences
Behavioural modelling of normal vs malicious patterns
Real-time adaptation as new exploit techniques emerge
Proactive identification of potential vulnerabilities
Integration of threat intelligence across systems

The challenge intensifies when AI agents leverage residential proxies. These proxies route traffic through real consumer IP addresses, bypassing location-based blocks. An AI agent operating through residential proxies can probe defences while appearing to come from legitimate users worldwide.

This combination of AI-driven exploit generation and residential proxy networks makes traditional controls much less reliable. Organisations that continue to rely on static rules face a growing risk of compromise.

Security teams should respond now:

Audit existing WAF rules to identify pattern-based weaknesses
Deploy behavioural analysis capabilities to detect malicious intent
Implement adaptive security controls that evolve with threats
Monitor for AI-driven probing attempts
Build detection for residential proxy traffic

Teams that wait risk watching their defences get mapped and bypassed by automated agents. Static rules alone are not enough for this level of probing.

This also requires a shift in how we approach security. Rather than only blocking specific patterns, we need to understand and control the broader context of system interactions. The goal moves from "preventing known attacks" to "identifying and blocking malicious behaviour, regardless of its specific form."

Adaptive security systems need to reason about traffic in the same context-aware way as the agents probing them. Static rules still have a role, but they cannot be the centre of the defence.

Security strategy needs to account for this now, because AI-driven probing is no longer hypothetical.

The Reasoning Model Revolution

The emergence of open reasoning models like DeepSeek pushes this further. Unlike traditional AI that follows programmed patterns, reasoning models understand context and adapt strategies dynamically. That creates harder security problems.

Consider how a reasoning model approaches security testing. Rather than simply probing for weaknesses, it builds a conceptual model of the system's defences. It understands the purpose of security controls and reasons about potential bypasses. That allows it to generate novel attack strategies that were not present in training data.

DeepSeek demonstrates this shift. Within months of release, it showed capabilities matching established players at a fraction of the cost. This rapid progress comes from reasoning models' ability to understand and adapt, not just pattern match.

For security teams, that is a material challenge. Reasoning models do not just find gaps in rules. They infer why rules exist, deduce the logic behind security controls, and generate attacks that exploit underlying assumptions.

By 2027, I expect reasoning models to handle most security testing and exploit development. Their advantages prove too compelling:

They understand system architecture and security principles
They generate novel attack strategies through reasoning
They adapt in real-time based on system responses
They share and build upon successful approaches

This shift pushes traditional security approaches past their useful boundary faster than many teams expect. Pattern matching and rule-based systems cannot reliably counter an opponent that understands and reasons about their operating logic.

The combination of reasoning models with residential proxies is especially difficult to defend against. Reasoning models devise sophisticated attacks while proxies mask their origin. Each successful breach feeds back into the model's understanding, improving future attempts.

Security teams must embrace a new paradigm focused on:

Understanding attack narratives rather than patterns
Detecting anomalous reasoning rather than known signatures
Building systems that adapt to novel attack strategies
Implementing security that reasons about intent
Developing defences that evolve through adversarial learning

Security systems need to reason about threats as effectively as the AI agents probing them. Traditional approaches will fail against opponents that understand the logic behind security controls and devise creative bypasses.

The age of reasoning security has begun. Static rules and pattern matching are no longer enough on their own.

The question is how quickly security teams can move from fixed patterns to adaptive, intent-aware defence.

A/B Testing, Bots, and the New Normal

2025-02-13T14:00:00+11:00

Marketing traffic is no longer just human traffic. We spent months analysing how AI agents interact with websites, and the findings put pressure on a core assumption behind digital marketing: that campaign data mostly reflects human behaviour.

Consider a simple A/B test on a landing page. The data shows a clear winner, with conversion rates 30% higher than the control. You ship the change, then the live result misses the forecast. One possible cause is that AI agents were counted in the test population.

This is not speculation. We have observed AI agents, from shopping bots to research tools, interacting with marketing campaigns in ways that weaken traditional metrics. These agents do not behave like people. They optimise for efficiency, not experience. They follow patterns, not preferences.

The consequence is straightforward. A/B test results may reflect what works best for AI agents rather than human users. Conversion metrics may combine human decisions with automated actions. A marketing funnel can end up optimising for the wrong audience.

The scale became clear when we analysed traffic patterns across various sectors. In ecommerce, AI agents now account for up to 40% of product page views. For content sites, the figure rises to 60% for certain categories. These are not simple scraper bots. They interact with content, follow links, and even complete transactions.

Blocking all of this traffic is not a practical answer. Many agents serve legitimate purposes, from price comparison to content aggregation. Marketing teams now need to recognise and measure a dual audience: human and AI.

Through our research, we have identified three shifts in marketing strategy:

First, marketing funnels need to bifurcate. One path should be optimised for human users, with emphasis on engagement and experience. Another should serve AI agents, with structured data and efficient access to information.

Second, A/B testing needs new frameworks. Analysis should separate AI and human interactions. That requires stronger detection methods and separate datasets for each audience type.

Third, attribution models need to evolve. When an AI agent researches products before recommending them to a human user, who gets credit for the conversion? The traditional last-click model does not capture that sequence well.

The problem is not limited to metrics. AI agents use residential proxies to mask their nature, making detection difficult. They learn and adapt, which means identification methods need regular updates. They also operate at scale, potentially overwhelming analytics systems.

There are useful opportunities here, but they need to be treated with discipline. Organisations that adapt their marketing strategies can optimise content delivery for both audiences. They can use AI agents as a marketing channel where that makes sense. They can also build campaigns that are less exposed to polluted measurement.

The path forward starts with a clearer operating model. Marketing teams need to treat AI agents as a distinct audience segment, with different behaviours and requirements. They also need tools and metrics that measure performance across both audiences without mixing them together.

Our research suggests several practical steps:

Implement robust bot detection systems to identify AI agent traffic. This gives teams the basis for separate analysis of human and AI interactions.

Develop structured data formats that serve AI agents efficiently while maintaining rich experiences for human users.
Create attribution models that account for the role of AI agents in the customer journey.
Monitor residential proxy usage to understand the true nature of website traffic.
Build marketing strategies that balance the needs of both audience segments.

Marketing data will keep changing as AI agents become more sophisticated. The line between human and automated interactions may blur further. Teams that recognise the split can make better decisions about testing, attribution, and campaign design.

Organisations that continue to treat all traffic as human risk optimising for the wrong audience. Organisations that separate the signal can make better use of both human and AI traffic.

Our next challenge is understanding how AI agents influence human decision-making. That is a separate question.

The views expressed in this article reflect our research and analysis of AI agent behaviour in marketing environments. We encourage organisations to conduct their own analysis and develop strategies suited to their specific circumstances.

When Bots Are Your Primary Users

2025-02-12T14:00:00+11:00

APIs have mostly been designed for human developers first. Reasoning models like DeepSeek make that assumption weaker. If an agent can inspect an API, plan a sequence of calls, and adapt as it goes, it becomes a different kind of consumer.

That is the part worth paying attention to. Many APIs still assume a human-first model while AI agents become regular, and in some cases primary, users. These are not simple scraping bots or automation scripts. Modern AI agents can plan, reason, and change their behaviour. They interact with APIs in ways many teams did not account for when they wrote their OpenAPI specifications and documentation.

A human developer reads documentation, tries a few calls, and works through errors. An AI agent can process the whole API surface in seconds, generate thousands of possible interaction patterns, and test them systematically. That difference changes both API design and API security.

The issue is not limited to technical specifications. API logs already show traffic patterns that challenge older assumptions. AI agents do not follow typical "business hours" usage. They do not slow down because a workflow becomes cognitively heavy. They process responses at machine speed and chain API calls in ways human developers rarely attempt.

This shift forces us to rethink several core aspects of API design:

Structure and Format

Human-readable formats still matter, but they are not the only target. JSON and REST endpoints work well for developers who need to read and understand responses. For AI agents, there may be room for more efficient formats that optimise for machine processing rather than human comprehension.

Rate Limiting and Quotas

Most rate limiting models still assume human consumption patterns. AI agents operate at machine speed and scale. New models need to account for that processing capacity while still preventing abuse. That may mean moving from simple request counts to complexity-based quotas.

Authentication and Security

Traditional API keys and OAuth flows centre on human developers. AI agents need security models that account for how they operate. The hard problem is verifying the identity and intentions of an AI agent without weakening the security controls around the API.

Documentation and Discovery

API documentation still focuses on human understanding. For AI agents, machine-readable specifications need to go beyond OpenAPI. They should describe what endpoints do, not just how to call them.

This also changes how we monitor and maintain APIs. Traditional metrics like response time and error rates remain useful, but they do not explain AI agent behaviour on their own. How do we measure the "success" of an API when its primary users are machines that can adapt to problems and work around them?

Performance optimisation changes as well. A human developer might tolerate occasional latency. An AI agent can make thousands of calls per second, which puts more pressure on caching, edge computing, and response optimisation.

APIs are likely to split into two parallel tracks: human-oriented interfaces that prioritise developer experience, and machine-oriented interfaces optimised for AI consumption. This is not a choice between one audience and the other. It is recognition that they have different needs.

The challenge extends to business models. How do we price APIs when consumers are AI agents that can process information at machine scale? Traditional per-request pricing may not make sense when an AI can make millions of optimised calls that would take a human developer years to replicate.

Residential proxies add another layer of complexity. They allow AI agents to appear as regular users, making it harder to distinguish between human and machine traffic. That pushes API access control beyond IP-based rate limiting.

The ethical questions also matter. As APIs become primarily consumed by AI agents, teams need frameworks for responsible use. That includes asking how an API might be used inside AI systems, and what guardrails should sit around that access.

This is not about replacing human developers. It is about recognising AI agents as a new class of API consumer, with their own needs and capabilities. API design, security, and management all need to account for that.

The APIs we build today will sit under tomorrow's AI-driven systems. They need to be designed for both human and AI consumers, with clear decisions about discovery, access, rate limits, authentication, monitoring, and abuse controls.

The shift to AI-first API design is already under way. The practical question is how quickly API practices can catch up.

Our APIs have to evolve with their users.

Why Reasoning Models Like DeepSeek Change Everything

2025-02-03T08:13:00+11:00

Open reasoning models change how we need to think about automation and security. Looking at models like DeepSeek, the important shift is not another small gain in AI capability. It is the move towards autonomous agents that can plan, reason, and adapt without human guidance.

This became clear while analysing recent credential stuffing attacks. The patterns showed attackers using AI agents to probe systems, identify vulnerabilities, and craft custom exploits. These were not pre-programmed scripts following rigid rules. They were agents making decisions based on the system's responses.

The implications go beyond security. Consider how marketing teams usually approach A/B testing and campaign optimisation. Most tools and frameworks assume automation follows fixed paths: if this happens, do that. Reasoning models do not fit that model. They can work without predefined decision trees or explicit step-by-step instructions. They observe, learn, and create their own strategies.

This forces us to rethink basic assumptions about digital interactions. When an API call could come from an AI agent rather than a script, how do we distinguish friend from foe? Traditional markers such as request patterns, user agents, and IP addresses carry less weight when an agent can analyse and adapt to detection methods.

The same problem applies to customer engagement. Marketing funnels designed for human decision-making now face AI agents that can evaluate options systematically, compare alternatives across multiple sources, and make optimised choices. The customer journey stops being a neat linear path and becomes a space where AI agents operate alongside human users.

Reasoning models also challenge the way we approach bot management. Traditional methods focus on identifying automated behaviour: patterns that deviate from human norms. But what happens when AI agents can mimic human behaviour while operating at machine speed? The line between human and automated traffic becomes harder to draw.

Through conversations with security teams, I have seen this pattern emerge. They report sophisticated attacks that adapt in real-time, probing defences and adjusting tactics based on system responses. These are not pre-programmed behaviours. They are reasoning models understanding and responding to defensive measures.

The business impact extends beyond security. Companies need to adapt digital infrastructure for a world where AI agents become primary users. That means rethinking API design, service architecture, and customer interaction models. The question is not whether to support AI agents, but how to do it safely and effectively.

Authentication is a good example. Traditional systems often rely on proving human presence through CAPTCHAs, behaviour analysis, and device fingerprinting. In a world of reasoning models, we need approaches that focus on intent and trust rather than a simple human versus machine test.

The path forward is a shift in perspective. Rather than only trying to block or restrict AI agents, we need systems that can interact with them safely. That means moving from static rule-based security to contextual analysis that understands and adapts to agent behaviour.

The strategic implications for businesses are significant. Success in this environment requires a clear understanding of how reasoning models operate. Companies must redesign digital interfaces to support both human and AI interactions while maintaining security and control.

From my analysis of current trends, this change is accelerating. Each advance in reasoning models expands their capability and autonomy. Organisations that adapt their strategies now will be better positioned as this digital environment changes.

The rise of reasoning models is more than another technology upgrade. It changes how we approach automation, security, and digital interaction. Organisations need systems capable of engaging safely and effectively with autonomous AI agents.

The question is not whether reasoning models will change business operations. They already are. The practical question is how quickly organisations can adapt their strategies and infrastructure, and whether they can do it without losing control of trust, security, and user experience.

Did Residential Proxies enable a $600 Billion loss?

2025-01-31T00:00:00+11:00

The DeepSeek story puts residential proxy networks under scrutiny as a possible factor in AI's latest market disruption. In January 2025, the Chinese startup's emergence erased $600 billion from Nvidia's market value by demonstrating AI capabilities that match industry leaders at a fraction of the cost.

The path to this capability raises a practical security question for AI platforms. Leading platforms protect their APIs with multiple security layers - rate limiting to prevent mass data extraction, bot detection to block automated requests, and geoblocking to restrict access from certain regions. These measures are meant to prevent the systematic collection of training data.

Residential proxy networks create a route around those protections. These networks route traffic through household IP addresses, so requests appear to originate from homes in permitted regions. A request from a restricted location could look like legitimate traffic from Sydney, Melbourne, or Perth.

The circumstances suggest this approach is plausible. By distributing requests across millions of residential IPs worldwide, each IP could maintain human-like patterns while staying below rate limits. The aggregate data could form a substantial training set without triggering security alerts.

Meta's lawsuit against Bright Data strengthens this possibility. The case exposed how proxy providers monetise residential IPs, often without homeowners' knowledge. That model creates a global network capable of bypassing traditional security measures - exactly the type of infrastructure needed for large-scale data collection.

The residential proxy industry threatens $600 billion in business value through data theft and security bypasses. DeepSeek's impact on Nvidia's market capitalisation highlights the real-world impact of residential proxies.

For AI platforms, the question is operational. How can platforms distinguish between legitimate users and well-crafted requests through residential proxies? When geographical restrictions lose meaning, what security measures remain effective? Traditional IP Intelligence based proxy detection based on historical usage is no longer effective; per-connection proxy detection is essential.

DeepSeek's emergence suggests AI security teams need to revisit their assumptions. The potential use of residential proxy networks to dissolve digital borders challenges current approaches to platform protection.

How Bots Are Corrupting Your A/B Testing Data

2025-01-20T00:00:00+11:00

Bot traffic contaminates A/B testing results and can undermine marketing strategy. Your testing programme is exposed when residential proxy networks generate fake interactions (e.g. click fraud) that appear to come from your target market.

These residential proxies hide behind real household internet connections in the targeted geographic areas. When a bot network routes traffic through Sydney IP addresses to masquerade as real Australians, your analytics counts that traffic as legitimate local engagement. Because the traffic matches your demographic and geographic targeting profile, traditional detection methods become less effective.

This contamination affects the accuracy of the marketing strategy. Your A/B tests should show clear winners, but the results are masked by bot behaviour rather than real user preferences. Marketing teams then optimise campaigns from false signals, wasting budget and time on the wrong opportunities. The data starts driving decisions that harm conversion rates and revenue.

The scale of the problem continues to grow. Residential proxy services now offer millions of local IPs in every market. They rotate these IPs automatically and match real browser characteristics. Without specialised detection such as Peakhour A/B Testing Protection, this traffic can appear identical to genuine users in your analytics.

Each day without detection compounds the damage. Tests generate misleading data that guides strategic decisions. Marketing teams spend hours analysing invalid results and implementing changes that reduce performance. Budget allocated to testing delivers diluted ROI as optimisations based on bot data decrease conversion rates.

Traditional bot protection fails against this distributed threat. IP-based detection cannot identify residential proxy traffic that matches your target geography. Rate limiting proves ineffective against attacks spread across thousands of residential IPs. These bots evade basic JavaScript challenges through sophisticated browser emulation.

Peakhour's A/B Testing Protection uses network fingerprinting to detect residential proxy traffic. Our system analyses subtle patterns in how these proxies connect and behave, identifying bot networks that other solutions miss. We maintain a real-time database of residential proxy services to block new threats as they emerge.

Our customers have discovered that 40% of their test traffic came from bots. After implementing protection, they achieved:

Valid test results reflecting real user preferences
Increased conversion rates from accurate optimisation
Reduced waste of marketing team time and resources
Protection of testing budget from invalid traffic
Confidence in strategic decisions based on clean data

The rise of residential proxies has amplified bot threats to A/B testing. Traffic that appears to come from local users may mask sophisticated bot networks. Protecting your testing programme requires detection that goes beyond IP addresses and basic challenges. Contact us to learn how Peakhour can help secure your A/B testing data and keep optimisation decisions grounded in real users.

How Bots Contaminate Your A/B Testing Results and Marketing Strategy

2025-01-15T13:00:00+11:00

Marketing teams invest heavily in A/B testing to optimise websites, campaigns and user experiences. These tests inform decisions about design, content and functionality. Bot traffic undermines the validity of those decisions.

The Scale of Bot Traffic

Our research shows that bots generate half of all internet traffic. This includes legitimate bots, such as search engines, and malicious bots conducting attacks. For marketing teams, this creates a direct problem: your A/B tests include manipulated responses.

Bot traffic skews test results in multiple ways. Bots do not interact with different test variants the way real users do. They follow programmed patterns rather than genuine user preferences. This contaminates the data marketing teams use to make decisions about website changes, campaign optimisation and user experience improvements.

The Impact on Marketing Strategy

Contaminated A/B test results lead to flawed strategic decisions. Marketing teams might optimise for bot behaviour rather than real user preferences. This affects several areas of strategy:

Website Design - Teams select layouts and features that perform well with bots rather than humans. Navigation flows optimise for automated traffic patterns instead of genuine user journeys. Content decisions target bot consumption rather than human engagement.

Campaign Optimisation - Bot interactions corrupt conversion rate data. Teams allocate budgets based on manipulated performance metrics. Campaigns end up catering to bot behaviour instead of real customers.

User Experience - Interface changes are skewed by bot behaviour patterns. Feature development prioritises elements that score well with automated traffic. Content strategy aligns with bot consumption rather than human needs.

The Residential Proxy Challenge

Residential proxy networks create a specific challenge for A/B testing. These proxies route bot traffic through real consumer IP addresses, making automated traffic look legitimate. Traditional bot detection methods struggle to identify this traffic.

Our research demonstrates that standard IP intelligence services miss up to 96% of residential proxy traffic. This means marketing teams include large amounts of proxy-based bot traffic in their test results without realising it.

Residential proxies mask sophisticated bot behaviour that mimics real users. The bots rotate through different residential IPs to avoid detection. They generate clicks, page views and conversions that appear genuine but represent automated rather than human interactions.

Protecting Your Tests

Marketing teams need protection measures that keep A/B test results valid. This requires a multi-layered approach to identifying and filtering bot traffic:

Detection starts with continuous monitoring of traffic patterns. Teams track user behaviour to identify automated interactions. This includes analysing click patterns, page view sequences and conversion flows that indicate bot activity.

Prevention requires sophisticated bot management capabilities. Our Bot Management solution blocks automated traffic while allowing real users to participate in tests. The system detects and filters residential proxy traffic so test data comes from genuine visitors.

Protection extends to API endpoints that support A/B testing infrastructure. Our API Security capabilities prevent bots from manipulating test data through direct API access. This ensures the integrity of test results across all interaction channels.

Making Informed Decisions

Understanding bot traffic helps marketing teams protect their investment in A/B testing. Data analysis must start by filtering bot interactions from genuine test results. Teams measure genuine user engagement rather than combined human and bot behaviour. This enables accurate assessment of test variants based on real user preferences.

Strategic planning improves once teams understand the impact of bots. Marketing decisions align with genuine user needs rather than artificial interactions. Campaign optimisation targets real customer segments instead of bot characteristics. Feature development prioritises elements that resonate with humans rather than automated traffic.

Budget allocation becomes more effective when based on clean data. Teams invest in changes that improve real user experiences rather than bot interactions. Campaign spending targets channels with verified human traffic. Development resources focus on features that drive genuine engagement.

Taking Action

Marketing teams must implement three key measures to protect A/B testing:

First, deploy comprehensive bot management to identify and block automated traffic. This forms the foundation for valid test results by ensuring participation from real users.

Second, implement residential proxy detection to prevent sophisticated bots from corrupting test data. This ensures traffic comes from genuine users rather than proxy networks.

Third, protect API endpoints that support testing infrastructure. Our Traffic Control solution provides protection across web and API interfaces.

Conclusion

Bot traffic undermines A/B testing and can push marketing teams towards flawed decisions. Past results may already contain bot interactions. The priority is to detect and filter that traffic before it shapes the next test, campaign or product decision.

Anti-Detect Browsers

2025-01-15T10:00:00+11:00

Anti-detect browsers matter to defenders because they attack the assumptions behind browser trust. Many bot and fraud controls look for consistency between the browser, network, session, and behaviour. Anti-detect tooling is designed to make automated or repeated activity look more like separate ordinary browser sessions.

This article is not a guide to using those tools. The defensive question is simpler: when a browser tries to look ordinary, what can still be observed safely, and how should that evidence affect a request decision?

Why They Create Risk

Anti-detect browsers are often discussed as a browser-fingerprinting problem, but the risk is wider than that. The same traffic may also involve residential proxies, credential lists, automation, and API requests that never run browser-side checks. A login attempt, account creation flow, product scrape, checkout request, or mobile API call may look valid at the protocol level while still being part of an automated campaign.

The hard part is that some signals are genuinely ambiguous. A changed browser, a new device, a shared network, or a privacy tool does not prove abuse. A defensive system has to treat those observations as risk inputs, not as standalone verdicts.

Browser Consistency Is Evidence, Not Identity

Anti-detect tooling tries to make browser-reported attributes look internally consistent. That weakens simple checks that only ask whether the browser appears plausible. Defenders need a broader view: does the claimed browser line up with the network stack, TLS and HTTP behaviour, session history, cookie continuity, route sequence, response-code pattern, and recent account behaviour?

That does not mean a fingerprint identifies a person. Fingerprints classify software, client behaviour, and connection characteristics. They can help separate likely automation from ordinary traffic, but they need to be combined with route, account, proxy, and behavioural context. The result should be a risk classification with evidence attached, not an unexplained block.

Residential Proxies Change the Decision

Residential proxies are a common companion signal because they make requests appear to come from consumer networks. That creates a false-positive problem. Real customers also use shared residential, mobile, office, carrier-grade NAT, and public Wi-Fi networks. Blocking every suspicious or shared source would damage legitimate traffic.

The safer approach is to use residential proxy detection as one input in the decision. A proxy signal on a public content page may be logged. The same signal on repeated login failures, account creation, checkout abuse, or sensitive APIs may justify a challenge, rate limit, or block. Context changes the action.

The API Gap

Browser-side checks are weakest where the browser is not present. Mobile apps, partner integrations, token routes, and direct API clients may not expose the same JavaScript or browser evidence that a web page does. Attackers do not need a convincing browser if the target workflow accepts valid-looking API requests.

That is why anti-detect risk belongs in the wider application security model. API routes need method, schema, authentication, token, request cadence, response-code, account, and bot context. If the only signal available is an IP address, the decision will be too blunt.

Observable Signals Defenders Can Use

The useful evidence is usually the mismatch between what the request claims to be and how it behaves over time. A browser may look plausible on one request, but the wider pattern can still show automation: repeated attempts across accounts, route sequences that normal users do not follow, cache-miss pressure on expensive pages, unusual response-code loops, or browser and network characteristics that drift in ways ordinary clients rarely do.

Bot Management works best when it combines these signals rather than chasing a single magic detector. IP intelligence, proxy classification, network and browser fingerprints, route-aware rates, API state, WAF findings, and behaviour should all feed the same action vocabulary: allow, challenge, rate limit, block, log, or review.

Safer Defensive Response

The defensive response should be proportionate. High-confidence exploit traffic can be blocked quickly. Uncertain browser or proxy evidence may be better challenged, rate limited, or logged until the pattern is clearer. Sensitive routes should have tighter policy than public content. Account-impacting actions should preserve enough evidence for review.

This is especially important for support teams. If a real customer is challenged or blocked, operators need to see which signal drove the action and which route was involved. Without that record, anti-bot policy becomes a black box.

Final Thoughts

Anti-detect browsers are a practical problem because they reduce the value of simple browser checks. They do not make traffic invisible. They leave request-path evidence in network behaviour, route sequences, account activity, API usage, proxy signals, and response patterns.

The right goal is not to identify a person from a fingerprint or to block every unusual browser. The goal is to classify risk with enough context to choose a safe action at the edge, then keep the evidence available for tuning and review.

The Hidden Cost of Click Fraud

2025-01-14T13:00:00+11:00

Marketing organisations are losing money to automated clicks and fake impressions. These attacks drain advertising budgets and corrupt the data CMOs rely on for strategic decisions. The lost money cannot be recovered, but understanding the scale and mechanics of click fraud helps marketing teams protect future investment and optimise campaigns.

The Scale of Click Fraud

Click fraud now consumes 40% of digital advertising budgets through fake clicks and impressions that never reach real customers. It affects every digital marketing channel, from pay-per-click and display advertising to social media campaigns, retargeting, and video advertising. The damage goes beyond direct financial loss, because it also corrupts the metrics teams use for decision-making.

Our research on bot traffic shows the percentage of fraudulent clicks continues to rise each quarter. Marketing teams that ignore this threat base their strategies on flawed data, which leads to misallocated resources and weaker campaign performance.

How Bots Generate Fake Clicks

Automated bots generate clicks and impressions at scale across digital advertising platforms. These programs target competitor advertisements to drain marketing budgets through fake clicks. They create artificial impressions that inflate metrics and send false engagement signals. Bots also manipulate bidding algorithms and skew attribution data, leading to misallocated advertising resources.

Modern bots use more advanced techniques to evade standard security controls. They mimic human behaviour patterns and rotate through different IP addresses to avoid detection and blocking.

The Residential Proxy Challenge

Residential proxies create a significant obstacle for click fraud detection systems. These proxy services route bot traffic through IP addresses assigned to real consumers' homes and devices, making fraudulent traffic look legitimate to traditional anti-bot tools.

Residential proxy networks build their IP pools through multiple channels. They partner with consumer VPN services, distribute browser extensions, embed code in mobile applications, and in some cases exploit compromised devices. This mix gives proxy operators access to millions of residential IP addresses.

Traditional IP reputation services fail to identify this proxy traffic. Our research demonstrates these services miss up to 96% of residential proxy traffic, leaving advertising campaigns exposed to fraud through these channels.

Impact on Marketing Strategy

Click fraud undermines three core areas of marketing decision-making. First, it distorts campaign performance metrics through false click-through rates and inflated impression counts. The fraud creates skewed conversion data and engagement metrics that mask true campaign performance.

In budget allocation, click fraud wastes marketing spend on non-existent users while reducing campaign ROI. Artificially inflated acquisition costs lead marketing teams to misallocate resources across channels and campaigns.

Strategic planning suffers when data is contaminated across multiple dimensions. A/B testing results become invalid when bots generate fake interactions. Geographic and demographic data lose accuracy due to proxy traffic. Competitive intelligence becomes unreliable as bot activity masks true market dynamics.

Marketing teams that base decisions on corrupted data take on significant risk. Their optimisation efforts target bot behaviour instead of real users. Campaign budgets flow to channels dominated by fraud. Strategic initiatives fail because decisions are based on artificial metrics rather than genuine customer behaviour.

Protecting Your Marketing Investment

Lost money from click fraud cannot be recovered, so marketing teams need protection measures for future investment. Detection forms the first line of defence through continuous monitoring of traffic patterns and IP reputation analysis. Teams track user behaviour to identify suspicious patterns that indicate fraud.

Prevention requires a multi-layered security approach. Marketing teams need systems that block known bot networks and detect residential proxies attempting to generate fake clicks. These controls validate real user traffic and filter out fraudulent clicks before they affect campaigns.

Campaign optimisation becomes more useful once fraud protection is in place. Teams can adjust targeting parameters based on genuine user data and reallocate budgets to channels with verified traffic. This supports updates to bidding strategies and refinement of audience segments based on real engagement.

Our Ad Fraud Protection solution protects marketing investment by blocking bot traffic, detecting residential proxies, and validating real users. This helps ensure ad spend reaches genuine customers rather than fraudulent clicks.

Making Informed Decisions

Understanding click fraud changes how marketing teams analyse data and plan campaigns. Data analysis starts with identifying corrupted metrics in campaign reports. Teams must filter bot traffic from their analytics to measure real user engagement. This enables tracking of true campaign performance based on human interactions.

Budget planning improves once teams understand the scale of click fraud. Marketing teams can allocate resources to channels with verified human traffic. This focus on real users optimises campaign spend and improves return on investment across marketing initiatives.

Strategy development depends on clean, accurate data. Teams make decisions based on genuine user behaviour rather than bot interactions. Campaign planning targets real audience segments with messages that resonate. Performance measurement reflects actual results rather than artificial engagement.

Taking Action

Marketing teams need protection measures across three key areas to secure their investments. First, bot protection forms the foundation through deployment of bot management systems. These systems block automated traffic while validating real users and monitoring for suspicious patterns.

The second protection layer focuses on residential proxy detection. Teams implement proxy detection to identify and block proxy networks. This helps ensure traffic comes from real IP addresses and prevents fraud through residential proxies.

The third component centres on protecting ad spend through traffic monitoring. Teams implement systems to block fraudulent clicks and validate impressions. This enables tracking of real engagement from genuine users.

Our Traffic Control solution combines these protection measures to help marketing teams secure their investments and base decisions on real user data.

Conclusion

Click fraud threatens marketing budgets and corrupts campaign data. Lost money cannot be recovered, but understanding and preventing fraud helps marketing teams protect future investment and make better decisions.

Residential Proxies - The Growing Threat to Ad Campaigns

2024-12-30T00:00:00+11:00

Digital advertising fraud costs organisations $42 billion annually through fake clicks and fake impressions. The growth of residential proxy networks has changed how this fraud reaches campaigns: bot traffic can now hide behind legitimate residential IP addresses, putting it outside the reach of many traditional checks.

Hiding in the crowd

Residential proxies make bad traffic harder to separate from real visitors. Unlike data centre IPs that traditional tools can often detect, residential proxies hide behind real households' internet connections. This means the traffic appears to come from genuine users in your target market. When a residential proxy network operates from Sydney suburbs to attack an Australian campaign, existing protection systems can be fooled into treating it as authentic local traffic.

The impact extends beyond direct financial losses. Your analytics may show engagement from what appears to be your target demographic, while the activity is bot traffic masquerading as potential customers. This contaminated data can push marketing strategy in the wrong direction and waste retargeting spend. Competitors can also use fake clicks to drain your budget while gathering intelligence on your campaigns.

Bad data then compounds the spend problem. Once bots are counted as engaged prospects, reporting and optimisation start from the wrong signal. The result is not only wasted media spend, but poorer decisions built on traffic that should never have been treated as customer intent.

A growing threat

The residential proxy industry continues to expand. Services now offer millions of residential IPs with precise geographic targeting capabilities. They rotate IPs automatically and match real browser fingerprints. Without specialised detection methods, the traffic can become indistinguishable from genuine users.

This is a budget problem, not just a technical one. Each day without protection means 30-40% of your ad budget feeds bot networks instead of reaching customers. The corrupted analytics drive decisions that compound these losses. As residential proxy services grow more sophisticated, basic controls fall further behind.

Traditional IP reputation and rate limiting fail against this distributed threat because the IP addresses are not obviously suspicious. Protection requires advanced network fingerprinting that looks beyond IP addresses. Peakhour's Ad Fraud Protection analyses subtle patterns in how residential proxies connect and behave, and detects the signs of proxy traffic that other solutions miss.

Knowledge is power

Peakhour integrates this protection with your existing ad platforms to stop fraud before it affects your campaigns. Our customers have reduced wasted ad spend by 35% while improving campaign performance through cleaner analytics. The system adapts as threat techniques change, so detection keeps pace with new residential proxy methods.

Residential proxies have changed ad fraud because traffic that appears local and legitimate may mask sophisticated bot networks. Protecting your campaigns requires detection that goes beyond IP addresses and treats residential proxy behaviour as its own signal. Contact us to learn how we can help secure your ad spend against residential proxy networks.

Next-Generation Application Security Defence Strategies

2024-11-15T14:00:00+11:00

As I look at recent cyber threat activity, the pattern is clear enough: AI is no longer only a defensive tool. Attackers are using it to probe, adapt, and automate application-layer attacks.

One recent incident made that plain. Our threat detection systems identified a series of probes against a client's infrastructure. These were not the typical brute-force attempts we are used to blocking. The attack patterns changed in real time, adapted to our defences, and probed for weaknesses in a way that pointed to AI-driven automation.

The individual attempts were not the main concern. What mattered was how the attack system learned and adjusted its approach. When we blocked one vector, it shifted to another. When we implemented rate limiting, it distributed its attempts through residential proxies. The attack showed a common trait of AI systems: rapid iteration and learning from failure.

This change in attack methodology puts real pressure on the traditional security model. Static defences, including controls that looked strong only months ago, are easier to route around. They might stop obvious threats, but more capable AI-powered attacks can keep testing the edges until they find a path.

The threat landscape has shifted in three practical ways. First, AI enables attacks to adapt and evolve in real time. Second, residential proxies give attackers a distributed network of IP addresses that appear legitimate, making traffic origin verification much harder. Third, AI can analyse and mimic legitimate user behaviour patterns closely enough to bypass traditional bot detection.

These changes require a change in defence strategy. Identifying and blocking known attack patterns still matters, but it is no longer enough on its own. We need systems that can anticipate and adapt to new threats as quickly as they emerge.

In our security operations, we've begun implementing what we call "contextual defence dynamics." The approach moves beyond simple pattern matching to analyse the intent and behaviour behind each request. We examine not just what a request does, but how it fits into broader patterns of behaviour and what it might indicate about the attacker's objectives.

That approach has already proved useful. When we implemented contextual defence dynamics for a major e-commerce client, we identified and blocked an AI-powered credential stuffing attack that had evaded traditional detection methods for weeks. The attack used residential proxies to distribute its attempts and mimicked human behaviour patterns, but our system identified subtle anomalies in its timing and response patterns.

That case highlighted a useful point: while AI-powered attacks grow more sophisticated, they still exhibit patterns. Those patterns may not appear in individual actions, but they do appear in broader behaviour and objectives. By shifting our focus from blocking specific actions to understanding and responding to these broader patterns, we can maintain effective defences even against evolving threats.

This approach requires a different way of thinking about security. We must move from a model of static defences to one of dynamic response. Our security systems must learn and adapt as quickly as the threats they face. This means implementing machine learning systems that can identify new attack patterns, updating defence strategies in real time, and maintaining awareness of emerging threat vectors.

The implications extend beyond technical implementation. Organisations need to treat security budgets and strategies as ongoing commitments, not one-off purchases. The era of "set and forget" security solutions has ended. Continuous adaptation and review now sit at the centre of effective defence.

I expect this arms race to keep accelerating. AI will continue to enhance both attack and defence capabilities. The organisations that maintain strong security will be those that accept this dynamic and build their defences around continuous adaptation.

For security professionals, this means developing new skills and approaches. We must understand not just the technical aspects of security, but the patterns of attack and defence that emerge in AI-driven systems. We must build systems that can learn and adapt, and we must be prepared to change strategy as the threat landscape evolves.

The security arms race has entered a new phase. The advantage will not sit with the strongest static defences alone, but with teams that can adapt and evolve their protection strategies in real time. The focus must shift from building walls to creating intelligent, adaptive defence systems that can match the sophistication of AI-powered threats.

This shift in security thinking is practical, not theoretical. The threats we face are becoming more capable, and defensive tooling is improving as well. The important step is recognising the change and adapting how we build, operate, and review application security controls.

Managing Bots For Application Security

2024-09-15T00:00:00+10:00

This guide separates bot management into three maturity levels: basic, intermediate, and advanced. The point is not to rank feature lists. It is to understand what kind of bot decision each level can safely make on the request path.

Bots now target revenue, data, accounts, inventory, APIs, and origin capacity. Some are obvious crawlers. Others run credential stuffing, account creation, scraping, inventory hoarding, click fraud, or Layer 7 pressure through traffic that looks close to normal. A useful bot control has to decide whether to allow, challenge, rate limit, block, log, or review a request without punishing legitimate users who happen to share a network or device pattern.

For more on account impact, read our article on the Business Impact of Credential Stuffing.

Basic Protection

Basic bot management is built around visible signals: user-agent checks, simple IP reputation, known bot signatures, and broad rate limits. These controls still have a job. They can manage well-behaved crawlers, block obvious automation, and stop a single noisy source from hammering a site.

The limitation is that basic controls assume the source or header tells most of the story. That breaks down when automation behaves like a browser, rotates infrastructure, or spreads requests across residential and shared networks. A flat IP limit may slow one scraper and still miss a credential attack distributed across many exits. It may also catch real users behind a busy office, mobile carrier, or public Wi-Fi network.

Basic protection is suitable when the risk is low, the main concern is crawler hygiene, and the business can tolerate coarse controls. It is not enough for account protection, high-value scraping targets, or API workflows where abuse can arrive through valid requests.

Intermediate Protection

Intermediate bot management adds more request and client evidence. JavaScript signals, header consistency, cookie behaviour, basic network fingerprints such as TLS fingerprinting, and route-level observations make the decision less dependent on the IP address alone. This level can catch automation that fails to behave like a normal browser or that exposes inconsistencies across requests.

It is a meaningful step up, but it still has limits. Anti-detect browsers and modern automation can keep browser signals plausible for long enough to run a campaign. Residential proxy networks can make source reputation noisy. API requests may not run browser-side checks at all. If the bot system treats browser, route, credential, and API context as separate problems, operators end up tuning several partial controls rather than one decision.

Intermediate protection works for general scraping, noisy automation, and non-persistent abuse. It starts to struggle when attackers adapt, slow down, distribute requests, or target sensitive routes where a small number of requests can cause business harm.

Advanced Protection

Advanced bot management is combined signal decisioning. The difference is not "more techniques" in a checklist. The difference is that IP intelligence, residential proxy status, network and browser fingerprints, route-specific rates, behaviour, credential risk, API context, WAF/WAAP findings, DDoS pressure, and logs feed the same action model.

That context changes the decision. A high request rate on a public image route is not the same as repeated failed logins. A suspicious proxy signal on a cached page is not the same as the same signal on account recovery. A browser fingerprint mismatch may be logged on a low-risk page but challenged when paired with exposed credentials and rapid account switching.

Advanced protection is designed for persistent abuse: credential stuffing, account takeover attempts, scraping at scale, inventory hoarding, fake account creation, API bot traffic, and bot-driven Layer 7 floods. It should support web, mobile, and API traffic, and it should preserve evidence so security, platform, and support teams can see which signal drove an action.

Choosing the Right Level

The right level depends on what the bot can damage. A brochure site may only need crawler management and basic rate limits. An ecommerce site needs protection for search, product, checkout, promotion, and account routes. A marketplace, bank, gaming platform, ticketing site, or API-heavy business usually needs route-aware decisions that combine proxy, fingerprint, credential, account, and behaviour context.

The false-positive risk matters just as much as the attack risk. Shared networks, carrier-grade NAT, privacy tools, corporate egress, and normal browser drift can all make a simple signal look suspicious. A mature bot programme does not block every unusual request. It uses uncertainty to pick safer actions: log, challenge, rate limit, or review before escalating to a block.

Peakhour's View

Peakhour's Bot Management connects bot decisions to the rest of the application security path. Residential Proxy Detection, IP Intelligence, Advanced Rate Limiting, API Security, WAAP/WAF controls, and Log Forwarding all support the same request outcome: allow, challenge, rate limit, block, log, or review with evidence.

That is the practical maturity model. Basic controls handle obvious bots. Intermediate controls add client and request evidence. Advanced controls combine signals into decisions that match the route, risk, and business impact.

The Challenge of Proxy Detection

2024-07-19T10:00:00+10:00

Our recent survey found that only 15% of Australian organisations use residential proxy detection. That leaves many teams relying on controls that were not built for current proxy traffic, especially where CGNAT and NAT make IP-level decisions unreliable.

The Shortcomings of Traditional Methods

Legacy bot protection providers often combine IP reputation, network characteristics, header analysis, and JavaScript-based checks to identify proxy usage. These methods struggle against well-run residential proxies:

IP and ASN categorisation: Ages quickly as new proxy networks emerge.
Network-level checks: Well-configured proxies can work around them.
Header analysis: Proxies can alter HTTP headers to mimic legitimate traffic.
JavaScript-based detection: Struggles against headless browsers and leaves API endpoints vulnerable.

The CGNAT and NAT Challenge

A practical limit of traditional methods is their inability to distinguish legitimate traffic from proxy traffic when both originate from the same IP address. Carrier-Grade NAT (CGNAT) and Network Address Translation (NAT) make this common:

CGNAT: Used by ISPs to conserve IPv4 addresses, resulting in multiple users sharing a single public IP.
NAT: Commonly used in home and business networks, allowing multiple devices to use one public IP address.

As a result, legitimate users and residential proxy traffic can appear to come from the same IP address. IP reputation and geolocation alone cannot separate these traffic types.

This creates a difficult tradeoff:

Blocking suspicious IPs risks denying service to legitimate users.
Allowing all traffic from these IPs opens the door to potential abuse via residential proxies.

Traditional methods cannot reliably pull apart these different types of traffic, so teams either block too much legitimate traffic or allow too much proxy traffic through.

The Need for Sophisticated Network Fingerprinting

To detect and mitigate residential proxy threats while allowing legitimate traffic from shared IPs, detection needs to move beyond IP identity. Network fingerprinting addresses the limits of traditional methods:

Deep packet inspection: Analyses traffic patterns and characteristics beyond basic IP or header indicators.
Protocol behaviour analysis: Identifies subtle anomalies in how network protocols are implemented across the proxy chain.
TLS fingerprinting: Examines unique characteristics of TLS handshakes to detect proxy usage.
Timing analysis: Measures small differences in network latency that can indicate the presence of a proxy.

Used together, these techniques can detect proxy usage on a per-connection basis for both web traffic and API calls, even when traffic originates from shared IP addresses. This approach provides several advantages:

Improved accuracy: Significantly reduces false positives and negatives compared to traditional methods, including in CGNAT and NAT scenarios.
API protection: Secures API endpoints, which are often overlooked by JavaScript-based solutions.
Real-time detection: Allows for immediate action against detected proxy usage without impacting legitimate users.
Adaptability: Can be updated to detect new proxy technologies as they emerge, regardless of IP sharing.

Implementing Effective Proxy Detection

To implement proxy detection that accounts for modern network complexity, organisations should consider the following:

Deploy solutions that use network fingerprinting techniques capable of distinguishing between different types of traffic from the same IP.
Ensure protection covers both web applications and API endpoints, as both are vulnerable to proxy-based attacks.
Implement real-time mitigation capabilities to respond swiftly to detected threats without impacting legitimate users.
Regularly update and tune detection algorithms to keep pace with evolving proxy technologies and network architectures.

Together, these practices improve an organisation's ability to detect and mitigate residential proxy threats across credential stuffing, account takeover, and related activity, while keeping access available for legitimate users.

Learn more about our proxy detection solution, which uses network fingerprinting to address the challenges posed by CGNAT and NAT.

For more detail, explore our learning resources:

As proxy technologies and network architectures change, detection and mitigation need to change with them. Network fingerprinting gives organisations a more reliable way to identify residential proxy abuse without treating every shared IP as suspicious.

Quantifying The Residential Proxy Threat

2024-07-18T10:00:00+10:00

Our 2024 survey found that only 15% of Australian businesses use residential proxy detection. That leaves a measurable blind spot in many security programmes: traffic routed through real consumer connections is harder to separate from legitimate users. This article looks at why residential proxy detection is difficult and how to quantify the risk before choosing controls.

Understanding the Residential Proxy Threat Landscape

Residential proxies use IP addresses assigned to residential internet connections, so malicious traffic can look legitimate. This weakens controls built around IP reputation, GeoIP, and simple request thresholds, and creates a specific detection problem for security teams.

The effectiveness of residential proxies stems from their ability to:

Use legitimate IP addresses, often from unsuspecting users
Bypass IP-based rate limiting and traditional bot detection methods
Evade geolocation restrictions, making GeoIP filtering less reliable
Support large-scale attacks without triggering typical alarm thresholds
Mimic legitimate user behaviour, which makes detection more difficult

These capabilities make residential proxies useful infrastructure for credential stuffing, data scraping, and attempts to bypass fraud detection systems. Because the traffic is distributed across many residential connections, attacks can stay below the thresholds that conventional controls rely on.

Limitations of Conventional Security Approaches

Conventional controls have clear gaps when they are applied to residential proxy traffic:

IP-based detection misses constantly changing, legitimate-appearing IP addresses.
GeoIP filtering becomes less useful against globally distributed residential IPs.
User agent analysis struggles because proxies can mimic legitimate browsers.
Standard rate limiting falters when attacks appear to originate from many unique IPs.
Behavioural analysis based on known bot patterns may miss more careful proxy-based attacks.

These limitations point to a practical requirement: security teams need controls that assess context, not just static request attributes. Residential proxies make simple rule-based decisions less reliable, especially when attacks are distributed and deliberately low-noise.

Quantifying the Risk

To make a sensible decision about residential proxy controls, organisations need to quantify the risk. This involves:

Assessing the potential financial impact of successful attacks via residential proxies
Evaluating the likelihood of such attacks based on industry trends and organisational attractiveness to attackers
Determining the effectiveness of current security measures against this specific threat
Calculating the return on investment for implementing advanced detection and mitigation strategies

Risk quantification gives businesses a clearer basis for investing in residential proxy detection. It aligns security spending with actual threat levels and potential impacts, rather than broad concern or industry pressure alone.

Reframing Security

The challenge of residential proxy detection is less about one new control and more about how signals are combined. A useful approach includes:

Contextual Analysis: Analyse the full context of each request, not just its origin. This includes examining patterns of behaviour across multiple sessions and users.
Continuous Monitoring and Adaptation: Use real-time monitoring systems that can detect subtle patterns indicative of proxy use. These systems should continuously adapt to new attack vectors.
Risk-Based Authentication: Use dynamic authentication mechanisms that adjust based on the assessed risk of each session or transaction.
Holistic Data Analysis: Correlate data from multiple sources - including login attempts, transaction patterns, and user behaviour - to identify anomalies that may indicate proxy use.
Proactive Threat Hunting: Actively search for indicators of residential proxy use within your network and user base, rather than waiting for attacks to trigger alerts.

This approach moves beyond simple allow/block decisions and gives teams a better view of user and network behaviour.

Implementing Advanced Detection Strategies

Residential proxy threats need detection that looks beyond the source IP:

Machine Learning-Based Behavioural Analysis: Use AI and machine learning to identify patterns consistent with proxy use, even when individual actions appear legitimate.
Device Fingerprinting Beyond IP: Use advanced fingerprinting techniques that identify individual devices based on a combination of factors, making it harder for proxies to mimic legitimate users.
Network Traffic Analysis: Analyse network behaviour at a granular level to identify patterns consistent with proxy network traffic.
Adaptive Challenge Mechanisms: Deploy targeted challenges based on risk assessment, without disrupting legitimate user experiences.
Cross-Organisational Data Sharing: Participate in threat intelligence sharing networks to gain broader insights into residential proxy activities and emerging attack patterns.

When used as part of the broader security stack, these strategies improve defence against residential proxy threats.

Elevating Security Through Risk Quantification

Residential proxies are not only a technical detection problem. They change the risk model for web applications because attacker traffic can borrow the appearance of ordinary residential users. By adopting a risk quantification approach and implementing advanced detection strategies, organisations can:

Align security investments with actual threat levels
Improve detection of sophisticated, proxy-based attacks
Strengthen overall security posture against evolving threats
Make data-driven decisions about security priorities and resource allocation

Organisations that handle this well will be able to quantify their risk, adapt their security strategies, and implement intelligent detection mechanisms. The goal is practical: identify, analyse, and mitigate sophisticated threats before they cause material damage.

Effective protection starts with understanding the risk well enough to measure it.

2024 Survey Insights

2024-07-16T10:00:00+10:00

Our recent survey of Australian CISOs and CTOs looked at account protection controls, planned security measures, and how teams are responding to credential stuffing and residential proxies. Key findings:

Multi-Factor Authentication (MFA) Adoption: 76.23% of Australian businesses use MFA, showing broad adoption of a baseline account security control.
Bot Protection: Currently implemented by 39.34% of organisations, with an additional 34.65% planning to adopt it.
Bot Management Solutions: Cloudflare is the most common bot management provider in the survey, used by 48.24% of respondents.
Residential Proxy (Resip) Detection: Only 13.11% of organisations currently use this technology, although many plan to implement it to address residential proxy traffic.
Credential Stuffing Concerns: Businesses are planning measures to reduce credential stuffing risk, including bot protection, MFA, and checking credentials against known breaches.
Mobile Security Gap: Low adoption of Web Application and API Protection (WAAP) suggests gaps in mobile application security.
Executive vs. Engineer Priorities: The survey showed different cybersecurity priorities between executives and engineers.

These findings point to the need for account protection strategies that go beyond MFA and address automated traffic, breached credentials, and residential proxies.

2024 Survey Insights

2024-07-16T10:00:00+10:00

Recent customer account takeovers have put account protection back on the agenda for Australian businesses. Our 2024 survey of Australian CISOs and CTOs shows how respondents are using MFA, bot protection, WAAP and residential proxy detection to manage credential stuffing and account takeover risk.

Account Protection: Current State and Future Plans

Our survey found 76.23% of Australian businesses use Multi-Factor Authentication (MFA). MFA is widely adopted, but it is not a complete account protection strategy on its own.

39.34% of organisations currently use bot protection. That matters because credential stuffing is automated by design. Another 34.65% of businesses plan to implement bot protection in the future.

The pattern is clear: many organisations are treating MFA as a baseline and looking at additional controls around it.

Current Bot Management Solutions

The survey also asked which bot management solutions Australian businesses currently use. Cloudflare was the clear leader, with nearly half of respondents using its services.

The breakdown of bot management solutions is as follows:

Cloudflare: 48.24%
AWS WAF Bot Ruleset: 10.59%
Other solutions make up the remaining percentage

This distribution is concentrated around Cloudflare. Outside that, the remaining respondents are spread across other solutions rather than one clear alternative.

Tooling matters here. Residential proxy traffic weakens IP reputation and simple rate limits, so detection capability, request grouping and response controls matter as much as vendor name. If residential proxies continue to feature in credential stuffing tooling, this mix may shift as teams look for more advanced protection measures.

The Rising Threat of Residential Proxies

A key finding from our survey is the low adoption rate of residential proxy (resip) detection, with only 13.11% of organisations currently using this technology. Planned adoption suggests teams are starting to account for the risk, but current coverage is still low.

Resips are difficult for account security teams because malicious traffic can look like normal ISP traffic. They enable attackers to:

Bypass traditional IP-based rate limiting
Evade geolocation-based restrictions
Conduct large-scale credential stuffing attacks
Scrape sensitive data undetected

The planned adoption of resip detection points to a shift in security strategies, away from simple IP-based controls and towards more specific network signals.

Learn more about the threat of residential proxies and how to detect them

Credential Stuffing: A Persistent and Growing Concern

Credential stuffing attacks continue to be a major concern for businesses. These attacks exploit password reuse across multiple sites, allowing attackers to gain unauthorised access to user accounts.

Respondents said they plan to implement several measures to reduce credential stuffing risk:

34.65% plan to implement bot protection
32.67% intend to add multi-factor authentication
31.68% aim to check credentials against known breaches

These plans point to layered account protection rather than reliance on one control.

Mobile Applications: An Emerging Attack Surface

While mobile applications were not directly addressed in our survey, the data suggests a possible gap in mobile security strategies. The low adoption rate of Web Application and API Protection (WAAP) - implemented by only 27.87% of respondents - indicates many businesses may be underprepared to protect their mobile assets.

As mobile apps become primary interfaces for critical operations, this gap leaves businesses exposed to attacks that use the same automation and resip infrastructure seen on web login flows.

Balancing Security and User Experience

The operational problem is familiar: increase assurance without making login unusable. Key considerations for enhancing account protection while preserving usability include:

Expanding beyond MFA
Implementing bot protection
Adopting WAAP solutions
Monitoring credential leaks
Focusing on API security
Implementing residential proxy detection

Explore strategies for balancing security and user experience

Executive vs Engineer Perspectives

Our survey found differences in cybersecurity priorities between executives and engineers:

Figure 3: Comparison of cybersecurity priorities between executives and engineers

The gap matters because budget, architecture, and incident response are often owned by different teams. Account protection plans need to cover both executive risk concerns and engineering realities, including the threat from RESIPs.

Final Thoughts

Our 2024 survey results point to a simple position: MFA is widely used, but it is not the whole account protection strategy. Bot protection, breached credential checks, WAAP and residential proxy detection are still unevenly adopted. That matters because credential stuffing does not depend on one weakness; it combines reused credentials, automation, proxy networks and weak response controls.

Australian businesses do not need every control at once, but they need a layered plan that reflects how account takeover attacks are run now. For teams reviewing their controls, resip detection and mobile/API coverage are worth checking explicitly because both are easy to miss if the programme is still centred on MFA and IP reputation.

Application Security Beyond MFA

2024-07-15T10:00:00+10:00

Multi-factor authentication (MFA) remains a useful defence against account takeovers, but it is not a complete control. Attackers increasingly work around MFA with social engineering, automation, and infrastructure that makes malicious traffic look ordinary.

MFA answers one narrow question: can the user present a second factor at this point in the flow? That is valuable. It does not prove the password was safe, the session will remain safe, the device is trusted, or the person entering the code has not been manipulated. Account protection needs to cover the request path before MFA, around MFA, and after MFA.

OTP Bots Target the Human, Not the Cryptography

A Kaspersky article describes the rise of OTP bots: tools that call or message users and convince them to hand over one-time passwords. The attacker does not need to break the MFA system. They need the victim to read out a fresh code at the same moment the attacker is logging in.

The usual flow is simple. The attacker obtains a working username and password from a breach, phishing kit, or credential stuffing result. They attempt a login, which triggers an OTP. The victim receives a call or message claiming to be from the bank, retailer, courier, or support team. The story is urgent enough to make the code feel like part of protecting the account, not compromising it.

AI phone assistants such as Lucy are built for legitimate business use, but similar conversational technology lowers the effort required to run more convincing criminal call flows. The security issue is not that AI magically defeats MFA. It is that a fluent, responsive call can make social engineering less scripted and harder for a user to dismiss.

This is why "we have MFA" should not end the account protection conversation. MFA can stop many stolen-password logins, but it cannot reliably stop a user from being tricked in real time.

Residential Proxies Weaken the Surrounding Checks

Attackers also work to make the login itself look unremarkable. Residential proxies route traffic through IP addresses assigned to ordinary home or mobile internet connections. That lets malicious traffic borrow the appearance of normal customer traffic.

Traditional controls often lean too heavily on IP address, geolocation, and request volume. Residential proxy networks weaken all three. An attacker can rotate through many IPs, keep each source below a simple rate limit, and choose an exit location that roughly matches the victim's country or city. If the login looks local enough, the MFA challenge may be the only control left.

That is a poor place to put all the risk. A login with a correct password, a plausible IP address, and a successful OTP can still be an account takeover. The system needs to keep evaluating the request: device and browser signals, network fingerprint, known breached credentials, velocity across accounts, and behaviour after login.

Automation Happens Before and After MFA

MFA is usually visible at the point of login, but account takeover campaigns are broader than one prompt. Bots test credential pairs across login forms and APIs. Tools such as OpenBullet and similar automation frameworks can replay login flows at scale. Breached credential lists give attackers a cheap starting point because password reuse remains common.

Once an attacker gets through, the next actions matter. They may change the email address, add a device, disable notifications, alter delivery details, use stored payment methods, transfer value, or test what the account can access. If monitoring treats a successful MFA as the end of risk, those actions can happen inside a trusted session.

The defence needs to be layered around the actual attack path:

Check credential risk before and during login, especially known breached username and password pairs.
Use bot and browser signals to detect automation even when traffic is distributed.
Rate limit on better keys than IP alone, such as TLS or HTTP/2 fingerprints, headers, routes, ASNs, countries, and account behaviour.
Treat residential proxy evidence as a risk input, not just an allow-or-block label.
Monitor session and account changes after MFA, then challenge, hold, revoke, or review when behaviour changes.

This does not mean every login needs more friction. It means the system should have more choices than "ask for MFA" or "allow". A low-risk login from a known device can keep moving. A login using breached credentials through proxy infrastructure can be slowed, challenged, or blocked before the user receives a confusing call. A successful login followed by high-risk account changes can trigger fresh verification or session invalidation.

Controls Around MFA

Peakhour's Advanced Rate Limiting helps reduce reliance on IP address by grouping and limiting requests using signals such as HTTP/2 and TLS fingerprints, ASNs, countries, request headers, and route context. That matters when credential stuffing is spread across residential proxies.

Peakhour's Bot Management adds another layer by looking for automation, browser inconsistency, suspicious device patterns, and residential proxy use. The aim is to identify the machinery behind the attack before it becomes a clean-looking login attempt.

Peakhour's Account Protection brings those signals closer to the account decision. Breached credential checks, bot evidence, rate limits, proxy context, custom rules, and monitoring should all feed the decision to allow, challenge, rate limit, block, log, or review.

User education still has a place, especially around OTP sharing and unexpected calls. It should not be the main control. Users are asked to make security decisions at bad moments, often under pressure, with limited context. Technical controls should reduce the number of times an attacker can create that moment.

MFA Still Belongs in the Stack

The point is not to remove MFA. Strong MFA, especially phishing-resistant methods, raises the cost of account takeover and should remain part of the stack. The mistake is treating MFA as proof that the account is safe.

Account protection works better when MFA is one decision point inside a wider system. The login attempt, credential history, network path, device, session, account changes, and transaction behaviour all carry evidence. MFA is useful evidence. It is not the whole case.

How To Exclude Query String Parameters from Search Engine crawling

2024-05-21T13:00:00+10:00

Last year we wrote about the problem of excessive crawling from search engine spiders. Search engines such as Google and Bing aim to index as much content as possible. For ecommerce sites, this often means indexing pages with query string parameters used for sorting, filtering, or pagination. Those parameters help users navigate the site, but they can cause a few predictable crawler problems:

Over-Crawling: Search engines may spend too much time crawling similar pages with different parameters, wasting crawl budget.
Duplicate Content: Pages with different parameters can be treated as duplicate content, weakening SEO performance.
Server Load: Excessive crawling can increase server load, slow down your site, and affect user experience. Search engines typically account for 30-50% of page requests to an ecommerce store. Managing their crawling effectively can have a material effect on site speed and server spend.

Another common cause of over crawling is internal searches being indexed.

In our previous article we mentioned using the webmaster tools provided by Google and Microsoft to manage crawler behaviour by adding ignored parameters. Since then, both tools have been updated and no longer allow you to add parameters to ignore during a crawl.

Differences in Crawling and Indexing

Search engines maintain an 'index' of web pages. Pages in this index are what appear in search results. To maintain the index, the search engine crawls a website to 'discover' new content and keep existing entries up to date. Webmasters can control what gets indexed with tags or headers in their web pages. These include:

Canonical Tags can be used to indicate the preferred version of a page. This helps consolidate link 'juice' and tell the search engine which URL to index.
Noindex tags can be used to prevent specific pages from being indexed. This is useful for thank you pages, admin pages or any content you don't want to appear in search results.
Nofollow links can be used to indicate to a search engine not to pass on SEO value to the linked page.

However, controlling what does or does not get indexed does not prevent content from being crawled. The only way to do that is via the robots.txt file. You may be familiar with the Disallow directive in the robots.txt file, but you can also use wildcards to prevent crawling of url parameters.

An example...

Consider an ecommerce store that has a category page which can then be customised with the following parameters:

    orderBy
    colors
    brands
    page
    results

These may appear in any order, and the combinations can result in 100s or even 1000s of variations of essentially the same page. Google is fairly smart when presented with this scenario, but Bing.... Bing can crawl very aggressively and it likes to try everything. In our example above, we may want to stop crawling everything except the page number, in which case an effective way to control crawler behaviour would be:

    User-agent: *
    Disallow: /*?*orderBy=*
    Disallow: /*?*colors=*
    Disallow: /*?*brands=*
    Disallow: /*?*results=*

We can't really do this in a single Disallow because the parameters might be in any order. By including the ? in the url we're ensuring that the parameter names are only in the query string, not in the main url path. This prevents crawlers from wasting crawl budget and putting unnecessary load on server resources.

Final Thoughts

Search engines can often make up 30-50% of the overall page requests to a website. Managing their behaviour helps maximise useful crawling and minimise server utilisation. Keep an eye on your access logs for unwanted behaviour, and use robots.txt where it gives you the right level of control.

Rate Limiting for API Security

2024-01-24T13:00:00+11:00

Rate limiting prevents servers from being overwhelmed by too many requests in a short period of time. Typically, rate limiting is configured using rules made up of a filter, for example a path like /login, and a limit on the number of requests a user can make in a given time, such as 10 requests in a minute. If a user exceeds this limit, they are usually blocked for a timeout period.

But how do you identify a user? Traditionally rate limiting has used the IP address for grouping requests, assuming that requests from the same IP address will be the same user. That assumption is now weak. IP addresses are rarely static and are often shared. For example, an office network might have hundreds of individual computers in it but present a single IP address for all those computers to the internet. Mobile operators commonly use carrier-grade network address translation (CGNAT) to share the same IP across thousands of devices or users. Bot networks, seeking to avoid security controls like rate limiting, will rotate their requests through thousands of different IP addresses. This makes rate limiting based on IP addresses a poor choice from both a functional and a security perspective.

Introducing Advanced Rate Limiting

Peakhour's Advanced Rate Limiting service lets you create filters using any HTTP request characteristic, for example URI, request method, headers, cookies, country, network fingerprints and more. You can also use response headers and response codes, so a rule can count failed login attempts, repeated 404s from a scraper, or traffic that crosses an API threshold.

For counting requests you can use the following fields for grouping:

IP Address
ASN
Country Code
HTTP/2 Fingerprint
TLS Fingerprint
Any combination of Request Headers

You can use one of those fields, or a combination of them, to identify users with more control than IP address alone.

You can also separate the filter and mitigation expression. For example excessive attempts to /login can be blocked on the entire site.

This matters because rate limiting is not just a request counter. In Peakhour it sits beside bot management, WAF, DDoS protection, traffic controls, and origin shielding on the same managed edge path. That gives operators a practical way to set different thresholds for verified crawlers, suspicious automation, authenticated API clients, and normal visitors without pushing every policy change into the application. It also gives them allowed, blocked, and threshold-hit evidence to tune the rule after it is deployed, whether Peakhour is the active edge or adding controls beside an existing CDN or cloud edge.

Putting it into action

Advanced Rate Limiting can help protect applications from attacks like Layer 7 DDoS, Account Takeovers, Credential Stuffing, and more. Here are some real world examples you can configure using our dashboard and API.

Protect against general site abuse

Our example website is a medium-sized ecommerce store that has page URLs ending in /. It serves Australian clients and typically sees around 100 page requests a minute from non-search-engine traffic during peak traffic times. With that baseline, we can set up rate limiting to prevent general site abuse and protect against layer 7 DDoS attacks.

Peakhour rate limiting starts with zones. You specify your request limits in these zones.

Here we've specified a maximum of 45 requests in 1 minute. We're going to apply this limit to page loads only. Since our typical maximum for all users on this website is 100 in a minute, it seems reasonable that a real user is not going to view 40 pages in 1 minute. We could also specify a value for error responses in a minute. An error could be a 404, which a scraper might typically get when looking for removed URLs.

Now let's define our filter and our counter. For our filter we mentioned that pages end in /, so we'll use that, but exclude verified bots to make sure they're not restricted when crawling the site. A verified bot is a crawler like Google or Bing, that Peakhour has verified as legitimate by using reverse DNS to confirm they are who they say they are.

Attackers, scrapers, and others looking to abuse a site will launch an attack using a particular piece of software. That piece of software will have a TLS fingerprint (like JA3) that remains the same, even as the attacker rotates their user-agent, IP address, and other characteristics, so we'll use the TLS fingerprint as our request counter.

Rate Limit authenticated API Users

It is common for APIs to require an Authorization header as part of the request to authenticate access. By grouping requests on the value of this header, we can rate limit a specific API client even if it uses multiple applications, or if its credentials are stolen.

Protecting from Account Takeovers

Account Takeover attacks have been in the news recently, with several high-profile websites being victims. Credential Stuffing and Brute Force attacks rely on attempting lots of logins to identify valid credentials. Along with lots of attempts come lots of failures. Attackers will rely on software like openbullet to carry out their attacks, using proxy networks to constantly rotate IP addresses and defeat traditional rate limiting.

The program the attacker is using will present a consistent TLS fingerprint. We can make a special rule for our login form that tracks failed login attempts by TLS Fingerprint, effectively tracking the attacker as they rotate IP address.

If the attack is low and slow, we can track failed attempts over a longer timeframe by using the response from the server when adding to our counting zone.

Final Thoughts

Advanced rate limiting is a practical response to the limits of IP-based controls. IP address rotation is the standard amongst attackers and scrapers, rendering the traditional approach obsolete. Useful protection now needs to identify the actor behind the requests, protect the origin before expensive application work is triggered, and give teams enough evidence to adjust the policy without guesswork. Counting requests against a combination of network fingerprints, request fields, response signals, and bot context is how you stop abuse from scrapers, SEO spiders, and layer 7 attackers without treating every visitor the same.

Google Chrome's "IP Protection" vs Apple Private Relay

2023-10-25T13:00:00+11:00

Google Chrome's "IP Protection" vs. Apple's iCloud Private Relay

Google and Apple are both pushing browser-level privacy features that reduce how much a website can infer from a user's IP address. Google's recent announcement of its "IP Protection" feature for Chrome follows Apple's iCloud Private Relay, but the two approaches are not the same.

Apple's iCloud Private Relay: A Closer Look

In 2021, Apple introduced iCloud Private Relay for paid iCloud+ subscribers. The feature encrypts traffic from the user's device and routes internet requests through two separate relays. The intention is to stop any single party, including Apple, from building a comprehensive user profile from IP address, location, and browsing activity.

However, this feature is specific to Apple's Safari browser. It is not a full VPN; it is a browser-centric service that protects Safari traffic on iOS, iPadOS, and macOS. The user's internet requests are routed first through an Apple server, then through a partner network like Akamai, Cloudflare, or Fastly, before reaching the intended destination. This dual-hop design means neither party has a complete view of both the user's IP address and the browsing destination.

Google's "IP Protection": Playing Catch-up?

Google's "IP Protection" for Chrome appears to be an answer to Apple's initiative. By masking users' IP addresses using proxy servers, Google aims to preserve user privacy while keeping essential web functions working. Unlike Apple's solution, which is limited to Safari, Google's feature potentially has wider application within the Chrome ecosystem.

However, Google's solution is still early, with phased implementation and limited domain application. Apple has already integrated and offered iCloud Private Relay to its users; Google is still testing its feature.

Can Apple Allow Google's Feature on Chrome?

Given the competitive nature of the technology industry, it remains uncertain whether Apple will allow Google's IP Protection feature on Chrome for Apple devices. With iCloud Private Relay already in place, Apple may see Google's feature as redundant or conflicting with its privacy objectives.

The Bigger Picture: Ad Tracking and Platform Control

Both companies present these changes as privacy improvements, but the platform context matters. Hiding IP addresses does not remove ad tracking, and privacy features can also reinforce platform control. By making privacy protections part of their own browsers and ecosystems, Google and Apple can reduce some third-party visibility while keeping users inside platforms they operate and measure.

Apple's iCloud Private Relay and Google's "IP Protection" both improve some aspects of user privacy, with different approaches and coverage. As Google plays catch-up to Apple in this area, users should understand what these features do and what they leave in place. The goal should be genuine online privacy, and as we've discussed in our article on TLS fingerprinting, network-based fingerprinting is becoming increasingly important for protecting services in this changing environment.

Google Chrome's "IP Protection" and Online Privacy

2023-10-24T13:00:00+11:00

Google plans to introduce an "IP Protection" feature in Chrome. The feature is intended to improve privacy by masking IP addresses through proxy servers. It may also affect ad tracking and who controls access to online platforms.

Understanding IP Addresses and Google's Strategy

IP addresses can let websites follow user activity across platforms. Over time, that can build detailed profiles and create real privacy concerns. Google's "IP Protection" is designed to reduce that signal by sending third-party traffic through proxies, hiding user IPs. The feature will start as optional, then focus on domains thought to track users.

At first, Google will use a dedicated proxy for its own domains. As testing continues, the system may change. Google is also considering a 2-hop proxy system for better privacy, with an outside CDN handling the second proxy.

Google wants to use proxy connection IPs to give users broad locations, not exact ones. It will test this on platforms like Gmail and AdServices, in Chrome versions 119 to 225.

VPN Growth and Other Browsers

The growth of VPN use points to demand for online privacy. VPNs, like Google's IP Protection, hide user IP addresses. Firefox and Opera have added VPN features to their browsers. Apple, known for user privacy, has worked with CDN companies on similar privacy improvements.

This change has trade-offs. Sending traffic through Google's, or others', servers can make it harder for security teams to handle threats. Google has suggested fixes like checking users with the proxy and rate-limiting to tackle these problems.

What It Means

Traditional safety tools like IP reputation and GeoIP methods are becoming less reliable. This change highlights the role of network-based fingerprinting now. For more on this, read our article on TLS fingerprinting.

While firms talk about hiding IP addresses, ad tracking is still common. These changes might also push users to certain platforms. Even if users think they're safe, big tech's tracking tools can still watch them. That can give users a false sense of safety. Real privacy still needs practical tools and clear public understanding.

HTTP/2 Rapid Reset Attack Deepdive

2023-10-12T00:00:00+11:00

Distributed Denial of Service (DDoS) attack vectors keep changing. The recent spike in HTTP/2-based DDoS attacks has been notable for its volume, with some attacks surpassing 398 million requests per second. Peakhour observed these attacks and worked through how to understand and mitigate them. This article explains how they work and what operators can do to reduce exposure.

The Rise of HTTP/2 in DDoS Attacks

HTTP/2 was designed to make web traffic more efficient. The same features that improve performance for legitimate users can also be abused in DDoS traffic.

Much of HTTP/2's efficiency lies in "stream multiplexing." It allows multiple messages to be sent over a single TCP connection. While HTTP/1.1 processes each request serially, HTTP/2 can manage multiple concurrent streams on a single connection. This means a client can send multiple requests in a single round trip, increasing how much work each connection can drive.

The 'Rapid Reset' Attack Explained

The "Rapid Reset" attack is a specific DDoS technique built around HTTP/2. The attacker starts by opening multiple streams, much like in a standard HTTP/2 attack. However, instead of waiting for responses, they cancel each request immediately.

The client does this by sending a RST_STREAM frame, indicating that a previous stream should be cancelled. The rapid request-and-reset sequence means the server spends resources processing the request, only for it to be cancelled before a response is generated. This tactic amplifies the server's workload without the attacker needing to wait for responses, which increases the pressure each connection can place on the server.

Variants of the Rapid Reset Attack

Attackers also used variations of the Rapid Reset attack:

One variant involves delaying the reset action. The attacker opens multiple streams, waits, then cancels the streams and instantly opens new ones. This method can evade some rate-based defences.
Another variant avoids stream cancellations. Instead, the attacker tries to open more streams than the server allows. This aims to keep the server continually busy, processing a near-constant flow of requests.

Effective Mitigation Techniques

Mitigation is not as simple as blocking individual malicious requests. A more effective approach is to close the entire TCP connection when malicious activity is detected. The HTTP/2 protocol supports connection termination through the GOAWAY frame. This feature needs to be used aggressively to prevent Rapid Reset attacks, rather than relying on the more passive, standard implementation.

Deciding which connections to treat as malicious is a challenge. One potential strategy is to monitor connection statistics. If a connection exceeds a set threshold of cancelled requests, it might be deemed malicious. Responses to suspect activity could range from sending a GOAWAY frame to terminating the TCP connection.

For the non-cancelling variant, the best approach is to shut down connections that breach the concurrent stream limit, either immediately or after a few violations.

Broader Protocol Implications

These attack techniques are specific to HTTP/2, but the wider protocol lesson still matters. The HTTP/3 (QUIC) protocol isn't directly vulnerable in the same way. As a precaution, server implementations should consider limiting the work done by a single connection.

The Importance of Industry Collaboration

When the threat of the Rapid Reset attack became apparent, the industry collaborated to address the issue. The vulnerability was disclosed to key HTTP/2 implementers, helping to devise and distribute effective countermeasures. The vulnerability is logged against CVE-2023-44487.

HTTP/2 'Rapid Reset' DDoS attacks pose a serious risk to services using the protocol. To reduce exposure, service providers should promptly apply available software patches and updates.

Learn how Peakhour's Application Security Platform helps protect against Layer 7 DDoS attacks, including the HTTP/2 Rapid Reset vulnerability. Contact our team to secure your infrastructure.

Understanding the HTTP/2 Rapid Reset Attack

2023-10-11T00:00:00+11:00

The discovery of the HTTP/2 Rapid Reset flaw exposed a serious weakness in a widely used version of the HTTP protocol. When exploited, it can be used to generate large Distributed Denial of Service (DDoS) attacks against HTTP/2 services. This post explains how the attack works and what operators can do to strengthen their defences.

A Deep Dive into the HTTP/2 Rapid Reset Flaw

HTTP/2 is widely deployed, so a flaw in how implementations handle rapid stream resets can have a large operational impact. To take advantage of the issue, a malicious actor sends a request and immediately cancels it, then repeats that pattern over the same HTTP/2 connection. By scaling this "request, cancel" behaviour thousands of times, an attacker can overwhelm vulnerable HTTP/2 implementations. The result is DDoS attacks at the application layer, with potential downtime and disruption.

Major companies including Cloudflare and Google have dealt with this issue. Google, for example, mitigated a DDoS attack reaching a peak of 398 million requests per second that relied on this technique. For scale, this two-minute-long attack generated more requests than the total number of article views reported by Wikipedia in September 2023.

Mitigating the Threat

Large infrastructure providers have led much of the work to understand the attack mechanics and develop mitigations:

Patching Systems: Prompt patching is the primary control for the HTTP/2 Rapid Reset attack. Companies including Peakhour, Microsoft, and others have tested and patched their systems against this threat.
Rate Limiting: Advanced rate limiting has been a recommended action. It provides an extra layer of protection, minimising the risk of massive request inflows.
Collaborative Efforts: Google and Microsoft have both shared intelligence and collaborated with other cloud providers and software maintainers implementing the HTTP/2 protocol stack. This has resulted in patches and mitigation techniques now employed by numerous large infrastructure providers.

What's Next for Users and Enterprises?

If you serve an HTTP-based workload online, understand whether this attack affects your environment. Verify that servers supporting HTTP/2 are either not vulnerable or have applied the necessary patches. Stay informed and consider reaching out to your service providers or account representatives for configuration assistance and guidance.

The HTTP/2 Rapid Reset flaw is a serious application-layer DDoS risk, but it is manageable with the right mitigations in place. Apply the recommended patches and keep HTTP/2-facing services under active review.

Discover how Peakhour's Application Security Platform protects against Layer 7 DDoS attacks, including the HTTP/2 Rapid Reset vulnerability. Contact our team to secure your infrastructure.

The Rise of OpenBullet

2023-09-01T14:00:00+10:00

At Peakhour, we are seeing more automation tools used to simplify interaction with web platforms. These tools have legitimate uses, including automating repetitive tasks and testing applications, but they can also be misused. OpenBullet is one example: a flexible web testing suite that has become a common tool for web attacks such as credential stuffing. This article explains how OpenBullet works, why it creates risk, which libraries it relies on, and how defenders can detect and block it.

Overview of OpenBullet

OpenBullet is an automation suite for scraping, parsing data, and automated penetration testing. It is commonly used by bot developers for automated attacks, including credential stuffing. Released under the MIT open-source licence on GitHub, it is now in its second version, OpenBullet2, which, as of March 2023, had over 1.1K stars and was forked roughly 370 times.

It is particularly favoured by people with limited programming knowledge because it is easy to use and supports third-party plugins. The tool uses configurations that define the actions to perform on a website, and those configurations are easy to find online.

Types of Actions with OpenBullet

The actions OpenBullet can perform are categorised by the framework and library used. There are three broad types:

Browser Actions: Open or close tabs, maximise or minimise the browser window, and more.
Page Actions: Visit a page, fetch page attributes, set or clear cookies, click on page elements, take screenshots, and so forth.
Element Actions: Set or get element attributes, click on elements, check their status, fill in text forms, and more.

OpenBullet's versatility has made it attractive to users who share configurations freely. Advanced configurations for tasks such as scraping and credential stuffing can be found on forums and even sold.

OpenBullet Versus Other Testing Suites

One of OpenBullet's main advantages over other testing suites or automation frameworks is ease of use. It offers a visual mode, with a simple UI instead of lines of code. It also includes a high-level programming language for fine-tuning operations. It does not offer the same level of control as direct interaction with its underlying frameworks, but it can still cause significant issues for websites.

Why OpenBullet is Dangerous

OpenBullet is a threat because its simple UI lets people without programming skills create automated sequences for web attacks. Its integration with CAPTCHA farms also makes it effective against websites that rely on traditional CAPTCHAs for bot protection.

After installing OpenBullet, an attacker needs to create or import a configuration and manage bot behaviour. They can also configure proxies to distribute attacks, hide their real IP addresses, and sidestep traditional rate limiting.

OpenBullet also supports attacks like credential stuffing through a range of integrations. Attackers can add new credentials, store valid credentials, and set the configuration to run for any duration they choose.

We can, however, counter OpenBullet by detecting and blocking the frameworks on which it relies.

OpenBullet and Its Underlying Libraries

OpenBullet relies on several well-known bot automation libraries and frameworks:

Requests: A Python module for sending HTTP requests with forged attributes. It's highly scalable and can bypass traditional CAPTCHAs using external CAPTCHA farm services. However, it struggles against highly protected sites and mobile applications.
Selenium: This is a browser automation framework initially developed for testing web applications. It can interact with a web service as a human user would, helping attackers mask their bots with human-like behaviours.
Puppeteer: This Node.js library controls Chromium-based browsers. It's faster and lighter than Selenium, making it capable of running more parallel requests.

OpenBullet does not inherently simulate human behaviour; the bot developer has to implement that. Based on an analysis of online configurations, most do not include fake human behaviour features. OpenBullet does, however, support ad hoc JavaScript execution to enable them.

Detecting and Blocking OpenBullet

To detect and block OpenBullet, defenders need to understand where a request is coming from, especially when proxies are used to distribute attacks. OpenBullet can be effective in the wrong hands, but it is not invisible. Several signals can help identify and block its activity.

Identifying Unusual Patterns

Most automated tools, including OpenBullet, generate request patterns that differ from typical human behaviour. The frequency, timing, and sequence of requests can help identify potential OpenBullet attacks. For instance, a high volume of requests from a single IP address, or repeated requests with different login credentials, could indicate automation.

Analysing User Agents

User agents can also provide useful clues. OpenBullet can mimic different user agents to look like a range of browsers, but it may not simulate the broader spread of user agents an actual user base would generate. If an unusual number of requests come from a small set of user agents, it may indicate an automated attack.

Spotting IP Address Anomalies

OpenBullet, like many automated tools, uses proxies to mask its true location and appear to be many different users. Proxies have their own characteristics. Data centre proxies, for instance, do not behave like residential or mobile IP addresses, and they can be flagged as suspicious. Similarly, if many different user identities come from a single IP address, or if the geolocation of an IP address does not match the stated location of the user, it may signal proxy use.

OpenBullet in the Greater Cybersecurity Context

OpenBullet reflects a broader pattern in cybersecurity: tools built for testing can be repurposed for abuse. Its simple UI and automation capabilities show why online security cannot depend on basic controls alone. Although it was created as a web testing tool, its misuse reinforces the need to keep defences current as attack methods change.

The Need for Strong Password Practices

OpenBullet's popularity for credential stuffing attacks underscores the importance of strong password practices. Encouraging users to create unique passwords for each account and change them regularly can help reduce the risks posed by credential stuffing.

Implementing Advanced Bot Protection Measures

To secure online services against tools like OpenBullet, advanced bot protection matters. These measures can provide a multi-layered security strategy that runs in the background, collects detection signals, and challenges users only when necessary. They can help stop bot attacks from the first request, including those carried out through OpenBullet's automation framework.

As web platforms evolve, the tools used to exploit them evolve as well. Understanding threats like OpenBullet is the first step towards stronger online protection.

Peakhour's advanced Bot Management and rate limiting capabilities can help protect your site from automated tools like OpenBullet. Contact our team for a free security analysis.

Advanced Rate Limiting

One practical defensive measure against stuffing attacks, including those made using OpenBullet, is advanced rate limiting. Unlike basic rate limiting, which restricts the number of requests from a particular source within a specified time frame, advanced rate limiting provides a more nuanced and dynamic approach.

A critical feature of advanced rate limiting is its ability to group, or bucket, requests based on factors beyond the source IP address. These factors could include:

Autonomous System Number (ASN): An ASN is a unique number assigned to each network on the Internet. By grouping requests by ASN, it's possible to detect an unusual number of requests from a specific network, even if those requests are spread across many different IP addresses.
Country: Grouping requests by country allows the detection of a sudden surge of traffic from a specific geographic location, which might indicate a coordinated attack.
Device Fingerprint: A device fingerprint can be constructed from a range of attributes, including the device's operating system, browser version, and more. This allows the detection of repeated requests coming from the same device, even if other factors like the IP address or user agent are being manipulated.
Headers: By examining the headers in HTTP requests, it's possible to detect patterns or anomalies that might signify an automated attack. For instance, a high volume of requests with identical headers could indicate the use of an automation tool.

By grouping requests on these and other factors, advanced rate limiting can provide a nuanced and dynamic defence against stuffing attacks. It allows detection of complex attack patterns that might otherwise go unnoticed, adding a useful layer of security for online systems.

Fingerprinting and Behavioral Analysis

Alongside advanced rate limiting, technologies like fingerprinting and behavioural analysis add another layer of protection. By creating a unique identifier for each user based on a combination of factors, such as IP address, device information, and browser settings, and tracking that user's behaviour over time, these technologies can help distinguish genuine users from automated scripts.

These mechanisms can detect anomalies, such as repetitive non-human patterns, sudden changes in behaviour, or usage patterns that do not match the typical profile of a human user. This level of analysis can identify and block even sophisticated bots that are designed to mimic human behaviour, providing an effective defence against OpenBullet and similar tools.

OpenBullet's accessibility and versatility have made it a popular choice for web attacks. Its simple UI, support for CAPTCHA farm services, and ability to run human-like automation allow even users with limited programming skills to execute complex attacks. Despite these challenges, detecting and blocking OpenBullet is achievable by understanding the origin of requests and the nature of its operation. A combination of advanced rate limiting, fingerprinting, and behavioural analysis can provide an effective defence.

Headless Commerce Security

2023-06-28T00:00:00+10:00

At Peakhour, we spend a lot of time looking at e-commerce architecture trends. Single Page Applications (SPAs) and headless commerce keep coming up, with tools such as Nuxt.js, Strapi, Hydrogen, and Gatsby leading many builds. These tools can make frontend work faster and more flexible, but they also put more e-commerce data behind APIs that scrapers can target.

Single Page Applications (SPAs) and headless e-commerce have changed how many retailers build their storefronts. Frontend development tools like Nuxt.js and headless CMSs like Strapi are now common parts of that stack.

The trade-off is exposure. Product information is often available as JSON data, which makes it easier for scrapers to collect at scale. That raises a practical question: how do you secure data while still making it available through APIs?

Strategies for Data Protection

Data protection matters, but it is not a single control. These are the usual layers:

Rate Limiting: Controls the number of client requests to your API within a set time frame.
Bot Detection: Distinguishes between humans and bots based on behavioural patterns.
Page Load Authentication: Secures the page load through bot detection and authenticates subsequent API calls.
IP Threat Intelligence: Blocks suspicious IP addresses from accessing your API.
GeoIP Filtering: Regulates requests based on geographical origin.

As bots change, those controls need to change as well.

Facing the Challenge of Headless Scraping

Headless scraping uses browsers without a user interface to imitate normal browsing. It is difficult to detect, but network fingerprinting can help.

Network fingerprinting examines network features like Transport Layer Security (TLS) settings and HTTP/2 (H2) parameters. By analysing these, companies can detect and block bots, adding another security layer.

Client-side Security in SPAs

In SPAs, where much of the processing happens in the user's browser, the security concerns shift:

Data Exposure: Protecting sensitive data from leakage or manipulation is critical.
Injection Attacks: SPAs must guard against attacks like Cross-Site Scripting (XSS).
Authentication and Session Management: Properly handled, these prevent unauthorised access.
Insecure Direct Object References (IDORs): Proper authorisation stops attackers from accessing others' data.

Risks in JavaScript Packages

SPAs usually depend on JavaScript libraries and packages. They are useful, but they also add supply chain risk. Using only essential packages, keeping them updated, and sourcing them from trusted providers reduces that risk. Supply chain audit tools can help automate the work:

Security audits need to be frequent because vulnerabilities can appear quickly. Tools like npm's npm audit or GitHub's Dependabot, along with regular penetration testing, can help uncover potential weaknesses.

Final Thoughts

The move toward SPAs and headless commerce is a trade-off between development flexibility and security exposure. These architectures can improve user experience and speed up delivery, but they also introduce new security issues.

Client-side security in SPAs needs deliberate attention. Data exposure, injection attacks, and insecure direct object references all need to be managed, and the convenience of JavaScript libraries brings its own vulnerabilities.

Peakhour addresses these problems with rate limiting that manages request traffic and helps prevent attacks without harming customer experience. Our Web Application Firewall (WAF) examines all payload data, adding another layer of protection.

Frequent security audits still matter. They help e-commerce managers keep SPAs and headless commerce operations secure without giving up the efficiency these architectures can provide.

The Rise of the Dragon

2023-05-17T13:00:00+10:00

Camaro Dragon, a Chinese state-sponsored group, has developed a custom firmware implant for TP-Link routers. Once installed, it can turn compromised routers into residential proxies. That weakens traditional cyber-defences, including GeoIP blocking, because traffic can appear to come from ordinary local connections. This article looks at how the malware works, why residential proxies matter for enterprise security, and where GeoIP security measures fall short.

Understanding the New Malware

Check Point's research describes Camaro Dragon's sophisticated attacks on European foreign affairs entities. The group uses a custom firmware implant, known as 'Horse Shell', designed specifically for TP-Link routers. The malware includes a backdoor that grants the attackers continuous access to compromised networks and allows them to build anonymous infrastructure.

'Horse Shell' can execute arbitrary commands on the infected router, transfer files, and relay communications using SOCKS tunnelling. Its design can be adapted to different vendors' firmware, suggesting the possibility of a wider spread.

The People and Intentions Behind The Malware

Investigations into the origin of the 'Horse Shell' malware by Check Point Research, Avast, and ESET point to a well-known cyber threat actor: Mustang Panda. This advanced persistent threat (APT) group, linked to the Chinese government, is known for complex attacks that often exploit Internet-facing network devices.

The primary function of 'Horse Shell' is to relay traffic between an infected device and the attackers' command and control servers. This method obscures the true source and destination of the communication, making it difficult to trace back to the attackers.

Importantly, Mustang Panda appears to choose router implant targets indiscriminately. The infection of a home router doesn't imply that the homeowner is a direct target. Instead, each infected router becomes a node in a broader chain that connects main infections with command and control operations.

Researchers identified this approach when they found the 'Horse Shell' implant during an investigation of targeted attacks against European foreign affairs entities. The implant allows the attackers to maintain ongoing access, establish anonymous infrastructure, and move laterally within compromised networks.

The Implications of Residential Proxies

Residential proxies serve as intermediaries, using real IP addresses issued by Internet Service Providers (ISPs). They are used across a range of applications, including business web scraping and anonymising user online activity.

Residential proxies become more serious when malware such as 'Horse Shell' is involved. This malware infects routers, turning them into a network of residential proxies that can then be used for malicious activity, including data breaches and distributed denial-of-service (DDoS) attacks.

Most importantly, this use of residential IP space can make an attack look as if it originates from a domestic source within the target's location. That undermines traditional cyber-defences.

GeoIP Security Measures and Their Limitations

GeoIP blocking, a traditional cyber security tool, works by limiting access from specific geographical regions or networks frequently associated with cyber threats. However, this method is becoming less effective against the rising use of residential proxies.

Residential proxies can disguise the actual origin of a cyber attack, giving the illusion that it's originating from a trusted, usually local, location. This capability allows them to effectively bypass GeoIP blocking measures. Consequently, malicious actors using residential proxies can carry out their activities with less obvious attribution and often go undetected.

The key operational issue is the exploitation of home routers by malware like 'Horse Shell,' which turns these devices into unwitting participants in cyber attacks. This manipulation means an attack could appear to originate from a seemingly trusted domestic source, which can render GeoIP blocking ineffective.

This threat shows why cyber security needs a more layered approach. Sole reliance on GeoIP blocking is no longer enough. As malware evolves to exploit residential proxies, detection and defence strategies need to adapt. Specifically, it's important to recognise that relying solely on GeoIP blocking, or trusting apparently local connections and deny-listing countries like Russia and China, can create a false sense of security.

Detecting Residential Proxies: The Role of Network Fingerprinting

The rise of residential proxy malware makes network fingerprinting important for identifying these threats. Five techniques can help detect residential proxies:

TCP Fingerprinting: Proxied requests may generate TCP fingerprints that don't match the expected device type. For example, a request from a residential IP address that bears the fingerprint of a server OS could be a strong signal of a proxy.
TLS and HTTP/2 Signatures: As with TCP fingerprints, unusual TLS and HTTP/2 signatures could reveal proxies. An incoming request using a version of TLS or HTTP/2 not commonly used in residential networks might indicate a proxy.
JavaScript-based Fingerprinting: This method identifies the specific browser in use. Discrepancies in JavaScript fingerprints, or the absence of a fingerprint, could suggest the presence of a residential proxy.
Timing Analysis: The timing of requests can also be a signal. Proxied requests might exhibit longer or inconsistent intervals between requests, indicating a residential proxy.
Port Scanning: This technique can detect open ports that could indicate the presence of SOCKS or other proxies, revealing possible exposure to threats.

While residential proxies have legitimate uses, such as web scraping, those applications sit beside a more serious risk: compromised trusted or local networks can be turned into proxy infrastructure at scale. Cyber threats like 'Horse Shell' use residential proxies to undermine traditional GeoIP security measures, which means defence strategies need to keep evolving.

In Part 1 of our series on residential proxies, we provide an overview of this topic and why it matters to security teams. From basic uses to their role in complicated cyber attacks, we cover the key points.

Learn how Peakhour's Application Security Platform protects against account takeovers and credential stuffing. Contact our team to secure your user accounts.

Cohen, I., Madej, R., & Threat Intelligence Team (2023). The Dragon Who Sold His Camaro: Analyzing Custom Router Implant. Check Point Research. Retrieved from https://research.checkpoint.com/2023/the-dragon-who-sold-his-camaro-analyzing-custom-router-implant/ ↩
Goodin, D. (2023, May 17). Malware turns home routers into proxies for Chinese state-sponsored hackers. Ars Technica. Retrieved from https://arstechnica.com/information-technology/2023/05/malware-turns-home-routers-into-proxies-for-chinese-state-sponsored-hackers/ ↩

Residential Proxy Detection

2023-05-17T13:00:00+10:00

Residential proxies are under increasing scrutiny, both for how their IP addresses are obtained and for how those networks are used. They also expose how heavily many online services rely on GeoIP data, from content customisation to security controls.

That scrutiny reveals a complicated reality. Residential proxies can help businesses, researchers, and individuals preserve anonymity or work around GeoIP-dependent restrictions. The same properties also create ethical problems, particularly when the networks are misused.

This article explains what residential proxies are, how they work, where they are useful, and where the risk sits. The same properties that make them attractive for legitimate monitoring and research also make them useful for abuse.

Demystifying Residential Proxies

These proxies connect automated software to the internet through IP addresses tied to real-world residential locations. That lets the software look closer to ordinary internet usage, which can help it bypass geographical and network restrictions while adding a layer of anonymity.

Residential proxies need a clear legal and ethical distinction. Their use can be lawful, including for web scraping and data gathering, while still enabling activity that may breach the intended usage policies of some online services. This could include mass consumption of data intended for general use, such as scraping websites for machine learning datasets. These actions may not be strictly illegal, but they raise substantial ethical questions and are often unwelcome to the data providers.

Applications of Residential Proxies

The defining characteristic of residential proxies is that requests can appear to originate from local residential networks. That supports a wide range of use cases, including:

Concealing True IP Addresses: Residential proxies allow third parties to hide genuine IP addresses and location, making identity and origin harder to determine. By routing internet traffic through residential IP addresses, they can evade detection, bypass security rules, and access geo-restricted content.
Research and Monitoring: Residential proxies are often used by researchers, analysts, and market intelligence professionals to gather data and monitor online activity. By utilising residential IP addresses, they can emulate real user IP addresses and bypass restrictions.
Web Scraping and Data Gathering: Residential proxies are central to many web scraping and large-scale data collection workflows. With the capacity to rotate IP addresses and access a wide range of residential locations, third parties can scrape valuable data from websites without triggering anti-scraping measures. Residential proxies can make data scraping more discreet, with fewer access interruptions and cleaner collection results.
Ad Verification: Residential proxies are widely used for ad verification. Ad verification companies utilise residential IP addresses to confirm the accuracy and legitimacy of online advertisements. By mimicking genuine residential connections, they can check that ads are correctly displayed and monitor the performance and integrity of advertising campaigns.
Ad Fraud: Residential proxies can also be misused for ad fraud. Competitors or their agents may utilise residential IP addresses to falsely inflate the views of a rival's online advertisements. By using genuine residential connections, these entities can manipulate advertising metrics, compromising the accuracy and integrity of the ad's performance data. This abuse of residential proxies for ad fraud poses a significant concern for the online advertising industry.
Last Mile Monitoring: Last mile monitoring is another application for residential proxies, allowing companies to assess the user experience from a residential viewpoint. By using residential IP addresses, they can monitor website loading speeds, test service availability, and evaluate the performance of online platforms more accurately. This helps organisations pinpoint and rectify issues that may negatively affect user satisfaction.

Navigating the Risks and Concerns

Residential proxies create material risks, particularly when users are unaware that they are hosting one. Their use can introduce practical limits and security vulnerabilities that are easy to miss.

Despite their valid uses, residential proxies can be used for cybercriminal activity. Malicious actors may exploit them for account takeovers, fraud, or other targeted attacks.

Using residential proxies without the knowledge or consent of residential users creates serious security issues. These users, unaware of how their connections are being utilised, could face legal exposure, compromised privacy, and cyber threats. Their devices could unwittingly participate in malicious activity, leaving them exposed to legal consequences and reputational damage.

Exploring the Creation of Residential Proxies and their Implications

Residential proxy providers build their networks in several ways, some of which can have significant security implications.

Providers can obtain residential proxies through partnerships with Internet Service Providers (ISPs) or by leasing IP addresses from legitimate residential users. At the same time, some providers or private groups may use questionable practices to obtain residential proxies.

SDKs: Certain applications may include Software Development Kits (SDKs) that gather and sell user data, including their IP addresses. In some instances, these SDKs can be exploited by residential proxy providers to acquire residential IPs without the explicit consent or knowledge of the users.
Malware Exploitation: Malware, including botnets, can infiltrate the devices of unsuspecting residential users. Attackers may then exploit these infected devices as part of a broader residential proxy network, without user awareness. This unauthorised use of residential IPs poses significant security threats to both the affected users and the wider internet ecosystem.
Free VPN Services: Some free Virtual Private Network (VPN) services, which promise anonymity and privacy, may use users' connections as part of their residential proxy networks. Users unknowingly become exit nodes for other users' internet traffic, potentially exposing their connections to malicious activities.

Using residential proxies without the knowledge or consent of residential users raises serious security concerns. These users may not understand how their connections are being used, which can lead to legal consequences, compromised privacy, and exposure to cyber threats. Their devices might unknowingly participate in malicious activities, exposing them to potential legal consequences and reputational damage.

The Birth of 'Ethical' Proxies

An important part of the residential proxy discussion is the rise of providers claiming that their IP address pools are ethically sourced. These companies argue that they have obtained the consent of the original IP owners and provide transparency in how these connections are utilised. By positioning themselves as 'ethical' residential proxy providers, they aim to mitigate the associated risks and concerns.

Even where consent is obtained, the potential for misuse remains a significant issue. This is largely due to the inherent anonymity of residential proxies and the difficulty of tracing activity back to the original user. Despite claims of ethical sourcing, the complexity and opacity of the residential proxy environment mean that it remains a grey area, inviting scepticism and demanding further scrutiny.

The result is a nuanced market that consumers, providers, and regulators need to understand as the digital landscape continues to evolve.

From Hola VPN to the Camaro Dragon

Several publicised incidents show how residential proxies are formed and the impact they have had on the industry and users. These examples show the different ways residential proxies can be created and used, legitimately and otherwise.

Hola VPN is a well-known free VPN service that promises privacy, security, and access to blocked content. However, it fell under scrutiny when it was revealed that it was selling its users' bandwidth to its sister company, Luminati, which operates a residential proxy network. Users of Hola VPN unknowingly became part of a residential proxy network, with their connections being utilised by third parties. This raised significant ethical and security concerns, as users' devices could be implicated in illegal activities carried out using their IP addresses.
The residential proxy service known as 911 has been selling access to hundreds of thousands of Microsoft Windows computers for the past seven years. This service enables customers to route their internet traffic through these computers, allowing them to appear as if they are browsing from any country or city around the world. While 911 claims that its network comprises users who voluntarily install its "free VPN" software, recent research indicates that the proxy service has a history of obtaining installations through questionable "pay-per-install" affiliate marketing schemes, some of which were operated by 911 itself. The service primarily targets users in the United States but has a global user base. Residential proxy networks like 911 can serve legitimate business purposes, but they are often abused for cybercriminal activities due to the difficulty in tracing malicious traffic back to its source.
Cybercriminals are increasingly leveraging residential broadband and wireless data connections to anonymise their malicious traffic. One notable type of network, referred to as "bulletproof residential VPN services", has gained attention. These networks are constructed by acquiring discrete blocks of internet addresses from major internet service providers (ISPs) and mobile data providers. An investigation into one such company, Residential Networking Solutions LLC (also known as Resnet), unveiled that it had obtained a significant number of IP addresses, some of which were previously controlled by AT&T Mobility. Resnet leased these IP addresses, enabling it to resell data services for major providers such as AT&T, Verizon, and Comcast Cable. However, the precise nature of the relationship between Resnet and AT&T remains unclear, and the matter has been referred to law enforcement. Cases like this emphasise the potential abuse of IP addresses within residential proxy networks.
Infatica.io, a Singapore-based company, has developed a network of over 10 million web browsers that clients can rent to conceal their true internet addresses. The company achieved this by compensating browser extension developers to incorporate its code into their extensions. Many extension developers struggle to earn fair compensation for their work, making offers like these enticing. Infatica seeks extensions with at least 50,000 users and offers to pay developers between $15 and $45 per month for every 1,000 active users with the code included in their extensions. Infatica's code routes web traffic through the browsers of extension users, providing anonymity to the company's customers. The service's pricing depends on the volume of web traffic a customer wishes to anonymise. However, this approach raises concerns about privacy and the potential misuse of users' browsers for malicious activities. Developers, particularly those who author free software, can find the monetisation opportunity offered by residential proxies extremely tempting. The potential to earn revenue from their existing user base by incorporating such code into their extensions can present a persuasive proposition.
Camaro Dragon, a form of malware, provides a recent example of residential proxies being acquired through malicious means. This malware infects the devices of unsuspecting users, forming a botnet that can then be utilised as a residential proxy network. Infected devices can then be exploited for various cybercriminal activities without the knowledge or consent of the device owners. This example highlights the significant cybersecurity risks associated with residential proxies and emphasises the importance of robust protection measures.
Volt Typhoon is a state-sponsored actor based in China that typically focuses on espionage and information gathering. Volt Typhoon proxies all its network traffic to its targets through compromised SOHO network edge devices (including routers). Microsoft has confirmed that many of the devices, which include those manufactured by ASUS, Cisco, D-Link, NETGEAR, and Zyxel, allow the owner to expose HTTP or SSH management interfaces to the internet. Volt Typhoon has been active since mid-2021 and has targeted critical infrastructure organisations in Guam and elsewhere in the United States.

These examples illustrate the ethical, security, and legal issues surrounding residential proxies. They put transparency and consent at the centre of how proxy networks are acquired and used. The implications for users, the security industry, and the broader digital landscape are substantial, which is why regulation, user education, and responsible practices matter for protecting privacy, security, and the integrity of the internet.

Legal Consequences of Residential Proxies in Data Scraping Operations

Residential proxies are a concern because of their potential for misuse and their legal implications. Two notable cases, the Ticketmaster Case and the Meta vs Bright Data Case, have drawn attention to the challenges posed by the unauthorised use of residential proxies in commercial settings and data scraping operations. These cases show why the legal ramifications of residential proxy use need to be understood in real-world scenarios.

The Ticketmaster Case: In 2018, a major international case came to light when Ticketmaster sued Prestige Entertainment for using residential proxies to circumvent ticket-purchasing limits and scoop up large numbers of tickets for resale. This case underscores the potential misuse of residential proxies in commercial settings, and how they can be used to breach the terms of service of websites.
The Meta vs Bright Data Case: The legal case between Meta Platforms, Inc. (formerly Facebook) and Bright Data Ltd. demonstrates a contentious and potentially unlawful use of residential proxies in the real world. In this case, Meta accused Bright Data of operating a business designed to use automated software to scrape and sell data from various online platforms, including Facebook and Instagram. This scraping was allegedly facilitated using unauthorised tools and services that bypassed detection by Meta's security measures. Despite Meta's efforts to halt these activities, Bright Data purportedly continued its operations. The data involved included user profiles, follower counts, and shared posts. Bright Data was alleged to not only scrape this information but also advertised the sale of the scraped data. The scope of this operation was extensive, with the Instagram data set alone priced at $860,000.

These cases show how residential proxies are used in practice, the challenges they present, and why their use remains legally and commercially contested.

The Wider Implications for the Security Industry

The growth of residential proxies, and the way some networks are acquired, has broader implications for the security industry. It raises questions about transparency, ethical practices, and the responsibility of proxy providers.

Ethical and Regulatory Implications: The questionable practices some providers use to acquire residential proxies highlight the need for stronger regulation and industry standards. This would help ensure that residential proxies are obtained and used in a lawful and ethical manner, protecting users' privacy and the wider internet ecosystem. There is a clear demand for more transparency in how these services operate and procure their proxies.
Cybersecurity Implications: Residential proxies can enable malicious cyber activity, ranging from fraud to targeted attacks. This can increase the need for cybersecurity measures and protections, potentially reshaping strategies and priorities within the cybersecurity industry.
Legal and Reputational Implications: If individuals unknowingly become part of a proxy network, there could be legal repercussions for them if their connections are utilised for malicious activities. This could lead to greater scrutiny and liability for companies operating within this space.
State Actors and Residential Proxy Networks: State-sponsored actors have been known to establish their own residential proxy networks within foreign countries for various campaigns, including information warfare, disinformation campaigns, and surveillance, adds another layer of complexity to the issue. These activities pose significant geopolitical and security risks, requiring increased international cooperation and robust defence mechanisms.

The rise of residential proxies exposes a weakness in common security models: the assumption that residential and mobile IPs are inherently more trustworthy, and that GeoIP is a reliable reputation or security control. Widespread proxy use has shown how brittle that assumption can be.

Uncertain or unethical sourcing makes that trust problem harder. It can make online interactions less reliable and introduce security risks.

Residential proxies are not just tools; they highlight a deeper issue in how we approach digital access and security. Understanding what is already known, questioning current practices, and building better controls are practical steps towards using residential proxies responsibly and ethically. Recognising the false sense of security GeoIP restrictions can provide is part of that work.

Part 1 ends here. In Part 2: the Camaro Dragon malware, we look more closely at a specific case. This sophisticated malware uses residential proxies in a way that shows their potential for misuse. The next article covers how Camaro Dragon works, its impact on cybersecurity, and practical protection measures.

Mi, X., Tang, S., Li, Z., Liao, X., Qian, F., & Wang, X. (2021). Our Phone is My Proxy: Detecting and Understanding Mobile Proxy Networks. Retrieved from https://xianghang.me/files/ndss21_mobile_proxy.pdf ↩
Mi, X., Feng, X., Liao, X., Liu, B., Wang, X., Qian, F., Li, Z., Alrwais, S., Sun, L., & Liu, Y. (2019). Resident Evil: Understanding Residential IP Proxy as a Dark Service. Retrieved from https://www-users.cse.umn.edu/~fengqian/paper/rpaas_sp19.pdf ↩
Krebs, B. (2019, August 19). The Rise of "Bulletproof" Residential Networks. Retrieved from https://krebsonsecurity.com/2019/08/the-rise-of-bulletproof-residential-networks/ ↩
Krebs, B. (2022, July 18). A Deep Dive Into the Residential Proxy Service '911'. Retrieved from https://krebsonsecurity.com/2022/07/a-deep-dive-into-the-residential-proxy-service-911/ ↩
Krebs, B. (2021, March 1). Is Your Browser Extension a Botnet Backdoor? Retrieved from https://krebsonsecurity.com/2021/03/is-your-browser-extension-a-botnet-backdoor/ ↩
Meta Platforms, Inc. v. Bright Data Ltd. Retrieved from https://unicourt.com/case/pc-db5-meta-platforms-inc-v-bright-data-ltd-1374026 ↩
Volt Typhoon targets US critical infrastructure with living-off-the-land techniques. Retrieved from https://www.microsoft.com/en-us/security/blog/2023/05/24/volt-typhoon-targets-us-critical-infrastructure-with-living-off-the-land-techniques/ ↩

When Bots Break Bad

2023-05-16T13:00:00+10:00

Bots account for a large share of web traffic. Recent studies put automated traffic at nearly 50% of all internet requests. Some bots are useful, such as search engine crawlers that index your site. Some are clearly harmful, such as scrapers and sneaker bots. Others sit in a grey area, including backlink and marketing bots from services such as Ahrefs and SEMrush. Even useful bots can create problems when they crawl too hard. This article looks at the main bot types and how to manage them with robots.txt and bot management tools.

Understanding the Different Types of Bots

'Good Bots'

Good bots perform legitimate work. Search engine crawlers like Googlebot and Bingbot index webpages so search results can stay current and relevant. Other examples include uptime and performance monitoring bots.

'Bad Bots'

Bad bots harm websites, users, or both. Common examples include:

Scraping content, copying and repurposing data from websites.
Sneaker bots, automatically purchasing limited-edition products (like sneakers) before human users can.
Spam bots, posting unsolicited messages and advertisements in comment sections or forums.
Vulnerability Scanners, trying thousands of website URLs to find security vulnerabilities.
Account Takeover, attempting to gain access to existing user/admin accounts using either credential stuffing or brute-force attacks.

'Grey Bots'

Grey bots sit between good and bad. They often serve a useful purpose and may follow crawling directives in robots.txt, but they can still cause problems when they crawl too aggressively. Common examples include:

AhrefsBot: A backlink analysis bot used by Ahrefs, an SEO tool.
SEMrushBot: A bot used by SEMrush, another popular SEO and digital marketing tool.
MJ12bot: A bot used by Majestic, a service that provides backlink data and analysis.
ScreamingFrog: An SEO analyser run from a local desktop.

When Grey bots (and even Good Bots) go bad.

Left unattended, grey bots can create practical problems:

Slow page loading times, which affect user experience.
Strain on server resources, potentially causing crashes, downtime, and higher costs.
Distorted website analytics, when bot traffic is mistaken for human traffic.

Managing Grey Bots with Robots.txt

The robots.txt file is a simple text file that tells web crawlers which parts of your site they can or cannot access. You can use it to manage bot behaviour and protect your website from aggressive crawling. Useful controls include:

Disallowing specific bots: You can block specific bots from accessing your site by adding a "User-agent" and "Disallow" directive to your robots.txt file. For example:

User-agent: AhrefsBot
Disallow: /

Limiting crawl rate: You can ask bots to slow down their crawling by adding a "Crawl-delay" directive:

User-agent: SEMrushBot
Crawl-delay: 10

Not all bots will follow robots.txt. ScreamingFrog, for example, can be instructed to ignore robots.txt and crawl a site as quickly as possible. You would not want a competitor doing this to your site.

Bot Management Tools

In addition to robots.txt, bot management tools (like those provided by Peakhour) can protect your website from abusive bots. Good bot management tools automatically block most unwanted traffic using a combination of Threat Intelligence, Fingerprinting techniques, Reverse DNS verification, and Header Inspection.

Advanced techniques like rate limiting and machine learning can help identify more sophisticated bad bots.

Search Bots and Double Crawling

Search bots like Bingbot can sometimes blindly follow links and crawl the same page multiple times due to different URL parameters. This double, triple, or worse crawling can increase server load and make indexing less efficient. eCommerce sites are especially exposed because product catalogues often have several filtering paths. We've seen Bing go haywire on a number of sites. Most recently, it was issuing around 50,000 requests per day to the search function of a Magento 2 store while cycling through parameters. This dropped to 2-3k requests per day when fixed. On another store, Bing was responsible for nearly half of all page requests (40k page requests) on a busy OpenCart store. Configuring it to ignore parameters dropped this to around 4k per day.

Configuring Search Bots to Ignore Query Parameters

Note: Since publishing both Google and Bing have removed the ability to ignore parameters when crawling via their webmaster/search console tools. See using robots.txt to instruct search engines to ignore query string parameters

To help search bots crawl your site efficiently, you can configure them to ignore specific query parameters. Use these methods:

Configuring Bing Webmaster Tools

Bing Webmaster Tools provides an option to specify URL parameters that should be ignored during the crawling process. To configure this setting, follow these steps:

Sign in to your Bing Webmaster Tools account and select the website you want to manage.
Navigate to the "Configure My Site" section and click on "URL Parameters."
Click on "Add Parameter" and enter the parameter name you want Bingbot to ignore.
Select "Ignore this parameter" from the dropdown menu and click on "Save."
Configuring Bing Webmaster Tools this way helps stop Bingbot double crawling pages with specific URL parameters, reducing server load and improving indexing efficiency.

Managing Other Search Bots

For other search engines like Google, use the relevant webmaster tools to manage URL parameters. In Google Search Console, follow these steps:

Sign in to your Google Search Console account and select the property you want to manage.
Navigate to the "Crawl" section and click on "URL Parameters."
Click on "Add Parameter" and enter the parameter name you want Googlebot to ignore.
Choose "No URLs" from the "Does this parameter change page content seen by the user?" dropdown menu.
Click on "Save."
Specifying the parameters you want search bots to ignore can prevent double crawling and make indexing more efficient.

Final Thoughts

When good or grey bots crawl too aggressively, they can cause the same operational problems as malicious bots: overloaded servers, slower pages, and worse user experience. Monitor website traffic and server load, set clear robots.txt rules, and use the major search engines' webmaster tools to control inefficient crawling. Done properly, this improves website performance and can lower infrastructure costs.

Advanced Anomaly Detection

2023-05-15T13:00:00+10:00

Modern Application Security Platforms need reliable anomaly detection to identify and respond to emerging threats in real-time. For DevOps, SRE, and DevSecOps teams, machine learning algorithms such as Robust Random Cut Forest (RRCF) provide the foundation for automated threat detection and response systems that can operate at the scale and speed contemporary applications require.

Strategic Importance of Anomaly Detection in Application Security

Real-time anomaly detection is a core Application Security Platform capability. It helps identify threats before attacks affect application performance or security posture:

Enterprise Threat Landscape

Modern applications face attack vectors that traditional signature-based detection cannot address:

Adaptive Bot Networks: AI-powered bots that modify behaviour based on defensive responses
Zero-Day Exploits: Previously unknown attack patterns that bypass traditional security rules
Volumetric Attacks: DDoS attacks that scale dynamically to evade rate limiting
Insider Threats: Subtle anomalies in user behaviour that indicate account compromise

Application Security Platform Requirements

Effective anomaly detection needs to integrate cleanly with broader security capabilities:

Real-Time Processing: Threat identification within milliseconds of detection
Scalable Architecture: Analysis of millions of requests without performance degradation
Context Awareness: Integration with application metadata and user behaviour profiles
Automated Response: Immediate threat mitigation through dynamic rule deployment

Advanced Machine Learning for Security

Robust Random Cut Forest provides anomaly detection capabilities designed for streaming data environments common in Application Security Platforms:

Algorithmic Advantages for Security Applications

Streaming Data Processing: Real-time analysis without historical data dependencies
Dimensionality Handling: Effective analysis of high-dimensional security feature vectors
Adaptive Learning: Continuous model updates based on evolving traffic patterns
Computational Efficiency: Linear scaling suitable for high-throughput security processing

Implementation in Application Security Platforms

RRCF enables threat detection across multiple security dimensions:

Traffic Pattern Analysis: Identification of unusual request volumes, frequencies, and distributions
Behavioural Anomalies: Detection of user actions that deviate from established profiles
Network Fingerprinting: Recognition of abnormal connection patterns and protocol usage
Content Analysis: Identification of malicious payloads and injection attempts

RRCF Advantages for Application Security Platforms

Traditional batch-processing anomaly detection systems are a poor fit for Application Security Platforms that must respond to threats in real-time. RRCF's streaming approach provides practical advantages:

Real-Time Threat Detection

Immediate Analysis: Process and analyse security events as they occur, without waiting for batch processing
Adaptive Baselines: Continuously update normal behaviour models based on current traffic patterns
Memory Efficiency: Maintain configurable rolling windows of security data for optimal performance
Scalable Processing: Handle millions of security events per second without degradation

Security-Optimised Implementation

RRCF's forest-based approach is useful for security applications:

Multi-Dimensional Analysis: Analyse request patterns, user behaviour, and network characteristics at the same time
Shape-Sensitive Detection: Identify subtle changes in attack patterns that signature-based systems miss
False Positive Reduction: Leverage ensemble methods to reduce noise in security alerting
Contextual Awareness: Understand normal application behaviour patterns for more accurate threat detection

Application Security Platform Integration

Enterprise Deployment Architecture

Peakhour's Application Security Platform implements RRCF through high-performance Rust-based processing:

Edge Processing Capabilities

Global Deployment: RRCF analysis deployed across CDN edge locations for minimal latency
Distributed Learning: Aggregated threat intelligence from multiple geographic regions
Local Response: Immediate threat mitigation at the edge without central processing delays
Bandwidth Optimisation: Process security events locally to reduce data transmission requirements

Platform Integration Benefits

Unified Threat Detection: RRCF analysis integrated with WAF/WAAP, bot management, and DDoS protection
Automated Response: Dynamic security rule generation based on anomaly detection results
DevSecOps Workflow: API-first architecture enabling integration with security automation tools
Compliance Reporting: Detailed anomaly detection logs for security audits and regulatory requirements

Advanced Security Use Cases

Credential Stuffing Detection

Behavioural Analysis: Identify unusual login patterns that indicate automated credential testing
Geographic Anomalies: Detect impossible travel scenarios and location-based attack patterns
Volume Analysis: Recognise subtle increases in authentication attempts that indicate coordinated attacks
Success Rate Monitoring: Identify campaigns through abnormal authentication success/failure ratios

API Threat Detection

Endpoint Anomalies: Detect unusual API usage patterns that indicate reconnaissance or exploitation
Rate Pattern Analysis: Identify sophisticated rate limiting evasion techniques
Response Time Analysis: Detect performance impacts from malicious API usage
Authentication Anomalies: Recognise token abuse and API key misuse patterns

Zero-Day Threat Identification

Traffic Pattern Deviations: Identify new attack vectors through unusual request characteristics
Response Pattern Analysis: Detect exploitation attempts through server response anomalies
Protocol Anomalies: Recognise malformed requests that indicate exploit attempts
Payload Analysis: Identify suspicious content patterns in request bodies and parameters

Operational Excellence Through Advanced Anomaly Detection

Performance and Security Integration

RRCF implementation delivers measurable improvements across security and performance metrics:

Threat Detection Speed: Sub-millisecond anomaly identification for real-time response
False Positive Reduction: Ensemble methods reduce security alert fatigue
System Performance: Efficient processing maintains CDN performance whilst enhancing security
Adaptive Learning: Continuous improvement in threat detection accuracy over time

DevSecOps Enablement

Modern Application Security Platforms provide APIs and automation capabilities:

Security Automation: Programmatic access to anomaly detection results for automated response
CI/CD Integration: Security testing and validation integrated into development workflows
Monitoring Integration: SIEM and SOC platform integration for security operations
Custom Rule Development: Framework for developing application-specific anomaly detection rules

Final Thoughts

Advanced anomaly detection through RRCF is a fundamental capability for modern Application Security Platforms. By implementing machine learning algorithms at the edge, organisations can achieve real-time threat detection that adapts to evolving attack patterns whilst maintaining application performance.

The integration of RRCF with security capabilities including WAAP, bot management, and DDoS protection creates a unified platform that addresses the security requirements of contemporary applications and APIs. For DevSecOps teams, this approach enables automated threat response whilst providing the visibility and control needed for effective security operations.

Double MAD?

2023-05-15T13:00:00+10:00

This article explores the use of Double Median Absolute Deviation (Double MAD) for [anomaly detection](/learning/threat-detection/what-is-anomaly-detection/) in time series
data, particularly in skewed or non-symmetric distributions. Double MAD, which calculates two median absolute
deviations — one for data below the median and one for data above — provides a more nuanced approach than traditional
MAD, allowing for accurate detection of anomalies even in skewed data distributions. We also delve into its application
in identifying slow abuse, like bots, by catching lower range anomalies. However, it's important to note Double MAD's
limitations such as not capturing seasonal data shape and trends over time. A comparison is also drawn with the Z-score
method, highlighting that the choice between the two depends on the nature of your data. The article provides insights
into the practical implementation of Double MAD and its potential to improve your data analysis toolkit.

Operational systems increasingly rely on time-series data for decisions. Anomaly detection is one practical use: by identifying patterns that deviate from the norm, businesses can investigate potential issues early or understand unexpected opportunities.

One useful technique for anomaly detection is the Median Absolute Deviation (MAD) and, more specifically, its extension, the Double MAD. This article explains where Double MAD fits in time-series anomaly detection and how it can help identify anomalous clients.

Understanding MAD and Double MAD

MAD, a robust measure of variability, is less susceptible to outliers than standard deviation. It calculates the median of absolute deviations from the data's median, often providing a better representation of 'normal' behaviour in datasets with skewed distributions or outliers.

Double MAD is an extension of MAD, where two MADs are calculated — one for the data below the median and another for the data above. This split gives the detection process a better fit for asymmetric data, which is common in real-world time series data.

Why Double MAD?

While MAD provides a robust way to understand the 'normal' range of a dataset, it assumes a symmetric distribution of data around the median, which may not always hold true. Double MAD is useful where that assumption breaks down, offering an improved anomaly detection process for skewed or asymmetric datasets.

In time-series analysis, especially with 24-hour cycles like web traffic or server usage, patterns can exhibit seasonality and trend components. These patterns can often be asymmetric, making Double MAD a valuable tool for capturing the variability in different parts of the data.

Using Double MAD in Anomaly Detection

The Double MAD implementation provided uses Rust, a system programming language known for speed and memory safety. The code calculates the lower and upper MAD values, along with their respective thresholds. Anomalies can then be detected by comparing each data point to these thresholds.

An anomaly is defined as a data point that deviates significantly from the expected range. If a data point falls below the lower MAD threshold or above the upper one, it can be flagged as an anomaly. This approach is especially effective when handling datasets with high variability or extreme values.

Double MAD for Anomalous Client Detection

Beyond time-series data, Double MAD can also be instrumental in identifying anomalous behaviour among clients. By comparing each client's behaviour against the Double MAD of the time-series data, teams can pinpoint clients that deviate from the norm.

For instance, in the context of web service usage, an anomalous client might be one that is sending an unusually high or low number of requests. By using Double MAD, you can flag such outliers and take appropriate action, such as investigating potential misuse or reaching out to understand and address any issues they may be facing.

Detecting Lower-Range Anomalies: A Case of Slow Abuse

An interesting application of Double MAD is in detecting lower-range anomalies, a pattern often associated with slow abuse such as bots or Distributed Denial of Service (DDoS) attacks. These abuses are characterised by an unusually low frequency of activity that is consistent over a prolonged period. This consistent, low-level activity can fly under the radar of typical anomaly detection systems.

By setting a lower MAD threshold, Double MAD can effectively detect these lower-range anomalies, providing early warning of slow abuse. Its ability to detect both high and low anomalies makes Double MAD a flexible tool for anomaly detection.

The Math Behind Double MAD

To illustrate the power of Double MAD, let's consider a dataset from a right-skewed distribution. Applying the conventional MAD approach might lead to false positives where normal data points are marked as outliers. This is because MAD uses a symmetric interval around the median, which doesn't account for the skewed nature of our data.

With Double MAD, we instead calculate two MADs — one for the data below the median (MAD-lower) and another for the data above (MAD-upper). Outlier thresholds are then defined using these two MADs. The lower threshold is calculated as the median minus a multiplier (k) times MAD-lower. The upper threshold is the median plus k times MAD-upper.

This approach takes into account the asymmetric nature of our data, providing more accurate anomaly detection. For example, in a right-skewed distribution, Double MAD would correctly identify only the extreme right tail values as outliers without incorrectly flagging data points on the left tail.

Wrapping Up

Accurate anomaly detection matters when teams rely on time-series data to operate and investigate systems. The Double MAD approach provides a robust method for this, allowing businesses to better understand their data, spot potential issues early, and make more informed decisions.

Whether you're monitoring web traffic, server usage, or client behaviour, leveraging Double MAD can offer valuable insights and help ensure your operations continue to run smoothly. The ability to detect both high and low anomalies makes it especially powerful, providing protection against potential threats like slow abuse.

Understanding and implementing Double MAD gives your data analysis toolkit a more complete view of asymmetric data and helps you detect potential anomalies earlier.

IP Threat Intelligence

2022-07-15T13:00:00+10:00

Threat intelligence helps organisations make earlier decisions about cyber attacks. One of the most common forms of threat intelligence in cyber security is IP reputation lists. For example, a given IP address might have a poor reputation for spam, ddos attacks, malware, and several other categories. IP reputation lists often form a front line of defence in Web Application Firewalls and cyber security solutions.

How Peakhour uses IP threat intelligence

Peakhour supports threat intelligence across more than 20 categories, including:

Active DDoS attacks
Brute forcing
Active attackers
Computers infected with malware
Anonymous Proxies
Forum Spammers
TOR anonymous users
IPs with poor reputation
Unroutable and unassigned IPs
Robots and web scrapers
Datacenter
Hosting Providers
Crawlers
And more

Customers have access to all 10 lists, which can be enabled as blocklists or used as part of a custom firewall rule, rate limiting rule, or page rule. For example, you may want to disallow POSTs from forum spammers, rate limit proxies, and outright deny traffic from known brute-forcing IPs.

Creating a spammer can't post rule

How does Peakhour assemble these lists?

The IP reputation lists are sourced from third-party sources, including open source intelligence feeds (OSINT), commercial feeds, community feeds, and our own threat intelligence. IPs are categorised into our pre-defined lists and made available to the WAF and rules engine. Each list is re-evaluated and updated based on the data provider's update schedule; some are updated every minute.

Internally managed feeds include bot sources that are verified using reverse DNS lookups, PTR record lookups, and WHOIS verification (such as Facebook IPs). WAF hits across customers are consolidated and made available as the Active Attacker list, which is updated in near real time. Our Malware and C&C nodes lists are generated from various partnerships.

The Anonymous Proxies list contains known open proxies, services that relay traffic without authentication, whilst our targeted VPN list tracks known third-party VPN services.

IPs are fed back into our system for re-evaluation to help identify emerging behaviour within our customer data.

Data visualisation

Requests from IPs that match a blocklist are tagged with the lists they belong to. Firewall events are enriched with this information, providing visibility into security threats. This context helps you decide how to handle requests, whether they should be blocked, rate limited or observed.

Firewall events generated by reputation matches

Blocks generated by our reputation lists can also be viewed in our analytics section.

Firewall events generated by reputation matches

Future work

We are working on additional data sources to further refine and expand our lists. This includes further segregating our data centre lists and categorising IPs that appear on several lists. We are also introducing our threat research centre to discover possible threats and enrich data blocked only by our WAF.

IP threat intelligence adds another layer of security to a cyber defence system. Peakhour sources and maintains up-to-date threat intelligence, helping our clients better protect themselves against would-be attackers.

See how Peakhour's IP threat intelligence supports the first line of defence for your applications. Contact our team to discuss your security requirements.

Intelligent Rate Limiting

2022-05-16T13:00:00+10:00

Rate limits protect web applications from clients making excessive requests. Peakhour.IO supports rate limits with flexible controls for selecting which clients are limited and which type of limit applies.

What kinds of attacks are stopped by rate limiting

When an application is protected with rate limiting, the main attack patterns are:

Brute force and enumeration attacks
Denial of Service (DoS) and Distributed Denial of Service (DDoS)
Site scraping
Vulnerability scanners

What else can rate limiting protect

Public APIs and authenticated APIs can be abused or misused. Sensible rate limit policies on these endpoints can reduce attack traffic and help maintain service availability. Rate limiting can protect:

APIs
Overzealous 'good bots'

How does it work?

Rate limiting focuses on a connecting client and their IP address. The following measures can be used to track client requests for rate limiting:

Concurrent connections
Connections per interval
Hits per interval
HTTP 4xx responses per interval
HTTP 5xx responses per interval
Custom criteria

How granular can rate limiting be?

Using wirefilter rules, rate limiting can identify clients from both the HTTP request and response, allowing rate limits to be separated by endpoint or behaviour. For example, the URL /api can be rate limited separately from the /login endpoint. Rate limits can also be set on response codes; for example, the endpoint /search can be protected from scraping by rate limiting clients with excessive 4xx response codes.

What types of criteria can be used to define rate limits?

Rate limits can include any information defined in an HTTP request and response, including:

IP address
URL
Query string
Headers
Response codes
GeoIP information such as ASN or country code
Parsed user agent information allowing different rules for search engines vs generic 'bots'
Metadata we make available from our BOT protection service

Defining your rate limits

Picking sensible rate limits is difficult without adequate analytics on how the web application is typically used. The Peakhour dashboard includes rate-based analytics to help with setup.

Peakhour's Application Security Platform combines high-performance delivery and cache capabilities with security controls for applications and APIs. It maintains caching performance while applying advanced threat protection. Contact our team to discuss how rate limiting can improve application performance and security posture.

Rate limiting

2022-05-16T13:00:00+10:00

Rate limits are a useful control for protecting a web application from abuse. When setting them for a web application, the key elements to consider are:

What endpoints on the web application require protecting?
Do different endpoints require separate handling?
What is the normal request rate for the entire application over a time period?
How many concurrent connections are typically used by your clients?
What errors does your API endpoint return in response to requests?

Before setting those policies, it helps to understand how rate limits protect an application from abuse or misuse, the types of attacks they can reduce, and how the rate limiting algorithm makes decisions.

What kinds of attacks are stopped by rate limiting?

When protecting an application with rate limiting, common attack scenarios include:

Brute force and enumeration attacks
Denial of Service (DoS) and Distributed Denial of Service (DDoS)
Site scraping

What else can rate limiting protect?

Public APIs and authenticated APIs can be subject to both abuse and misuse. Sensible rate limit policies can be applied on these endpoints to help prevent attacks and maintain service availability. Rate limiting can help protect these endpoints.

How does rate limiting work with user logins?

A well designed web application should allow only a limited number of failed login attempts before locking an account and requiring a password reset. This is designed to protect against brute force attacks against an account. Bots commonly attempt to brute force logins to WordPress and other popular web applications. Determined attackers can also attempt to brute force API login endpoints.

Rate limiting on a login page can be applied to the IP address of a user attempting to log in. By rate limiting by IP address, you can limit both password brute force attacks and simpler username enumeration attempts.

Using Peakhour.IO rate limiting, responses to requests can be monitored and IPs blocked for administrator-defined periods. This saves origin server resources and stops repeated attempts before they reach the application.

How could API rate limiting work?

APIs are ubiquitous across the modern web. Single Page Applications (SPAs) can be built almost entirely on REST or GraphQL APIs, while legacy applications often use form submits. Even when browsing this blog, you have consumed a range of APIs.

Because APIs are often publicly available, rate limits are commonly used to reduce abuse. Rate limiting for APIs can protect against malicious attacks. An attacker could script a bot to perform many API calls and make the service unavailable for other users, causing unplanned downtime - a layer 7 DoS or DDoS attack.

APIs

Public and private APIs can be subject to abuse or misuse. Public APIs are discoverable by anyone and can be scripted for data mining or attacks. Rate limiting these endpoints based on fair use policies is commonplace. Keeping track of this within an endpoint can be expensive, so handling it through Peakhour can offload that work from developers.

Overzealous 'good bots'

Peakhour has seen websites where up to 65% of requests come from automated bots. These bots are typically indiscriminate when mining information, and they do not carry the operational cost when your site slows down or fails. Rate limiting good bots separately from your main users helps ensure these crawlers do not stop your site from generating revenue.

How is rate limiting implemented?

Rate limiting is typically implemented using several common methods:

Fixed window

Window-based rate limiting is the simplest to understand. Fixed window limits are easy to define, such as 5,000 requests per 60 minutes. Fixed window rate limiting is subject to spikes at the edges of the window. For example, 5,000 requests in the first 5 minutes of the window may overwhelm a service.

Sliding window

A sliding window keeps much of the simplicity of a fixed window, but uses a rolling window. This allows bursts to be smoothed.

Token bucket

A token bucket is an algorithm where tokens are placed into a fixed-capacity bucket. Tokens could be defined as bytes transferred or hits to an API. When a request is considered for rate limiting, tokens are removed from the bucket. If the bucket has a sufficient quantity of tokens, the request can proceed. If there are insufficient tokens, the request is considered to be non-conforming. Non-conforming requests are dropped.

Leaky bucket

Leaky buckets are a mirror image of token buckets. Instead of removing tokens from a bucket, tokens are added to a bucket. Tokens are removed from the bucket (leaks) at a fixed rate. When a request is considered for rate limiting, it is compared to the number of tokens in the bucket. If the bucket is full, the request is considered non-conforming and is dropped.

If rate limiting is something you need to do to protect and secure your website, reach out to see how we can help.

Learn how Peakhour's Application Security Platform can improve your application's performance and security. Contact our team to get started.

Why Manage Bots?

2020-11-30T13:00:00+11:00

Modern sophisticated bad bots often work around traditional security controls. They disrupt websites, mobile applications, and APIs. Malicious bot tactics include scraping user and pricing data, creating fake accounts, running advertising click fraud, exhausting online inventories, and taking websites offline with automated DDoS attacks.

About one-quarter of all website traffic in 2019 originated from bad bots, an increase of 18 percent over 2018. Advanced persistent bots (APBs) made up seventy-five percent of that bad bot traffic as they attempted to evade detection by cycling through random IP addresses, using anonymous/residential proxies, and changing their identities (user agent). The industries hit hardest by bad bots in 2019 included financial services, education, ecommerce, and government as well as media and airlines.

“Bot attack campaigns have become big business for threat actors, and major organizations are now fighting to support legitimate users and prospects while keeping attackers out of online applications and services,” says Paula Musich, Research Director, Enterprise Management Associates.

Bots have moved from simple scripts to distributed networks of automated agents that can mimic human interactions with machine learning techniques. They can avoid detection by network security technologies that have not kept pace with the way automated agents now operate.

Reducing the damage from bad bots means using security countermeasures that detect automated traffic and make attacks uneconomic, not just visible.

Bot Countermeasure Best Practices:

The following bad bot countermeasure practices cover network security, machine learning, and behavioural analysis. The aim is to reduce the economic harm that malicious bots inflict on businesses and end-users.

Web Application Firewalls

Web Application Firewalls (WAF) are a common first line of defence that filter out harmful Layer 7 web application (HTTP) traffic using rules or policies that protect organisations against Distributed Denial of Service (DDoS) bot attacks. WAFs also protect against cross-site forgery, cross-site-scripting (XSS), file inclusion, and SQL injection attacks. A WAF is considered a reverse proxy that protects servers and can be deployed as an appliance, server plug‑in, or filter, and customised by application type or use case. WAF rules can be updated or changed based on the type of bot attack.

IP Tracking and Reputation

Sophisticated bots can be detected with network forensics by inspecting web traffic and assessing whether requests come from actual users or bad bots. Requests can be analysed using data sources including Tor/proxy IPs, IP addresses, IP geo-location information, ISP information, and IP owners. Additional sources for real-time and near-time malicious IP threat data can come from network data, CERTs, MITRE and cooperating competitors.

Client/Device Fingerprinting

Fingerprinting attempts to identify devices, including PCs, Internet of Things (IoT) devices, mobile devices and servers, using data attributes that create real-time risk profiles to stop bot attacks. Using web page access data, a bot detection fingerprinting engine generates unique fingerprints for each end-user device and checks them against bad bots that use evasion techniques, including dynamic IP addresses and anonymous web proxies.

Machine Learning

Artificial Intelligence (AI) and machine learning algorithms are increasingly used to analyse malicious bot activity and make mitigation recommendations using data from sources such as user activity history, behavioural patterns and meta-data. Machine learning can use custom-tailored algorithms to target bots and iteratively process user data and identities to discern emerging bot attack patterns from very large amounts of real-time information.

Tarpitting

Tarpitting is a bot countermeasure that delays and slows down incoming malicious traffic from suspect connections. The technique is used to increase the financial and resource costs of bot attacks in an attempt to discourage malicious actors. Bad bot tar pits can delay bot request responses or take the bad bot IP address attack source offline completely. Innovative tarpitting techniques include requiring bad bots to solve computationally complex maths challenges to access resources or websites, thereby slowing down or stopping bot activity.

User Behavior Analysis

User interaction behaviour and identifying characteristics on a web page or mobile app differ from the behaviour of an automated malicious bot. Factors such as number of pages visited per session, time spent on each web page or within a mobile app and repeat visit frequency all help differentiate authentic users from bad bots. Defeating bad bots using Behavior Analysis involves creating a user model for individual sites with historical visitor data, then checking for anomalies that may indicate bad bot activity.

Intent-based Deep Behavior Analysis (IDBA)

Compared with Behavior Analysis, Intent-based Deep Behavior Analysis (IDBA) conducts behavioural analysis at the user intent level rather than the commonly used interaction-based behaviour analysis. IDBA consists of intent encoding, intent analysis, and adaptive learning. It also employs machine learning techniques to detect bad bots emulating on-site human behaviour interactions. Bad bot mitigation techniques include limiting attempts on login pages, web authentication pages and API call authentication pages.

Rate Limiting

Rate Limiting mitigates bad bots and DDoS attacks by restricting the amount of incoming traffic accepted by specific applications and API endpoints using pre-defined bandwidth limitation policies. Web applications, GET versus POST requests, APIs that receive queries, and login credentials can all be blocked if clients, IP addresses or IP and user-agent pairs violate Rate Limiting rules. Intellectual property scraping can also be protected by Rate Limiting policies that restrict repeated image or digital downloads.

Javascript Injection

JavaScript Injection techniques can help mitigate bad bot attacks in several ways. Scripts can be placed into web applications that “fingerprint” a user’s browser to distinguish humans versus bad bots emulating “human-like” mouse movements, keystrokes or clicks. Fingerprinting detection may also involve user agent identification, HTML5 canvas and audio fingerprinting, and protocol-level fingerprinting with TLS and HTTP2. JavaScript combined with browser cookies can also be used to identify anomalous behaviour from unwanted traffic or bad bots trending over time.

ANYCast DDoS Mitigation

Anycast is an IP addressing method that routes incoming traffic requests to the nearest location or “node.” Using ANYCast for selective routing enables network load resilience against DDoS attacks by routing high traffic across multiple servers and data centres. This prevents network resources from becoming overwhelmed with malicious or irrelevant traffic.

Alternative Content Serving

Serving Alternate and Cached Content when a bad bot is detected gives organisations a way to mislead bots without blocking them altogether. For instance, e-commerce sites may fool price scraping bots by serving alternative web pages that look like legitimate pages but with higher prices. Serving Cached Content when a bot is detected also minimises load on servers without affecting site performance.

Challenges

Requests from suspected bots can be redirected to Challenges or puzzles such as a CAPTCHA, also known as a Completely Automated Public Turing test, to help identify a bad bot versus a human. Online puzzles, such as letter matching, are easy for humans to solve but difficult for automated bots. reCAPTCHA, offered free from Google, is an advanced version of CAPTCHA puzzles that require users to identify text from real-world images such as street address signs, printed books or text from paper newspapers.

Final Thoughts

Bad bots hijack user accounts, create fake accounts, scrape websites for data and personal information, flood websites with traffic through automated distributed denial of service attacks and attack public-facing APIs using constantly changing techniques. They hide behind dynamic IP addresses, change their attack signatures, mimic human behaviours, and take over vast networks of hosts and IoT devices, creating zombie machines that distribute malware across the internet. Countermeasures ranging from Web Application Firewalls to sophisticated Machine Learning algorithms form an organisation's primary line of defence against bad bots.

Website Performance testing with WebPageTest.org

2020-09-13T13:00:00+10:00

WebPageTest is one of our favourite tools for measuring website performance: webpagetest.org

WebPageTest is a web page performance testing tool developed by AOL and open-sourced in 2008. It produces its metrics using real-world browsers to load the web page being tested. It's actively maintained by Google on GitHub, so you can download and install it on your own server if you prefer. It is also free.

The key advantage of WebPageTest is that tests can be run from locations around the world, using real browsers at actual connection speeds. That lets you test performance where your users are, and see real load times rather than arbitrary scores out of 100. You can run simple tests, advanced multi-step tests, video capture, content blocking, multi-site visual comparisons, and traceroute testing. While this is still a synthetic test, it is about as close to real-world performance measurement as you can get without using RUM.

Use WebPageTest when you need to understand why a page is slow. Use the Chrome UX Report when you need to know how real Chrome users experienced the page over time. Use Peakhour's website speed comparison tool when you want a fast comparison between your domain and competitor domains using CrUX field data.

Running a Simple Test

To run a simple test:

Visit webpagetest.org
Enter the URL of the page that you want to test in the 'Enter a Website URL' field. We usually enter only the domain name, with no www and no https:// at the front. This simulates someone typing your domain into a browser address bar and captures how much time any redirects add to the page load. Sometimes redirects can be very slow.
Select the location of the test from the 'Test Location' drop down. Choose locations that reflect where your users are. Note Australian locations are at the very bottom.
Choose the browser to run the test from. Chrome is the default.
Press 'Start Test'

Note there are advanced options but for simple testing you don't need to change these.

The test will perform three visits to the specified page, with the browser cache cleared in between each visit. This simulates someone visiting your site for the first time, including the parts of the page load that only happen on an uncached visit.

Interpreting the results.

A,B,C's

The results page has a summary section at the top giving you summary grades for several categories. Here's a sample report using our website, peakhour.io. Take note of the sections at the bottom, because we'll refer to them later.

While this section isn't representative of performance it can still give you actionable information. Here's a summary of the information (ignoring security since we're concerned with speed):

First Byte Time: How long it takes the server to respond to the browser request with the first byte of information. This is the same as the Web Vital Time To First Byte (TTFB).

Keep Alive Enabled: A server option that leaves the connection from the browser to the server open for a short time, usually a few seconds, after the server has finished transmitting a request. This allows the browser to reuse a connection and saves time because it does not need to reconnect as often.

Compress Transfer: When files are transferred from the server they are compressed, usually via gzip, to make sure transfer sizes are as small as possible. It looks like we have a problem here.

Compress Images: Images are usually the largest part of a web page by transfer size. Making sure they're well compressed is important for fast sites.

Cache Static Content: Checks that static files, e.g. JavaScript, CSS, and images, have appropriate cache headers so your browser doesn't re-fetch them every time it views a page. We're developing our site so have caching turned off at the moment, hence the bad score

Effective Use of CDN: Detects whether your website is using a CDN known by WebPageTest.

This report points to problems with transfer compression and static content caching.

Performance Metrics (The Important Stuff)

The next section summarises the key performance metrics of the page load. Google's web vitals are represented alongside lab metrics such as speed index, total blocking time, and page weight.

Despite some poor marks in the grading section, the peakhour.io website loads very quickly. Its Largest Contentful Paint (LCP) is less than 1s, well below Google's 2.5s target for a good result. We've already covered them in our web vitals, which also defines ideal values for each metric, so we won't cover them again here. One detail worth noting is that Total Blocking Time is still a useful lab signal for main-thread blocking. It is not the same as Interaction to Next Paint, because INP is measured from real user interactions in the field, but it helps identify JavaScript and third-party work that can harm responsiveness.

The Detail Section

Remember the sections immediately below the grade summaries? Now we're going to click on the 'Details' section. The part we want to highlight here is the Waterfall.

At the top of the waterfall chart is a colour key for reading the diagram. The key concepts are:

dns (Dark Green): This measures the time it takes for the browser to look up the location of your server.

connect (Orange): This measures the time taken to establish the TCP connection to download a resource. It should only appear on the first resource for a given host. Remember the keep-alives grade: if that is turned off, there will be more connections here.

ssl (Dark Purple): Any resources that are loading from a secure website will need to be processed as such – the purple will signify how long it is taking to connect to that SSL item.

The Waterfall View

The waterfall is an easy-to-read view of how your website loads, with all the resources listed in the order they're requested, along with the time taken to load each resource. You can click on any resource to view the request/response headers, file size, protocol and more.

In the example above the first line has a yellow background, which signifies a redirect. The three lines towards the bottom with red backgrounds signify 404 not found errors, which need to be fixed. The colourful vertical lines indicate where major load events, like first paint and document loaded, happen.

The other thing to look for is any resource that takes a long time to load. In our example the 2nd row, which is the main HTML document, only took 149ms, which is fast. A lot of websites take 2-5s to load the main document, putting the user experience under pressure before the rest of the page has started. The main concerns here are rows 27 and 28: two SVG images that took around three quarters of a second to load.

The Performance Review Section

The final section we're going to cover is the Performance Review section.

This section gives more detail for the performance grades at the top of the report. In the earlier example Peakhour.io scored poorly for Compress Transfer and now we can see why: we're not compressing SVG images, something that can save 234kb of file downloads.

Summary

WebPageTest gives you enough detail to see where a page is losing time, not just whether it passed a headline score. There is more in the tool than we've covered here, but this is enough to test a website, read the main report sections, and identify practical issues to fix. If you are starting with a market view rather than a single waterfall, compare your field data first with the website speed comparison tool, then come back to WebPageTest for the diagnostic pass.

Malicious Bot Threats

2020-08-12T13:00:00+10:00

Bots are software applications that automate repetitive tasks without human interaction. They have become part of the normal infrastructure of the internet. Some bots are useful; others are bad bots. The latter are the concern for application and security teams.

Bad bots keep changing and are increasingly difficult to detect. They can cause significant financial damage to organisations by disrupting online operations, overwhelming websites with traffic, and stealing information such as web content and ecommerce pricing data.

Bad Bot Types

Bad bots span a wide range of attack capabilities and scenarios. The following are the main categories these attacks fall into:

Spam Bots

Spam bots typically target blog comment sections, community portals and lead generation forms with 'garbage' or fake content. They can also insert unwanted ads, malicious phishing links and banners into real-time conversations to disrupt the service and attack users.

Scraping Bots

Price, content and inventory scraping bots steal prices and product listings. This can damage an ecommerce site's revenue stream and harm SEO rankings when duplicate content appears on competitor and bogus sites. These bots also scrape product reviews, news, product catalogues and user-generated content. Scraper bots can harvest email addresses, images and text from victim websites, then repurpose that material to pose as legitimate web pages.

Credential Stuffing Bots

Credential Stuffing Bots attempt to use login details from other sites, or run brute force guessing attacks against customer and admin accounts. If successful, they can make purchases, harvest personal information and purchase histories, make unauthorised cryptocurrency transactions, and transfer reward points and money to gift cards and air miles.

Ad Click Fraud Bots

Ad Click Fraud Bots can sabotage competitors by clicking on their ads to drive costs up and exhaust budget caps. They can also be used to scam advertisers with fake websites and ad clicks that pay the fraudster directly. In both scenarios, bots automatically generate interactions or 'clicks' with ads, promotions and media.

Credit Card Stuffing Bots

Carding bots make repeated attempts to authorise stolen credit card credentials. This can leave merchant payment processors with chargebacks and penalties, and may ultimately result in the victim merchant being prevented from accepting credit cards altogether.

Inventory Denial Bots

Cart Abandonment and Inventory Exhaustion bots automatically add hundreds of products to ecommerce shopping carts, then abandon them. This can block consumers from buying products, reduce sales, manipulate conversion rates and damage a brand’s reputation.

DDoS Bots and Botnets

Distributed Denial of Service (DDoS) attack bots and botnets are made up of thousands of compromised computers or Internet of Things (IoT) devices called "zombies". They can slow down a website or take it offline completely by flooding sites with massive amounts of artificially generated traffic. Researchers have found cybercriminals advertising DDoS services on the dark web with basic fees to attack unprotected sites ranging from $50 to $100, while an attack on a protected site can reach $400 or more.

Ticket Scalping Bots

Ticket scalping bots automatically buy tickets, enabling malicious users to resell them at a higher price. Examples include using a bot to purchase concert tickets for major events the minute they go on sale.

Fake Account Creation Bots

Fake Account Creation bots create fake accounts for criminal activities such as content spam, cryptocurrency laundering and malware distribution. Fake accounts can compromise brands and attack users with malware such as ransomware.

Hacker Bots

Hacker bots can distribute malware, attack websites and compromise entire networks by exploiting security vulnerabilities and injecting code into victim sites. Hacker bots can also perform DDoS attacks across web proxies with browser-like signatures to disrupt business operations.

Impersonator Bots

Impersonator bots copy human computer interactions and behaviours to fool users and bot mitigation defences while they conduct malicious activity. Impersonator bots also include propaganda bots that influence political opinions on platforms such as Facebook and Twitter. According to researchers at the University of Southern California who studied bot use during the 2016 U.S. Presidential election, “the presence of social media bots can indeed negatively affect democratic political discussion rather than improving it, which in turn can potentially alter public opinion.”

The Growing Threat

A report from Imperva found that roughly one-quarter of all website traffic in 2019 originated from bad bots, an increase of 18% over 2018. 75% of that bad bot traffic is made up by Advanced persistent bots (APBs) that attempt to evade detection by cycling through random IP addresses, using anonymous proxies, and changing their identities. The industries hardest hit by bad bots in 2019 included financial services, education, ecommerce and government, as well as media and airlines.

Companies offering "Bad Bots as-a-Service"* are also gaining ground. These data scraping services sell bots as easy-to-use packaged products that provide pricing and competitive intelligence, alternative data for finance, or competitive insights managed by Web Data Extraction Specialists and Data Scraping Specialists.

Malicious bot-for-hire services also offer personal and financial data harvesting, brute-force login services, ad click fraud, spamming services, transaction fraud services, and Distributed Denial of Service (DDoS) attacks.

Final Thoughts

Bad bot activity continues to increase, so websites need security controls that can identify and stop them. Our next article on bots will go over the common countermeasures used to combat bad bots.