Our recent survey found that only 15% of Australian organisations use residential proxy detection. That leaves many teams relying on controls that were not built for current proxy traffic, especially where CGNAT and NAT make IP-level decisions unreliable.
The Shortcomings of Traditional Methods
Legacy bot protection providers often combine IP reputation, network characteristics, header analysis, and JavaScript-based checks to identify proxy usage. These methods struggle against well-run residential proxies:
- IP and ASN categorisation: Ages quickly as new proxy networks emerge.
- Network-level checks: Well-configured proxies can work around them.
- Header analysis: Proxies can alter HTTP headers to mimic legitimate traffic.
- JavaScript-based detection: Struggles against headless browsers and leaves API endpoints vulnerable.
The CGNAT and NAT Challenge
A practical limit of traditional methods is their inability to distinguish legitimate traffic from proxy traffic when both originate from the same IP address. Carrier-Grade NAT (CGNAT) and Network Address Translation (NAT) make this common:
- CGNAT: Used by ISPs to conserve IPv4 addresses, resulting in multiple users sharing a single public IP.
- NAT: Commonly used in home and business networks, allowing multiple devices to use one public IP address.
As a result, legitimate users and residential proxy traffic can appear to come from the same IP address. IP reputation and geolocation alone cannot separate these traffic types.
This creates a difficult tradeoff:
- Blocking suspicious IPs risks denying service to legitimate users.
- Allowing all traffic from these IPs opens the door to potential abuse via residential proxies.
Traditional methods cannot reliably pull apart these different types of traffic, so teams either block too much legitimate traffic or allow too much proxy traffic through.
The Need for Sophisticated Network Fingerprinting
To detect and mitigate residential proxy threats while allowing legitimate traffic from shared IPs, detection needs to move beyond IP identity. Network fingerprinting addresses the limits of traditional methods:
- Deep packet inspection: Analyses traffic patterns and characteristics beyond basic IP or header indicators.
- Protocol behaviour analysis: Identifies subtle anomalies in how network protocols are implemented across the proxy chain.
- TLS fingerprinting: Examines unique characteristics of TLS handshakes to detect proxy usage.
- Timing analysis: Measures small differences in network latency that can indicate the presence of a proxy.
Used together, these techniques can detect proxy usage on a per-connection basis for both web traffic and API calls, even when traffic originates from shared IP addresses. This approach provides several advantages:
- Improved accuracy: Significantly reduces false positives and negatives compared to traditional methods, including in CGNAT and NAT scenarios.
- API protection: Secures API endpoints, which are often overlooked by JavaScript-based solutions.
- Real-time detection: Allows for immediate action against detected proxy usage without impacting legitimate users.
- Adaptability: Can be updated to detect new proxy technologies as they emerge, regardless of IP sharing.
Implementing Effective Proxy Detection
To implement proxy detection that accounts for modern network complexity, organisations should consider the following:
- Deploy solutions that use network fingerprinting techniques capable of distinguishing between different types of traffic from the same IP.
- Ensure protection covers both web applications and API endpoints, as both are vulnerable to proxy-based attacks.
- Implement real-time mitigation capabilities to respond swiftly to detected threats without impacting legitimate users.
- Regularly update and tune detection algorithms to keep pace with evolving proxy technologies and network architectures.
Together, these practices improve an organisation's ability to detect and mitigate residential proxy threats across credential stuffing, account takeover, and related activity, while keeping access available for legitimate users.
Learn more about our proxy detection solution, which uses network fingerprinting to address the challenges posed by CGNAT and NAT.
For more detail, explore our learning resources:
- Understanding Residential Proxies
- Network Fingerprinting Techniques
- In-Depth Review: TLS Fingerprinting
As proxy technologies and network architectures change, detection and mitigation need to change with them. Network fingerprinting gives organisations a more reliable way to identify residential proxy abuse without treating every shared IP as suspicious.
