Your Scanners See a Different Internet Than Your Users Do

A few years ago, cloaking was mostly a black hat SEO problem. Someone would serve one version of a page to Googlebot and another to human visitors, and the worst outcome was a spammy search result. It has since become one of the more effective evasion techniques in the phishing and malvertising toolkit, and most security teams are still measuring their external exposure from a vantage point that attackers stopped caring about some time ago.

The mechanics are not sophisticated. A phishing kit, a malvertising redirect chain or a fake software download page runs a check on the inbound request before it decides what to return. Source ASN. IP reputation score. TLS fingerprint. User agent string. Screen dimensions and colour depth. Timezone offset. Whether the browser reports plugins and fonts consistent with a real desktop. Whether the request arrived at three in the morning from a Frankfurt hosting range with no cookie history and no referrer.

If the profile looks like automation, the visitor gets something harmless. A parked domain notice. A holding page for a plumbing company in Leeds. Something so dull that the crawler logs a clean verdict and moves on to the next item in the queue.

If the profile looks like an actual person on an actual broadband connection in Ohio, the visitor gets the credential harvester.

The consequence is a widening gap between what threat intelligence platforms observe and what employees, customers and partners actually run into. That gap is not accidental. It is engineered, deliberately and specifically, around the infrastructure that defenders use to look.

Why the ASN gives you away

Almost every automated security scanner in production egresses from cloud infrastructure. That is a sensible architectural decision. It is elastic, it is cheap, it is easy to operate, and it scales without anyone having to think about physical hardware. It is also trivially identifiable from the other side of the connection.

Public ASN registries make the mapping between IP ranges and hosting providers a matter of open record. Anyone can pull the allocations for the major cloud providers, the well known scanning services, the university research crawlers and the commercial threat intelligence vendors, then load them into a filter at the edge of their own infrastructure. Several phishing as a service kits ship with that list pre populated and offer updates as a paid feature of the subscription. There are commercial anti bot products, sold openly and used legitimately by real businesses, that adversaries repurpose for exactly this. The tooling is mature and it is cheap.

So when a security team runs a takedown workflow against a suspected brand impersonation domain and the crawler comes back with nothing malicious, that verdict carries a lot less information than the dashboard suggests. The absence of evidence was manufactured on purpose, in advance, by someone who anticipated the exact request the crawler would make.

This is a problem that reaches well beyond phishing.

Brand protection and takedown work depends on evidence. Registrars and hosting providers usually want a screenshot or a captured payload before they will act on an abuse report. If the malicious content only renders for residential visitors in the target geography, the report gets closed as unsubstantiated and the domain stays live for another six weeks.

Ad fraud and malvertising campaigns are routinely geofenced and device fenced. A campaign aimed at mobile users in three European markets is effectively invisible to a scanner egressing from a datacentre in Virginia. It will still be invisible next week, and the reporting will keep showing green.

Third party script monitoring runs into the same wall. Skimming code injected into checkout flows increasingly fires only for sessions that pass fingerprinting checks, which is a direct response to the client side monitoring tools that got deployed after the last serious wave of Magecart incidents. The defenders adapted, then the attackers adapted to the adaptation.

Validating a data leak has a version of this problem too. Working out whether exposed credentials or assets are genuinely reachable often means reaching them the way an outsider would, from outside, rather than from a corporate egress point that may sit on an allowlist or may be blocked outright.

Even ordinary competitive and geographic testing suffers. Plenty of organisations have discovered that a service they believed was globally available has been quietly blocked in a market for months, because nobody was checking from inside that market.

Looking from somewhere the attacker did not exclude

The fix is easy to describe and awkward to build. Security teams need external observation points that are indistinguishable from ordinary user traffic, located in the places where their users actually are.

In practice that means moving away from a single cloud hosted crawler fleet and towards a distributed set of egress paths that carry residential ASN attribution. Most teams building this capability use static residential ISP proxies as the transport layer. These are IP addresses issued by genuine internet service providers rather than by hosting facilities, so they pass ASN based filtering, while still offering the stability and throughput that automated collection needs. Because the addresses are static rather than rotating, an investigation keeps a consistent identity across a session. That matters more than people expect. A target site that profiles behaviour over a sequence of requests will notice if the source address changes halfway through a login flow, and plenty of them do exactly that check.

The distinction between real ISP addresses and relabelled hosting space is worth being difficult about, because the market is genuinely muddy. A meaningful share of what gets sold as ISP egress is datacentre allocation with a different label on the invoice, which puts you right back where you started with a higher bill. Verification against ISP registration data before deployment is not a procurement formality. It is the whole control. If the addresses do not resolve to a real service provider in the databases the target site is consulting, you have bought nothing.

A few things tend to determine whether this works or turns into shelfware.

Geography has to match your exposure rather than your head office. If you have customers in twelve countries and you observe from two, you are producing a map with ten blank quadrants and calling it coverage. Cloaking logic is very often conditional on country, and sometimes on city or on mobile carrier. Campaigns run for weeks in secondary markets before they show up in the primary one, which is usually the point of starting there.

Session realism matters as much as address provenance. A residential IP paired with an obviously headless browser, a default viewport, no font entropy and a request cadence of forty pages a second still fails the check. The address is necessary and it is not sufficient. What carries the session is consistency across the whole stack, from the TLS handshake up through the JavaScript environment to the timing between clicks. Teams that get the network layer right and ignore the browser layer usually conclude that the approach does not work, when what actually happened is that they solved half of it.

Treat the egress infrastructure as in scope for governance from day one. Any capability that routes traffic outside the corporate perimeter attracts fair questions from risk, legal and compliance, and the questions do not get easier if you answer them a year later. Who owns it. What is it permitted to touch. Where do the logs go. What happens if someone on the team uses it to browse something they should not, or to reach a target that is not yours. Scope it, log it, put a named owner on it, and review it the way you would review any other privileged tool. A monitoring capability that bypasses your own controls in order to observe the outside world becomes an insider risk vector the moment it stops being managed.

Cost management is the boring one that sinks projects. Residential egress is more expensive per request than a cloud crawler, sometimes by an order of magnitude, so you cannot simply point your entire existing scanning volume at it and hope. The workable pattern is tiered. Cheap infrastructure does broad discovery and enumeration, and residential egress gets reserved for verification of the things that discovery flagged, plus a scheduled sweep of the assets that actually matter. Most teams end up running maybe five per cent of their total request volume through the expensive path, and that five per cent is where nearly all the useful findings come from.

The metric that is actually being contested

The most useful change here is not technical.

Most external monitoring programmes report on coverage. How many domains are being watched. How many assets were enumerated this quarter. How many alerts were triaged and how quickly. These are all real numbers and none of them answer the question that matters, which is whether what was observed resembles what a real user would have seen.

Coverage and observability are different things, and only one of them is being actively contested by someone with an interest in the outcome. Nobody is working to make your asset inventory less complete. Quite a few people are working to make your scan results look clean.

Here is a reasonable maturity test, and it takes about an afternoon. Pick five or six recent phishing, malvertising or impersonation incidents that affected your organisation and were discovered through some channel other than your monitoring stack. A customer complaint. A helpdesk ticket. A notification from a bank or a registrar or someone on the security team who happened to be looking at their personal email. Then go back and check whether your automated tooling had previously scanned that infrastructure, and what it returned.

If the answer is that the tooling scanned it and called it clean, you have not found a coverage gap. You have found that your tooling is being reliably deceived, and buying more of the same tooling will not change the result. It will just generate the same clean verdict faster and across more assets.

That is an uncomfortable finding to write up, particularly if the tooling in question was expensive and someone senior sponsored it. It is also the single most valuable thing an external monitoring programme can learn about itself, and it is cheap to check.

The internet your scanners see is a curated exhibit. It is maintained by people who have a direct interest in what you conclude from it, and who have put real engineering effort into shaping that conclusion. Assuming otherwise is a comfortable position to hold. It is getting more expensive every quarter.

Views expressed are the author's own. SecuritySenses welcomes contributions on threat detection, exposure management and security operations.