The Discrepancy Between the Results of Compliant Penetration Tests and What Really Defines an Organization's True Attack Surface

Organizations are investing large sums of money and resources in obtaining ISO 27001 certifications, SOC 2 attestations and performing yearly penetration tests, yet six months after the fact they hear about a breach involving one of their organizations in the media. This trend is so common, that many incident response professionals have used this as a recurring example when conducting post-breach analysis.

Although very few organizations’ problems lie with the standards themselves (ISO/IEC 27001:2022 provides a good outline for developing an Information Security Management System), the problem lies with translating a technical testing process into some type of auditable artifact. A penetration test that is designed to be compliant will generate a document that appears defensible, but does not represent how a determined attacker would evaluate the same environment.

This paper will look at where the process of penetration testing driven by regulatory requirements differs from the true definition of an organization’s attack surface, what ISO 27001:2022 really requires and ways security teams can bridge the gap while continuing to pursue their current certifications.

Why Passing an Audit Does Not Equal Being Secure

The empirical record on this is fairly consistent. The Verizon 2024 Data Breach Investigations Report analysed more than 30,000 security incidents and 10,000 confirmed breaches, and found that exploitation of vulnerabilities as an initial access vector nearly tripled year over year, driven largely by edge devices and web-facing infrastructure. Many of the affected organizations held current certifications at the time of compromise.

ENISA reaches similar conclusions. The ENISA Threat Landscape 2024 report identifies ransomware, DDoS, and data-related threats as the dominant categories across the EU, with initial access frequently obtained through exposed services, weak identity controls, and supply chain paths. These are the categories that a narrowly scoped annual pentest tends to underweight, because they sit at the edges of what an auditor typically asks to see evidence for.

Two structural factors explain the gap. First, audit scope and attack scope rarely match. An auditor wants to confirm that a defined set of controls exists and functions. An attacker does not respect that boundary and will pivot through whatever is reachable. Second, audit timing is predictable. Annual or biannual tests produce a snapshot, while real attack surface changes weekly through deployments, mergers, SaaS onboarding, and shadow IT.

What ISO 27001:2022 Actually Requires From Technical Testing

ISO/IEC 27001:2022 does not prescribe penetration testing as a mandatory control in the same rigid sense that PCI DSS does. The relevant reference is Annex A control 8.8, which addresses management of technical vulnerabilities, alongside 8.29 on security testing during development and acceptance. The ISO 27001:2022 standard leaves method selection to the organization, provided the choice is justified by risk assessment and produces evidence that vulnerabilities are identified and treated in a timely manner.

That flexibility cuts both ways. A well-designed program uses it to run targeted, threat-led assessments where they matter most. A weaker program uses it to commission a minimal external test that satisfies the auditor and moves on. The difference between those two outcomes is usually decided before the first packet is sent, in the scoping conversation. A practical account of how these conversations play out, and how they influence both the technical work and the audit outcome, is described in ISO 27001 pentests in practice, which walks through the tension between auditor expectations and actual risk reduction.

The 2022 revision of the standard also strengthened the link between threat intelligence (control 5.7) and vulnerability management. Read together, 5.7 and 8.8 imply that testing should be informed by current adversary behaviour, not by a generic checklist repeated from the previous year. In practice, this means aligning test cases with frameworks for example like MITRE ATT&CK, which catalogues techniques observed in real intrusions, rather than restricting scope to a static list of CVEs.

Threat-Led Testing vs. Checkbox Pentests

The methodological difference between the two approaches is well documented in public standards. The Penetration Testing Execution Standard (PTES) defines seven phases, from pre-engagement interactions through intelligence gathering, threat modelling, vulnerability analysis, exploitation, post-exploitation, and reporting. NIST offers a similar structure in SP 800-115, Technical Guide to Information Security Testing and Assessment, which distinguishes between vulnerability scanning, penetration testing, and red team exercises as distinct activities with different objectives.

A compliance-driven engagement often collapses those phases. Threat modelling is skipped or replaced with a scoping call. Post-exploitation is excluded because it makes stakeholders uncomfortable. The result is closer to an authenticated vulnerability scan with manual verification than to a penetration test in the sense PTES describes. That output can still be useful, but it should not be confused with an assessment of how an adversary would compromise the business.

Threat-led testing, by contrast, starts from a hypothesis about who might attack the organization and how. Financial services firms in the EU have formalized this approach through TIBER-EU, the European framework for threat intelligence-based ethical red teaming, coordinated by the ECB. TIBER-EU engagements are lengthy, expensive, and not appropriate for every organization, but the underlying logic scales down. Even a small company can ask its testers to model a specific adversary, choose techniques from ATT&CK that this adversary is known to use, and evaluate detection and response along the way, not just prevention.

The reporting difference is visible on the page. A checkbox report lists findings ranked by CVSS. A threat-led report describes attack paths, explains which controls broke down at each stage, and quantifies the likelihood that the same path would be replicated by the modelled adversary. Both include a severity table. Only the second answers the question the board actually asks, which is whether the organization would notice an intrusion in progress.

Building a Testing Program That Reflects Real Risk

Closing the gap does not require abandoning ISO 27001 or spending red team budgets. A few adjustments to how testing is planned and consumed tend to produce disproportionate improvements.

Scope should follow the attack surface, not the organizational chart. Modern environments include SaaS tenants, cloud accounts owned by product teams, contractor laptops, and OAuth integrations that predate the current CISO. External attack surface management tools have become inexpensive enough that maintaining a current inventory is realistic even for mid-sized organizations. The OWASP Attack Surface Analysis Cheat Sheet offers a practical starting point for defining what belongs in scope.

Testing frequency should be tied to change, not to the calendar. A quarterly external test plus continuous automated scanning catches more than a single annual engagement of twice the duration. For internet-facing applications with frequent deployments, integrating security testing into the CI/CD pipeline, as encouraged by ISO 27001 control 8.29, closes the window between introduction and detection of vulnerabilities.

Findings should be tracked as risks, not as tickets. A vulnerability without context tells the operations team to patch something. A finding framed as an attack path, with affected assets, business impact, and detection status, gives the risk committee something to prioritize. This is where the ISO 27001 risk register earns its keep, provided it is populated with real technical detail rather than generic control language.

Retesting matters more than most reports admit. The CREST Penetration Testing Guide recommends verifying remediation before closing findings, a step that is routinely skipped when budgets tighten. Without retesting, the delta between what the report said and what actually got fixed remains invisible until the next annual engagement, or until an incident.

Finally, testing results should influence the ISMS, not sit beside it. Control 10.1 on continual improvement is not decorative. When a pentest reveals that identity federation was misconfigured across three business units, that finding should trigger a review of the relevant policies, awareness training, and monitoring rules, not just a patch on the affected tenant. Organizations that treat their testing program as an input to the management system, rather than as an obligation the management system imposes on them, tend to see their finding counts drop over successive engagements. Those that treat it as an annual exam tend to see the same findings recur, with different asset names.

The certificate on the wall is worth having. It is not the same thing as knowing what an attacker would do next week, and no serious security program should confuse the two.