Penetration tests are a powerful process. But testing campaigns can deliver misleading results. Limitations on penetration testing scope, timing, and approach mean that pen testing is not a comprehensive solution for attack surface management.
Countless organizations use penetration testing to prove their security posture works. 70% of organizations with 3,000+ workers conduct penetration tests to assess their security posture. It is also one of the most ROI-effective security investments an organization can make. Conducting pen testing reduces the cost of data breaches by an average of $156,659.
However, organizations should be careful not to treat penetration testing as a one size fits all solution for understanding and managing their attack surface.
While pen tests are ideal for helping meet compliance requirements and testing new security configurations or deployments, no penetration testing campaign will give your organization a comprehensive overview of its attack surface.
Here’s why.
Penetration Testing Has Limited Scope
By design, penetration tests miss your most vulnerable assets, including shadow IT and unknown/forgotten assets.
Most penetration tests use a “white box” approach. Security teams provide a penetration tester with network access and a list of assets to be attacked and avoided. This methodology tests the effectiveness of particular controls or provides answers to defined questions, i.e., is a WAF able to protect a web application after a recent configuration change?
Keeping tests within strict bounds is cost-effective and reduces the risk that a test will impact an organization’s operations.
The downside of white box and slightly broader “gray box” penetration tests is they take a narrow view of an organization’s IT environment, usually ignoring unknown assets altogether. According to a recent report, 69% of organizations have been compromised by an asset they didn’t know existed.
The IT assets and attack vectors security teams know about are a subset of what’s actually plugged in.
Shadow IT is a significant source of breach risk. Shadow IT is estimated to make up almost half of all IT spending in a typical organization. In a 2022 KnowBe4 report, 1 in 2 employees admits to using unauthorized file services.
A standard pen test will not cover shadow IT or unknown or forgotten assets lurking in your network. This means that a testing report’s recommendations and findings will ignore the actual context that assets exist within. With lateral movement happening in almost 45% of attacks, this is a dangerous blind spot for organizations to have.
Irregular Tests Miss Real Attack Vectors
Penetration tests are not done often enough or long enough to replicate real-world threat behavior.
A penetration testing report might feature the disclaimer that its contents represent a “point in time.” This point in time refers to your network environment and the methodologies and exploits used by pen testers (and attackers). Both of these changes more rapidly than any organization can test.
Most organizations that perform pen tests do so less than once a quarter; 39% run penetration tests only once or twice a year.
The speed at which IT environments and the threat landscape surrounding them is evolving means that irregular tests go out of date quickly. 442,151 new malware strains were detected last year, double the number in 2020. Meanwhile, rapid cloud migrations and dev-ops-led development have left most organizations with sprawling network environments. No organization is testing often enough to mitigate these risks.
Penetration tests also don’t last long enough to replicate real attack behavior. A test engagement might feature only one week of actual testing. Even in a month-long contract, reviewing documentation, compliance, and internal and external meetings can take up considerable time. Report writing alone can occupy more than 40% of a typical test engagement.
Armed with highly evasive malware and custom exploits, attackers spend far longer in their target networks than penetration testers ever do. Research by Sophos shows that the average dwell time for threats is 15 days—36% more than the previous year.
There’s No One-Size-Fits-All Approach
When defenders don’t know how to plan tests or react to pen-test recommendations, pen-testing engagements can fail to deliver meaningful results.
Testing is expensive and can be complicated. Most organizations struggle to find skilled people to conduct penetration tests in-house. This means most penetration testing is contracted out.
In the last few years, countless companies have sprung up to offer penetration testing services. Many IT consultancy companies have also added penetration testing to their list of services, but service quality can vary widely, and not all testing engagements will deliver actionable results.
Skills shortage is a real problem across our industry, and it’s not uncommon for large firms to put experienced pros on sales calls only to bait and switch clients with junior operators when it comes to testing time. Many will only run automated tools and then take a copy/paste approach to the reporting processes they use for different clients.
Even top-quality testing organizations need to rely on the organization contracting them to know exactly what they want to achieve with a test and how to customize the findings they produce.
Defenders need to be ready to check a report’s advice against what is possible and ask testers for alternative solutions when a recommendation clashes with their capabilities. For example, a test report might recommend replacing EOL software on a critical server, something that the organization cannot realistically do without massive disruption.
The onus is on organizations that contract penetration testers to ensure the quality and fit of a testing campaign. No test provider will provide a perfect fit testing regime out of the box.
Boost Testing with CART
Use Continuous Automated Red Teaming (CART) to test your networks like real attackers do.
As they consider whether their security posture is effective, every organization needs another layer of testing. Solutions like CART are essential for this purpose. Instead of providing a snapshot view into how a handful of controls function, Randori’s CART helps teams see the bigger picture.
Learn how Randori can help your organization go beyond penetration testing with a free zero-friction external assessment.
