Read The Total Economic Impact™ of The CyCognito Platform — a Forrester study. Cost Savings And Business Benefits Enabled By The CyCognito Platform. More...
If you are like many CISOs, you feel pressure to increase the value of your security testing budget. And if you are one of the 53% of enterprises reporting stagnant or decreasing budgets in 2024, you have even more work cut out for you.
Increasing testing value requires a re-evaluation of nearly everything. Tackle tool sprawl. Optimize workflows. Reduce false positives. Review cloud spend. All while demonstrating ROI even in the absence of incidents.
This post is about ways to reach these goals. We will then work through an example to provide some color. And at the end I will talk about how CyCognito can help accelerate them.
The good news is that whatever path you take, you will have a stronger, more valuable security testing program with data to back it up.
Defining Security Testing Value
Let’s look at value first. Traditionally, value is understood as a ratio of benefit to cost. Here we define value as a ratio of tool effectiveness to cost — how much each dollar spent on testing tools helps reduce risk and mitigate threats.
Both terms are expressed in currency. Your decisions must achieve the right balance of cost and effectiveness. Staff health needs to be at the forefront – there is a limit to how much costs can be reduced without creating an unhealthy work environment. The bare minimum is a ratio greater than 1, where “$ effectiveness” exceeds “$ spend.”
Strategies for Reducing Costs
In many organizations, reducing hard costs—for example, licensing, compute, storage, and telemetry — are the most straightforward path to increased value. This makes sense – a good movie at $5 a ticket is better value than the same movie at $20 a ticket.
Consider these as a starting point:
Eliminate redundant or unused tools: Tool sprawl is a real issue. You may have tools that serve similar purposes or that aren’t fully utilized. Use them or remove them.
Consolidate tech stack: Security tools have evolved considerably. Consider replacing a set of technologies with a single modern equivalent.
Optimize cloud storage and telemetry costs: Many organizations over-collect and over-store data in SIEMs and cloud storage — reduce to essential data only.
Right-size license counts: Ensure that license counts align with what you need. For example, you may be paying for testing 100% of assets but only testing 10%.
Negotiate licenses and contracts: Work with key vendors to take advantage of cost savings measures that may be in place, such as volume discounts or usage-based pricing.
Scale back customer support levels: If the top-tier SLA doesn’t align with your team’s needs, consider scaling down to a more affordable level.
Reduce time and effort: Time spent configuring and scheduling tests, updating systems and reviewing data adds up. Find bottlenecks and fix (or replace the tool).
A note on labor costs. While labor is technically part of hard costs, for this example we will focus on increasing the efficiency of existing staff, not reducing headcount. Security testing and incident response requires healthy people, or it won’t work no matter what you do.
Unused tools? Time to let them go.
An average organization manages 76 security tools. Even a third of that number is considerable. Some tools are there because it’s difficult or expensive to replace them. Others because they’ve always been there. Regardless of the reason, if you’re spending money on tools that add marginal value, get rid of them.
How to Identify Tools to Eliminate
Finding tools to remove requires a structured approach:
Inventory: Get a clear picture of what’s currently in use.
Evaluate utilization: Capture the tool’s coverage, frequency and accuracy.
Look for functionality overlaps: Identify tools that do the same thing.
Then, talk to your testing teams about what is needed. Have them rank tools from 1 to 3, with 3 being critical and 1 being unhelpful. Provide why. They likely have strong opinions about what is working and what isn’t.
Strategies for Increasing Tool Effectiveness
Next, let’s look at effectiveness. A tool’s effectiveness is the measurable benefit it provides. The more effective the tool, the higher its value.
Here are ways to increase effectiveness:
Expand coverage: Increase the test scope to more exposed assets.
Adjust frequency: Set the test cadence to match your ability to respond and your risk tolerance.
Fine-tune policies: Adjust tool policy and payload to improve accuracy and reduce false positives.
Automate and integrate: Take advantage of integrations to streamline data sharing between tools and people (for example, incident response).
The irony is that these improvements can add costs (usually labor) so you have to weigh the benefits against the expenses. Automation should take precedence–a single short term investment in automation pays dividends long term.
Here are some questions to consider when evaluating the effectiveness of tools. Make sure to add numbers to the assessment so that it can be measured.
Does the tool reduce incident volume? How much?
Does the tool reduce mean time to detect/respond? How much?
How much time does it take to manage?
Does it produce false positives? How many?
What asset coverage and frequency is the tool applied?
Does the data require additional manual effort to be useful?
Security teams have strong opinions about the tools they use. If a tool is effective but difficult or time consuming to use it isn’t as valuable unless it is essential for occasional use. But even then, if it’s expensive, consider alternate tools that can get the job done on an ad-hoc basis.
Lets Walk Through an Example
Assume an organization with several thousand externally-exposed web applications and network services. This organization uses vulnerability scanning, app sec, manual pen testing, security ratings and bug bounties.
Calculate Costs
Lets list the technologies in a table along with license cost and the number of staff dedicated to it. License costs are examples only and labor costs are simplified to $166K per FTE, fully loaded.
Testing
Total Annual License $
Dedicated FTEs
Vulnerability Scanning with Add-on Modules (e.g. Tenable, Rapid7)
$85K
3 analysts
DAST (e.g. Burp Suite Pro, Invicti)
$51K
1 engineer, 1 architect
Manual Pen Resting (e.g. Metasploit, Nessus Pro)
$38K
2 pen testers
Bug Bounty
$50K
–
Security Ratings Service
$30K
–
SUB-TOTALS
$254K
$1,162K
TOTAL
$1,416K
Determine Effectiveness
Effectiveness is the savings you receive when the security testing tool uncovers a vulnerability early. Detecting a critical/high vulnerability early eliminates associated incident costs. Assumptions for calculating effectiveness include: 20 vulnerabilities per asset annually, 1% of those vulnerabilities are critical/high severity, 20% of those critical vulnerabilities lead to incidents and an average incident cost of $26,000. The formula is (Number of critical/high vulnerabilities) × (20% incidents) × $26,000.
Test Type/Approach for external testing
Total Number of Critical /High Vulnerabilities Discovered
Effectiveness value ($)
Vulnerability Scanning 80% coverage, 26X per year
420
$2,184K
DAST 10% coverage, 4X per year
60
$312K
Manual Pen Testing 10% coverage, 4X per year
100
$520K
Bug Bounty 1X per year
20
$104K
Security Ratings Service
3
$16K
SUB TOTAL
$3,136K
Incidents from unmanaged or inadequately tested assets
14
– $364K
TOTAL
$2,772K
Calculate Value
Based on this data, the example organization has a security tool effectiveness of $2,772K/$1,416K or $1.96.
In other words, for every dollar spent the example organization is receiving nearly $2 in value based on incident count reduction.
What is the right number? Clearly, higher is better. But there isn’t a single number to pursue. You must create a benchmark value for your organization and use it for comparison after changes. More details are better – for example false positives per technology to represent higher costs from time chasing inaccuracies.
Tip → Use this workflow to illustrate ROI even in the absence of incidents.
Boost the Value of Your Security Testing with CyCognito
CyCognito increases your testing effectiveness by eliminating gaps that lead to incidents. It allows a reduction in costs, including redundant or underperforming tools. And since it is delivered as a fully automated SaaS, your teams spend time on results, not configuration.
With CyCognito:
Increase testing effectiveness
Significantly enhance the volume and rate of your security tests (including app sec on all exposed web apps using DAST)
Remove gaps in testing with a dynamic asset inventory across all business units and brands
Increase effectiveness of manual tests by starting with CyCognito’s recon, test results and evidence
Increase integration and workflows, such as with ServiceNow, Workato and Jira
Reduce costs
Eliminate overlapping vulnerability scanning and security rating service add-ons
Reduce services like bug bounties that are infrequent and aren’t required for compliance
Reduce wasted time on false positives
Reduce time collecting evidence: links, URL patterns, certificates, headers, deployed software, screenshots, and more – 175+ data points
Stop manual remediation validation
Interested in calculating the value of your testing program? Use CyCognito’s Cost Savings Calculator to provide an estimate of cost reduction and efficiency gains. It’s an invaluable resource for security leaders and a fast way to kick start your effort.
Then reach out to learn more about why experts recognize CyCognito as a best-of-breed EASM provider, supporting application security testing and complex organizational structures.