Security leaders often struggle to communicate how well they are performing. Similarly, to justify new investments, budget increases, or ongoing costs, security teams find it difficult to articulate the value of security investments. It is not enough to claim we prevented something bad from happening.
Traditional security management and reporting methods often fall short. Heat maps with high/medium/low designations do not have the necessary rigor. Multipage security reports may have the appearance of rigor, but are lacking actual business context. The metrics themselves are often ineffective:
- Activity-based metrics may be easy to collect, but provide little substance (e.g., number of spam messages blocked)
- Assurance-based metrics alone are too simplistic, ignoring uncertainty inherent to security risks (e.g., these controls are in place; therefore, critical vulnerabilities are mitigated)
Properly designed security metrics, a core element of the security operating model, help organizations address questions business leaders care about. They want to know if security investments and efforts are achieving their intended results. Metrics must enable the security organization to learn and improve by evaluating security efforts against expectations and adjusting them appropriately.
No two security organizations are the same, and each faces its own set of security issues and challenges. An organization’s metrics should be relatively unique to its situation. Leaders should start with the end in mind, understand what outcomes are necessary (often in the form of risks and risk mitigation), and work backward to develop a metrics package that informs their progress toward these outcomes. An outcomebased approach, similar to the goal/question/metric method, can identify the most critical measures to evaluate the performance of the organization’s security operating model.
This security outcome approach includes the following steps:
- Establish a set of outcomes focused on the organization’s risk priorities and security operating model core functions
- Develop questions that define those goals as completely as possible in a quantifiable way
- Define the metrics required to answer those questions
- Determine the sources of information that best source these metrics
- Collect, validate, and analyze the data in real time to monitor security performance and identify gaps requiring corrective action
- Analyze metric trends to gauge security performance toward achievement of functional outcomes and generate recommendations for future improvements to the security operating model
The approach is iterative. It must account for both new questions and new outcomes as they arise. And it allows you to truly explore the outcomes and reevaluate outcome confidence as learnings reveal themselves.
The approach is a powerful way to instrument your security performance—measuring outcomes, not activities. These metrics provide greater visibility and transparency into goal attainment rather than “stick counts.” A secondary benefit is this approach improves goal clarity by forcing outcome definitions to be quantifiable.
For example, a company implements a set of security controls aligned with the NIST cybersecurity framework. In addition to measuring assurance with these controls, each function identifies its top three cybersecurity/physical security risks, pursues risk mitigation, and reports progress. A cybersecurity function identified phishing attacks as a critical risk in need of targeted mitigation. They used the outcomebased approach to develop metrics to monitor their performance:
- Outcome: Reduce the impact of phishing attacks
- Questions:
- How many phishing attacks are successful?
- Are remediation efforts having an impact?
- Are parts of the organization particularly susceptible?
- What is the return on investment for phishing attack remediations?
- Metrics:
- Click rate for phishing simulation emails – simulation test results provide an indicator of phishing success rates
- Click rate of phishing simulation emails trend – this can be synced with phishing remediation activities (e.g., awareness training) to understand remediation effectiveness and further be refined with A/B testing with targeted employee groups
- Click rate by department or other employee segments
- Phishing remediation cost avoidance/remediation cost
- Estimate the cost of a phishing attack by analyzing industry costs of successful attacks, adjusted for your company
- Estimate baseline phishing attack probability
- Estimate improvement to attack probability based on improvements measured in (a) and (b) above
If developing a security metrics program is a priority for you, connect with a ScottMadden security expert to discuss what you can do to achieve your goals. We would be happy to share ideas as well as additional resources.
