Information Security Metrics

Scenario

Claire is the chief information security officer at a medium-sized state university that for several years has been experiencing budget pressures, in large part due to shrinking state support. At the end of the previous budget cycle in June of last year, the administration told several units—including IT—that starting next year they would be asked to demonstrate the effectiveness of existing programs to maintain funding. Moreover, new budget requests would need to include clear plans to evaluate their effectiveness.

When the announcement was made, Claire had recently started outlining a set of metrics for the university’s security programs, and the news prompted her to move that effort to the top of her list. Over the summer, she and her staff refined the models they created and tested the data that they had been collecting for years. By the time the fall semester began, they were ready to run real data through some of the metrics they had devised, and they also had set up three small-scale programs to test specific security-awareness campaigns. Their efforts helped Claire and her staff identify some security initiatives that appeared to be relatively ineffectual, given the resources to conduct them.

Now that the spring semester is under way, the security metrics have enough data to start drawing conclusions about the pilot efforts, as well as for several of the longer-term security initiatives. The work that Claire and her team have done has produced several new ideas for ways to improve IT security, and they have also looked to similar institutions for effective security controls. Evaluating not only a security program but also how one would measure its impact has given Claire a broader perspective on her work and how well it aligns with the institution’s overall strategic goals. She is confident that she can make a strong, focused case to the administration for the programs that work well and should be preserved. She also understands that as the metrics program grows over time and includes data from more semesters, it will become an even better tool to assess existing initiatives and to plan for future ones.

1. What is it?

An information security metric is an ongoing collection of measurements to assess security performance. Some of the data in security metrics come from an IT organization’s operational activities, but data can come from other sources as well. At its core, an information security metric helps an institution determine whether its security program improves security, shedding light on many facets of the program and how it supports the institution’s mission. Metrics can provide insights about program effectiveness, regulatory compliance, and the ability to address security issues. Metrics can also help identify the risks of not taking certain actions and, in that way, provide guidance for future investments. Because metrics provide concrete facts and a common vocabulary for communicating risks, they might also raise the level of security awareness in an organization. Finally, with knowledge gained through metrics, those responsible for information security programs can more credibly answer hard questions such as: How do information security investments further institutional mission and goals? Are we more secure today than before? How do we compare to others?

2. How does it work?

Information security metrics measure a security program’s implementation, effectiveness, and impact. Metrics must be meaningful, tailored to measure the actions or specifics of a security program. Although metrics can be categorized in multiple ways, the National Institute for Standards and Technology divides security metrics into three categories—implementation, effectiveness, and impact—corresponding to levels of security program maturity. Implementation metrics illustrate an institution’s progress in putting security controls or policy into place. An implementation metric might measure the percentage of critical information assets for which a risk assessment has been performed, as required by a regulatory agency. Over time, an institution might reach 100% for implementation metrics. Effectiveness metrics show whether security controls are implemented correctly and meet a desired outcome, such as reducing vulnerabilities. An effectiveness metric might reflect the change in the number of critical vulnerabilities detected on servers since the last reporting period, showing that security controls are operating as intended. Impact metrics assess the overall impact of an information security program and are usually specific to an institution and its goals. For example, an impact metric might measure the change in number of system servers containing Social Security numbers, quantifying the reduction of institutional risk if a goal is to decrease the storage of such information.

3. Who’s doing it?

Like many performance-based measurements, information security metrics began in the business world, often where process improvement frameworks such as Six Sigma Breakthrough Strategy had already taken hold. Businesses use metrics to justify their security-related expenditures, typically based on return on investment. Some companies have tied security metrics to corporate security standards compliance; others espouse an audit-based approach that verifies compliance with industry-standard control objectives and locally defined standards. Because IT infrastructure, services, and security oversight tend to be centralized in a corporate environment, quantifying ROI for a given security expenditure is often a simpler task for business than higher education.

4. Why is it significant?

Metrics identify levels of risk in information security strategies. A robust metrics program improves the effectiveness of selected strategies, increasing accountability for the controls implemented. Metrics programs can also demonstrate compliance with a growing range of regulatory obligations. Such metrics highlight the important role information security plays in supporting institutional goals. Because higher education institutions face increasing budgetary, regulatory, and accountability pressures, these metrics provide needed assessment of information security programs and compliance and the ability to address security problems. An information security metrics program offers a clear understanding of vulnerabilities and how well they are being addressed.

5. What are the downsides?

Producing reliable and effective information security metrics is not easy. Information security departments can measure a large number of variables, but the data collected must yield an unambiguous metric that facilitates decision making. Also, measurement data must be collected in a reliable manner to ensure the metric is trustworthy. Because metrics are ongoing, periodic changes in systems and processes must not adversely affect the measurement data, and institutions must maintain their commitment to the program with resource support. If an institution does not have the resources necessary to analyze the collected measurements, its metrics program will not be successful. Finally, an institution should frequently review its selected metrics to ensure they remain meaningful and relevant as its information security program evolves.

6. Where is it going?

Mature security metrics programs collect effectiveness and impact measures, allowing targeted security controls to be identified and implemented. While IT trends offer expanding opportunities for knowledge discovery, cost savings, improved user experience, and so on, with these trends come new risks for security/privacy gaps and subversive activity. As institutions implement metrics, those responsible for information security will be better prepared to respond effectively to difficult questions about these trends, with knowledge and data gained through their metrics. Moreover, security metrics will become increasingly sophisticated in the types of data collected and the means of data analysis. For example, security vendors already leverage big-data techniques to develop new products for gaining deeper insight into security risks, resulting in more-accurate and more-nuanced metrics.

7. What are the implications for teaching and learning?

Higher education increasingly operates in a regulated and results-driven environment, with limited budgets. In addition, various requirements compel institutions to assess the efficacy of their information security programs. Properly collected and analyzed metrics can help organizations support a current information security program and justify improvements to it. Additionally, effective metrics can bring visibility and awareness to the underlying issue of information security and highlight areas that have been heavily resourced or don’t produce results, allowing for reallocation to more effective efforts through benchmarking, evaluation, and assessment of quantified data. This can put institutions in a proactive stance regarding information security and demonstrate support for leadership’s priorities.

JANUARY 2014

Members of the Governance, Risk, and Compliance Working Group of the Higher Education Information Security Council (HEISC) contributed to this publication.

© 2014 EDUCAUSE. This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 4.0 License. EDUCAUSE is a nonprofit membership association created to support those who lead, manage, and use information technology to benefit higher education. For more information, please contact us at info@educause.edu.