The Importance of Security Metrics

From the Editors of ITSC

Security metrics are a hot topic for Chief Information Security Officers, who need to show just how well (or how not so well) they are doing in security. Why would you want to show that your security is not doing well? Here is one example:

A CISO for a Fortune 500 company used the metrics gathered at the firewall to gain additional funding for security. Among other things, he demonstrated that 80,000 attacks had been detected against their firewall, but they did not have adequate staff to respond to these attacks. In other words, there was no one to respond to 80,000 attacks.

Metrics are valuable when they are used to improve the business, whether in responding to attacks, deploying patches, or tracking compliance. This article covers the value that metrics bring to the business, important metric categories that some CISOs are using, and what some visionaries are doing to track performance.

Every company has its own initiatives and unique security concerns. Therefore, it's not surprising that different organizations track different things. What you track may depend on what you want to protect, measure, analyze, or improve. However, don't try to measure everything. Figure out what critical metrics support and provide value to your business.

In a recent security metrics panel at the Executive Women's Forum (EWF), Rhonda MacLean, former CISO at Bank of America and now the founder of Risk Partners, said, "Having metrics that are meaningful and support the business and its customers requires a disciplined process approach. At the heart of the process is the word value. The value of measuring and establishing repeatable and predictable metrics that show progress and trends to the business and its customers is absolutely fundamental to success. Determining how and what will be measured must be driven by first getting the important input of the customer. Would the customer for whom the metric is being developed see it as providing value? If not, should it be measured?"

Tracking metrics that create value to the business

Value needs to be at the core of every metric that is developed. If it does not have value to the business or to your customers, you should take a good look at why you are measuring it in the first place. Some security metrics used in the past may not be valid today, such as the total number of port scans. That might have been important to measure five years ago, but is it today?

Getting people excited about measuring security is not easy, and creating and implementing metrics might even require a shift in the company culture. Some security professionals blanch when they hear phrases like metrics, total quality management (TQM), and key performance indicators (KPIs). Today, however, given the amount of money that corporations are spending on security and regulatory compliance, security teams cannot avoid the fact that security must be tracked, measured, and improved over time. In fact, it should not be any different from any other part of your business. So just what is being tracked? At the EWF conference this year, some leading CISOs and Security VPs (a total of 64 respondents) listed some of the metrics they are currently tracking.

The lessons of total quality management

Building a security metrics and measurement component into your information security program can resemble other efforts that may already be under way within your company -- for example, Six Sigma for quality management. The challenges you face in developing a metrics and measurement program for security may also be similar to those faced in creating a TQM program.

Successful TQM programs show parallels with information security programs in terms of metrics and measurement. The initiators of TQM have always been seen by the practitioners (the people who have to worry about quality management) as an impediment to conducting day-to-day business. In fact, there is a term for that:  the cost of quality. Similarly, there will be a cost of security to consider.

It is important, then, to identify what lessons were learned when carrying out other initiatives in your organization. How were they developed? How were they managed? How did they transform the company's goals to align with these quality goals? How did people engage with the organization to assist in developing best practices? How do you now apply those lessons to security? By examining what has been done in the past, you will be better able to determine the best security approach for your organization.

Tracking performance

Establishing benchmarks and KPIs for different functional areas in security is vital. For each of these areas, you will need to develop both KPIs and corresponding key performance measurements (KPMs) in order to develop benchmarks, set goals, and track progress.

For some areas, such as vulnerability testing and user awareness training, metrics might be more straightforward to establish, but you will need to examine how to manage information security in all the functional areas within your organization and determine how to develop metrics for each of them.

Here are some specific examples:

• Risk assessment  Tracking the percentage of systems that have had a formal risk assessment performed, documented, and reviewed by management.
• Vulnerability testing  The percentage of systems whose security controls were tested and evaluated in the past year.
• Incident response  The average time elapsed between when a vulnerability or weakness is found and when corrective action is taken. This shows your response team's ability to deal with vulnerabilities as they're discovered.
• Infrastructure protection  The percentage of systems with the latest patches or other protections installed. For some organizations, it is the percentage of systems moved to a new network infrastructure.
• Access control  The percentage of systems that have access-control policies defined and implemented.
• IT security training  The percentage of employees with major security responsibilities (such as privileged access to systems or information) who have undergone effective background checks and have received specialized training.
• Regulatory compliance  The number of incidents reported to both the SEC and the Federal Computer Incident Response Center. For each area you need to determine exactly what to monitor, how to monitor it, and what reporting frequency works for your organization.

Developing a security scorecard

Tracking and reporting security performance is critical. You can do this by creating a scorecard. To create your own scorecard, begin by identifying the functional components that you want to measure. For example, user awareness training, policy, audit management, incident response, remote access, business continuity and disaster recovery, PKI infrastructure management, risk assessment, and so on.

You will also need high-quality reporting mechanisms to make sure each department is held responsible for its performance. It also helps to report performance to each business unit. For example, say your research and development did not deploy three critical security patches on mission-critical servers. That means a high-risk business unit is not in compliance with your security policy. Susan Gueli, Vice President for Information Risk Management for Nationwide, said, "What we've seen in our experience is that the more we can personalize our metrics for each executive, the more invested and engaged they become in driving the necessary action." 

Performance dimensions

The Sand Hill Group, a research company, developed a scorecard that shows measurements in five different categories. The three internal categories are people, process, and technology; the external categories are customers and cost of security. The key performance indicators tracked reflect performance in one of three areas:

• Performance  How well are we doing?
• Value  What is the value to the relevant stakeholder?
• Relative Performance  How am I doing compared to my peers?

To review:

1. Determine specifically what is going to be measured.
2. Determine resources within your organization that can provide input.
   Set up a plan defining the what, the how, the baseline, the goals, and the progress.
3. Include a reporting component that makes the results personal.

Then, as in any program, empower your employees to help define and set what needs to be measured and how. This is very important. Getting their involvement does not mean you will use only internal resources, however. You can also use external resources, such as those from the National Institute of Standards and Technology (NIST). They provide examples for devising metrics. Or, you may already belong to business organizations that provide this type of information. Further, consider reaching out to your peers. Find out what they have tried, and what they consider best practices.

Conclusion

So why this need for continuous excellence? Because it is critical that an organization knows how well an organization is doing (or not doing), and metrics provide concrete information that can help improve performance and gain funding. Make sure to track the things that are important to the business or to customers. Both for building customer loyalty and for building your brand, make information security work to your best advantage.