IT Security Benchmarking – Compare yes, but insist on hard data too.   
By Philip M. Cronin, CISSP, and Bruce Eissner

Benchmarking techniques can provide a meaningful evaluation of a company’s IT security. While compliance will tell a company what it must do, benchmarking can indicate what a company ought to do. Selecting the right mix of objective measurements, comparative targets and some hard data will provide unique measures of IT security. This paper discusses how management can design a powerful approach and apply the results.

The big question: If there’s one question that management always has in mind, it is, “Am I secure enough and how can I prove it?” Getting the answer with specific regard to IT security does not lie in
just the size of the IT budget or in meeting regulations. Big budgets and full compliance do not necessarily mean that a company has made all of the right security decisions for its business, or that its security posture is optimized. Nor do they ensure that, if something
untoward happened, management’s actions would be seen as appropriate and effective.

IT security benchmarking supplies both a qualitative and quantitative set of measurements for management. By selecting carefully the benchmarking measurements, management can get honest answers to questions such as:

• Does this company have the right security posture for its industry and its size?
• How does this company’s security posture compare with its peers and its competitors?
• Can management demonstrate why they have made certain choices and that they have shown due diligence?
• Is the company more secure now than it was (a year ago)?
• Is the company’s management spending the right amount of money?
• Is the company’s management spending that money wisely?
• Is the company properly organized to meet its IT security needs?

Comparisons matter: In one respect, these questions do not yield to hard answers. They are comparative. For example, if anagement knows for certain that its organization compared well to the best managed companies, it could assertively answer the questions and have the data to back it up. Standards, best practices, and regulations may be too narrow or inadequate to provide real guidance for management; the best answer is often a well-drawn comparison. But the comparison is useful only if the comparables – whether objective or subjective – are themselves properly selected.

But details supply the evidence: Detailed technical measurement and analysis of essential security factors and evaluation against specific industry standards of best practices provide the irrefutable facts necessary to make enterprise-impacting decisions. This helps to ameliorate the frustrating – and legitimate -- board room reactions of, “Yes, but,” or “That can’t be us,” or “Show me.” In order to fully measure a company’s environment, hard, detailed facts are required in addition to comparisons. A benchmarking sampling of the critical security factors in the environment is needed. Hard detailed facts in technically critical areas such as server hardening metrics and network penetrability are necessary
(discussed later.)

In each of these areas, benchmarking will supply executives with health checks by comparing key security performance metrics to peer organizations, best practice firms, and established standards. The results can:

Look both within and outside of your vertical: The choice of which company, sector, or vertical to reference is important. The authors’ company participates in an alliance of security firms with accesses
to a database of over 1,000 companies from across a broad spectrum of industries. The database provides records of how well each of those companies fared against the ISO 17799 Standard (further discussed below). In addition, the database allows for specific comparisons. Combined with experience and judgment, this database allows for substantial insight. But selecting the comparative enterprise or enterprise class is even more important.

Considerations for selecting the best-in-class comparisons may include: the size of the organizations, their budgets and their resources, the maturity of the organization and of its industry, asset values, levels of tolerable risk, similarity of technologies, etc. Often, the best in class is not in the same class.
For example, the authors have found that benchmarking certain measures against the financial segment (banking, insurance, etc) is often a better comparison for best-in-class than a non-financial industry might find within its own vertical. Many verticals have had a mixed history of commitment to IT security. Benchmarking against a weak class will only ingrain weakness. Selecting a forward-looking
segment will lead to better comparisons, reduce risk, ensure forward thinking, and can often lead to solutions that are efficient and effective.

Start with a goal: A successful benchmarking project must start with its goal in mind. Management needs to articulate a clear understanding of how the results will be utilized within the organization. From the beginning of the study, the team needs to be prepared to help managers uncover, interpret, display,
and communicate effectively both the findings and the next steps. The objective of the study is to support executive leadership in making decisions and setting priorities. With such an objective in mind, it is also essential to plan effective vehicles for interpreting,
displaying, and communicating the findings of a study. Some examples follow:

Overall results: The sample diagram to the right represents the results of the snapshot overview and demonstrates areas in which
XYZ (‘XYZ’ implies a sample client) surpasses the performance of the other respondents in the database, as well as those areas in which XYZ requires remediation. Note that there are three sets of points on the graph. The dotted greenline represents an arbitrary ‘passing ISO grade’ as determined by a consortium of security experts from
government, private, and non-profit sectors that developed this concept. The yellow band represents the average scores of all other respondents (for example ‘peers’ alternatively, best-in-class) in the database, and the red band represents XYZ ’s score. In this example, it is significant to note that XYZ scored quite well in this survey as compared with the average respondent. XYZ beat the averages in seven of the ten vectors. Clearly, thorough, detailed description of each of these security vector results needs to accompany the findings.

Results compared to industry: The sample chart to the right demonstrates XYZ’s results relative to the averages of
the other respondents by industry(for example, peers). It
is also instructive to note XYZ’s placement in the overall
distribution spectrum. The chart demonstrates a distribution cone of scores in the overall database. The chart indicates what percentage of the respondents scored below XYZ and how many scored better. In this example, the chart demonstrates, the preponderance of respondents scored below XYZ.

Areas for special attention: The 3rd chart shows information that requires special attention. In the example, the special area is access control and password use. This example shows XYZ personnel do not have an appreciation for the need for
complexity in passwords and do not understand the issues around password selection. The kinds of passwords found in XYZ’s systems are characteristic of the passwords identified by the SANS/FBI as among the ten most serious vulnerabilities found in all organizations today. The sample chart shows some results from the awareness survey that demonstrate that nearly 30% of XYZ users believe ‘banana’ is an acceptable and safe network password, and nearly half believe the same of ‘fido23’.

Whether measured against peers within a vertical or against the best-in-class organization, executive management needs such clear pictures of strengths and weaknesses. Based on the findings, a
strategy for maintaining information security strength and a mandate for action to mitigate weakness can be developed and tested periodically against meaningful standards.

No longer ‘soft’: Benchmarking has evolved to become a more quantitative discipline. Expert benchmarking practitioners now recognize that, while comparative benchmarking continues to be useful to management for setting overall direction, today’s industry leaders need and require quantitative benchmarking. Detailed technical and quantitative analysis of key security factors, combined with an evaluation against equally detailed peer industry --or more rigorous vertical --standards of best practices provide the irrefutable facts that are necessary for making enterprise-impacting decisions. Quantitativeindustry-accepted standards must be recognized and accepted in addition to qualitative comparisons to
peer organizations. At the enterprise level, ISO 17799 is the most widely used and accepted IT security management standard. In addition, detailed, quantitative measurements of such critical security factors such as server security metrics (patches, audit policies, unneeded services, access controls, often called ‘hardening’ metrics) and the penetrability of the external network must be assessed as part of a valid benchmarking study. Standards from CIS can provide a reference for server hardening. Utilizing the
financial vertical experience can provide a reliable, valid reference for penetration tests.

ISO 17799: The ISO 17799 standard is an internationally recognized standard that is widely used as a means for evaluating and building sound, comprehensive information security infrastructure. The ISO 17799 standard is recognized and referenced by NIST, FFIEC, SANS and NERC.

CIS Standard: The Center for Internet Security (CIS) is a nonprofit enterprise whose mission is to help organizations reduce the risk of business and e-commerce disruptions that result from inadequate technical security controls. CIS has led in the development of the “Gold Standard Benchmarks”. Those standards reflect consensus of technical specialists from CIS members, including the National Security Agency, Defense Information Systems Agency, General Services Administration, NIST and the SANS Institute. Those are also the federal agencies that recommend the benchmarks as the minimum baseline security configurations for their own agency’s systems.

Getting a meaningful result: There are some basic benchmarking rules. First, the benchmarking goals must be clearly established, and then the comparables should be identified. In addition, as this
paper has shown, detailed technical measurements and analyses of essential security factors utilizing industry standards of best practices for comparison are essential. Moreover, hard data, such as server hardening metrics and network penetrability easurements must be obtained. This approach will provide a meaningful set of benchmarking comparisons along with irrefutable facts that, together, are necessary for management to make enterprise-impacting decisions.

© 2006    Polar Cove