Study Identifies Foundational Controls That Have Biggest Impact on IT Operations, Security, and Audit Performance

By | May 8, 2006

Information Technology Process Institute (ITPI), a not-for-profit membership organization, today released a research report that identifies a set of Foundational Controls that differentiate top performing IT organizations.

Spending on IT compliance and IT control activities has increased significantly as Sarbanes-Oxley and other privacy and industry specific regulations have taken effect. Ongoing spending has resulted in IT executives looking for return on IT compliance and control investments. This study was designed to help IT organizations understand where to focus resources as they implement ITIL(R) best practices and COBIT control activities — in order to achieve return on investment through improved IT performance.

In collaboration with Carnegie Mellon Software Engineering Institute and Florida State University College of Information, the ITPI research suggests that IT controls do improve performance, and that a subset of Foundational Controls have the greatest impact on performance measures.

Key findings indicate that a set of 21 control activities, what the ITPI calls Foundational Controls, have the broadest impact on key performance measures. Top performers had higher performance measures in key operational metrics such as 12 to 37 percent less unplanned work than medium and low performers, 12 to 26 percent higher change success rate, and 2.5 to 5.4 times higher server to system administrator ratio. In addition, the presence of specific Change and Configuration controls, among others, appear to be the primary differentiators of top performing organizations.

“IT is awash with process standards that describe how to set up and manage IT Risk Management and Control functions. But until now, there has been no clear guidance on the relative effectiveness of control implementation efforts,” said Jennifer Bayuk, managing director of IT security at Bear Stearns. “This study is groundbreaking in its cold hard look at why we implement controls, and the fact that the researchers were surprised by some of the results demonstrates the authenticity of its findings.”

The ITPI´s unique approach to studying top performing IT organizations is to pair IT professionals with university-based researchers. In this study, a team of volunteers developed a survey designed to identify the impact of IT control activities on operations, security, and audit performance. They analyzed 63 COBIT controls and 25 key performance measures to identify a correlation between the presence of IT controls and different performance levels. COBIT controls analyzed include access controls, change controls, configuration controls, release controls, service level controls, and resolution controls.

“This work is developing the empirical evidence of the value and relevance of IT control activities the industry desperately needs,” said Andy Moore, of Carnegie Mellon´s Software Engineering Institute. “When IT management can bring quantitative science to bear on important investment decisions, they can reduce risk and also increase operating performance. That helps justify capital for future projects.”

The IT Controls Benchmarking Study, which was funded in part by grants from Tripwire and BMC Software, was completed by volunteers from 98 IT organizations from a broad range of industries primarily in North America, from August 15 to October 30, 2005. Survey responses were analyzed based on their use of controls, and top performers were identified based on how many of the 25 measures they scored in the top 50th percentile of all respondents.

Leave a Reply