Patch Management: Q&A with Thom Bailey

By | June 21, 2005

Patches are small, sometimes temporary “quick fixes” that address often critical software flaws. Patches are usually made available by software vendors in between service packs or version upgrades upon confirmation that a software flaw affects a large number of users with usually serious consequences.

More often than not, patches address vulnerabilities impacting security. These patches are therefore referred to as security patches. Typically, patches are made available for download from the software vendor’s support Web site.

What exactly defines patch management?

Patch management deals with all aspects of patching—from scanning computer systems for existing or missing patches, identifying the availability of new patches, downloading patches, deploying and installing new patches, to validating patches for confirmation that the patched systems are working properly and that the patch is successfully installed across the entire domain. In a well-managed computing environment, the administrator understands the current patch status of all systems he controls, has up-to-date information on the latest threats, vulnerabilities and available patches, and has the ability to act quickly to test new patches, to determine which patches are appropriate for which systems, and to ensure that new patches are deployed and installed quickly and securely.

What makes patch management such a hot topic today?

While the need to manage software has been a long-standing requirement, organizations are now more than ever painfully aware of the risk involved in not applying patches, standardizing system configurations, and anticipating the critical need for rapid recovery from vulnerability exploits or faulty patches. Companies are vulnerable to security breaches, mass outages, loss of productivity and even the loss of customer confidence. With the number of vulnerabilities on the increase and exploits spreading faster than ever, companies have a shrinking window of opportunity to remediate vulnerabilities—which in turn makes patch management more critical today than ever before.

What are some of the statistics showing this increase in vulnerabilities and the decrease in the window of opportunity to remediate vulnerabilities?

Symantec documented 1,403 new vulnerabilities between July 1 and December 31, 2004, which translates into an average of 58 new vulnerabilities per week. 70% of these vulnerabilities were considered easy to exploit and 97% were considered moderately or highly severe. This means that not only do organizations have to contend with almost ten new vulnerabilities per day. but also almost all of these vulnerabilities could result in a partial or complete compromise of the targeted system. Symantec also documented more than 7,360 new Win32 virus and worm variants during this period, which represents an increase of 64% over the previous six-month period. The MyDoom worm infected email systems across the world—at its peak, one out of every 12 emails on the Internet carried MyDoom. In addition, Symantec data indicates that the average time between the announcement of a new vulnerability and the appearance of code designed to exploit that vulnerability is 6.4 days.

In your opinion, what is one of the main reasons for the increase in the rate at which vulnerability exploits are spreading? One factor that has significantly increased the rate at which worms and viruses are spreading from machine to machine is today’s “always on” connectivity that has dramatically changed both our consumer lifestyles and the way we conduct business globally.

Given the shrinking window of opportunity to remediate vulnerabilities after they are discovered, how do organizations ensure they are protected while the IT team is testing new patches before they can be deployed? Once notified of a new vulnerability, several steps need to be taken inside an organization before the vendor provides a patch that is then being tested by the IT team. For example: A security analyst receives information of an exploit of a newly identified vulnerability. He issues a trouble ticket to alert the appropriate IT administrator.

As a safeguard, the IT administrator increases the frequency of backups to minimize data loss. He also locks down existing backups. Symantec’s Managed Security Services team receives the alert and learns that the vendor has not yet issued a patch. As a proactive measure, the Managed Security Services team updates signatures and protection configurations to the network security appliance, blocking the exploit and buying time until a patch becomes available. The onsite security team runs Symantec Enterprise Security Manager to identify which systems are potentially at risk—based on available asset information. They determine that the online order entry system could be a potential target.

The Managed Security Services team learns that a patch is now available and notifies the onsite IT team. The onsite IT operations team runs a patch management solution, such as Symantec LiveState Patch Manager, downloads, tests, deploys and validates the patch. Once the patch is deployed, the security team runs Symantec Enterprise Security Manager and discovers that a remote system has been infected and files have been deleted. Using recovery tools, the team restores the files and patches the system. All systems are checked again, and it is determined that all systems are compliant. The trouble ticket is closed. To prevent future attacks, the team launches a program on security awareness.

There are different industry approaches to patch management. Some vendors see patch management as an issue that needs to be addressed by itself, while others favor a more comprehensive approach. Which side are you on?

Because of the many dependencies, Symantec believes that patch management is only a small, yet critical aspect of an organization’s overall systems and configuration management plan and needs to be addressed as such. Most industry experts agree. A September 29, 2003 META Practice report by Dan Vogel (Process Use Scenario: Patch Management—Operations Excellence Infusion, Operations Strategies, Service Management Strategies) suggests that patch management touches a variety of different areas within an IT operations process catalog, such as application optimization, business continuity, change management, configuration management, problem management, security management, service-request management, software distribution, software management, and test lab management.

Addressing patch management as part of a bigger systems and infrastructure implementation requires the involvement of various teams within the IT organization. How do you ensure a combined approach is successful?

Symantec has the advantage of seeing the problem from a security perspective. Typically, in medium-sized and large enterprises, IT organizations have a security team and an operations team. Each team within the IT organization has a different focus while trying to maintain the integrity of the organization’s infrastructure. The security team is responsible for detecting the vulnerabilities. The operations team, once alerted, remediates the problem. Just-in-time communication, process and workflow are critical to achieve resolution.

All too often, the two teams within the IT organization—because of different challenges, goals and priorities—do not work in concert or at the same speed. When choosing solutions, there are clear advantages to selecting a vendor who understands the needs of both teams within the IT organization. Symantec offers solutions that bridge existing gaps by helping an organization´s security and IT operations teams leverage each other´s expertise to solve today´s recurring patch management challenges.

How do you ensure the successful integration of a patch management process into existing IT processes?

The existence of a patch management process is a key requirement. In fact, META Group concludes in a November 19, 2003 META Practice report by Mark Vanston (Patch Management: Building the Process—Operations Strategies, Enterprise Data Center Strategies, Service Management Strategies) that “successful patch management requires a robust process that uses core competencies from other processes, such as change and configuration management.”

The scope of integrating such a patch management process into existing IT processes depends primarily on the size of the organization and the complexity of its infrastructure and its existing IT processes. Small businesses and smaller organizations will in very few cases have complex IT processes in place that would make the integration itself complex.

Larger organizations with a complex infrastructure and complex existing IT processes may want to take advantage of professional services offerings to facilitate the integration. Symantec provides extensive consulting services to deliver consultation based on proven methodologies and best practices. Symantec´s consulting services assess, design, plan and implement effective security, systems and storage management solutions and processes. Recommendations for a patch management process would include an executive summary that illustrates how a proactive vulnerability management and patch management process is a continuous process, an illustration/chart that shows the individual elements of the process in a linear fashion, and detailed discussions of each element of the process. Combined, Symantec´s solutions and services enable organizations to establish and maintain a more secure and available business environment.

Whether to choose an agentless or an agent-based patch management solution is an ongoing debate. What is your opinion?

The debate is an interesting one. When deciding on the best patch management implementation, organizations need to take a look at which model best fits into their particular environment. Small businesses, small, medium-sized and large organizations each have different approaches to systems and infrastructure management. Enterprise organizations with highly-distributed heterogeneous environments will likely prefer agent-based implementations that can be incorporated into a larger systems or infrastructure management solution and that provide benefits such as reduced network traffic and the ability to be configured to comply with security policies that require encrypted data. Smaller organizations on the other hand may prefer an agentless implementation because it is easier to install and configure. In short, there is no such thing as a “one size fits all” approach to patch management. The only right approach is to decide between an agent-based and an agentless implementation depending on the individual organization’s environment.

The industry’s main focus seems to be on Microsoft vulnerabilities and patches. What is the reason for that, and is it really justified?

Without doubt, Microsoft has suffered the brunt of the industry’s criticism on vulnerabilities. The fact that Microsoft operating systems and applications are so prominent in most organizations, especially on the client side and in particular in smaller and medium-sized organizations and small businesses, certainly represents one of the main reasons why Microsoft vulnerabilities are such a hot topic and why many IT departments are primarily concerned with managing Microsoft vulnerabilities. It is also clearly the reason why most patch management solutions available today concentrate on the Microsoft environment.

However, we believe that this Microsoft-centric approach is about to change. Information available in forums such as SecurityFocus ( clearly demonstrates that a large number of vulnerabilities actually occur on other platforms, most notably in open source environments. As the role of Linux in the corporate server environment is significantly growing and as IT departments are becoming increasingly aware of the need for patch management on the server side, we will start seeing a need for patch management implementations that support more than just Microsoft environments.

How do you respond to a customer who sees the benefits of implementing a patch management solution but considers patch management not critical enough to warrant the expense?

The implementation of a patch management solution is absolutely critical in today’s environment. Compared to the potential costs and revenue losses associated with security breaches, mass outages, loss of productivity or the loss of customer confidence, the expense of implementing a patch management solution is minute. Given the threat landscape today, companies cannot afford to ignore the need for patch management. Also, the sheer number of vulnerabilities and associated patches make it impossible today to successfully manage the patching process manually—aside from the fact that the manual process is ultimately more expensive. In the META Practice report “Patch Management: Building the Process” mentioned earlier, Mark Vanston summarizes the business impact by saying that “inefficient patch management can have serious impacts on an organization’s ability to provide service to its customer base.” And a May 30, 2003 Gartner Market Analysis “Patch Management Is a Fast Growing Market” estimates that it costs $300,000 a year to manually deploy patches to 1,000 servers, whereas a patch management solution may cost no more than $50,000. According to Gartner, patch management is better at what it does and does it much cheaper.

Leave a Reply