MOC 3386 Implementing System Center Operations Manager 2007 at Microsoft

INTRODUCTION
The Microsoft server environment consists of roughly 7,950 production servers, 2,700 pre-production servers, and another 50 development, staging, and test servers. Microsoft IT is responsible for monitoring all of them and the more than 2,000 line-of-business applications that they host, many of which are critical to the operations of Microsoft. The previous solution, MOM 2005, was a compilation of many autonomous management groups that were managed by several different internal support groups because of its lack of granular rights delegation. In addition, many application support teams and lab managers deployed their own MOM 2005 infrastructures to monitor their applications and lab server hardware. At one point, more than 160 active MOM 2005 management groups existed in the Microsoft IT environment. To streamline operations and the infrastructure, Microsoft IT needed a centralized monitoring solution. With the deployment of OpsMgr 2007, Microsoft IT made a fundamental shift toward centralizing the way it provided monitoring services to organizational groups.
Challenges of Using Microsoft Operations Manager 2005 for Monitoring
Although the MOM 2005 implementation gave the individual teams substantial control and agility over their monitoring, it also presented several problems that Microsoft IT needed to consider during the design of the OpsMgr 2007 environment:
 There was a significant duplication of effort to provide fault monitoring. Each management group required both hardware and people to maintain the infrastructure. Multiple management groups often monitored the same asset. This meant that one failure could engage multiple teams.
 Even with all the autonomous management groups, gaps existed in the service offering. Most lab environments lacked hardware monitoring, and several applications were not being monitored.
 There was no comprehensive, holistic view of the health of the Microsoft IT network environment. Network, hardware, Microsoft SQL Server™, and application monitoring existed in separate systems.
 Many smaller groups depended on a single expert to manage their MOM 2005 implementation. If that person left, the team often lost its monitoring expertise.
Benefits of Using System Center Operations Manager 2007 for Monitoring
The OpsMgr 2007 deployment provided several benefits for Microsoft IT:
 The introduction of role-based security and improvements to both scalability and Run As accounts removed the roadblocks that Microsoft IT previously had in using MOM 2005 to deploy a centralized monitoring platform to be used by many teams within Microsoft. The centralized design enabled Microsoft IT to eliminate monitoring fragmentation and to better utilize resources.
 By using the Distributed Application Management Wizard, Microsoft IT has been able to rapidly develop management packs for custom applications.
 Microsoft IT invested a substantial amount of time and energy into creating and tuning custom management packs for MOM 2005. Use of the management pack conversion

tools in OpsMgr 2007 enabled Microsoft IT to retain and carry forward the work that the team had previously invested in its management packs.
 OpsMgr 2007 management packs are based on Service Modeling Language (SML), which is a part of the Dynamic Systems Initiative (DSI). As a result, Microsoft IT has been able to use existing management packs and develop new ones based on health models. Health models in turn use classes to provide a more dynamic, relational representation of the state of a system, service, or application.
 OpsMgr 2007 provides a more robust monitoring service offering due to improvements in product scale and support of clustering key infrastructure roles, as well as log shipping for disaster recovery of the Operational database.
 OpsMgr 2007 significantly reduced agent deployment efforts by using Microsoft Systems Management Server (SMS) packages. Agents can discover their configuration automatically from the Active Directory® directory service.
 The introduction of OpsMgr 2007 Audit Collection provided an efficient means of collecting and storing security event logs.
 OpsMgr 2007 provides a full software development kit (SDK) that provides Microsoft IT a complete interface to develop automation of administrative tasks.
Note: For more information about server roles in OpsMgr 2007,
MICROSOFT IT MONITORING SOLUTION
OpsMgr 2007 gave Microsoft IT an opportunity to reassess its service offering and to completely rebuild its monitoring infrastructure. Based on the needs of the organization and the existing customers of the monitoring solution, Microsoft IT developed a new offering, running on a new infrastructure, that provides parity for the earlier service that was based on MOM 2005. The application has undergone significant improvements to many of the features introduced in previous versions of Operations Manager, and it offers a number of new features that improve its capabilities in areas such as security auditing, event log collection, and desktop monitoring. To build and deploy the new monitoring solution running on OpsMgr 2007, Microsoft IT used the following phases:
1. Planning the OpsMgr 2007 environment
2. Designing the OpsMgr 2007 infrastructure
3. Deploying OpsMgr 2007
Planning the System Center Operations Manager 2007 Environment
Prior to any planning work, the engineers on the monitoring team within Microsoft IT held a series of meetings with representatives from their existing customer base as well as a set of prospective monitoring customers to discuss their requirements. The purpose of these meetings was to understand what each of these groups needed from the monitoring solution. The result of these meetings was a detailed list of requirements ranging from the functionality that absolutely must exist to those that were optional. From this list, Microsoft IT was able to begin the process of planning the deployment, as well as the aspects of the service offering, targeted toward meeting the requirements. Informed by the requirements gathered, Microsoft IT approached the design of the OpsMgr 2007 environment with three main objectives:
 Provide a complete monitoring service offering.
 Provide a centralized and holistic monitoring platform.
 Reduce resources devoted to monitoring.
Provide a Complete Monitoring Service Offering
With the MOM 2005 deployment, Microsoft IT had deployed monitoring agents only to production servers. Although this sufficed for the purposes of just monitoring the production aspects of the data centers, Microsoft IT's customers had become increasingly interested in expanding their monitoring to non-production systems. Because of that interest, Microsoft IT opted to expand the agent base with OpsMgr 2007 to all servers in Microsoft data centers. This expansion fulfilled customer requirements and served as the foundation for providing a complete monitoring service for the future.
Provide a Centralized and Holistic Monitoring Platform
With MOM 2005, a number of different groups within Microsoft opted to deploy their own management groups to monitor their specific servers or applications. Although this gave

individual support groups a substantial amount of control and agility over their monitoring needs, it required more resources than were necessary for monitoring. More importantly, it prevented Microsoft IT from viewing the health of the entire environment in a comprehensive manner. With OpsMgr 2007, Microsoft IT planned, from the beginning, to design an infrastructure that would be capable of the necessary scale while at the same time providing flexibility and control to each support group. An additional benefit to this design is that all management packs would be functioning together in the same management group, giving all users a holistic view of devices, applications, or services from hardware to software.
Reduce Resources Devoted to Monitoring
With the expansion of the scale and the comprehensiveness of the monitoring deployment, the Microsoft IT goal was to reduce the resources required to provide monitoring overall. The resource savings were made possible through hardware consolidation and centralized monitoring, which improved the productivity of the monitoring team. The key focus of reducing hardware costs for Microsoft IT came in the form of consolidation. Early in the planning phases, through conversations with other teams within Microsoft, Microsoft IT determined that if it could provide an infrastructure capable of scaling appropriately and a service offering that was complete and flexible enough, the other teams running their own infrastructures would abandon their systems in favor of running their monitoring on a centrally managed platform. In the area of reducing people resources, the focus was twofold:
 Freeing people from maintaining their own monitoring solution
 Using new OpsMgr 2007 features to reduce existing support costs
The first half of the objective is really a byproduct of the consolidation. As support teams merge their monitoring into Microsoft IT's centralized monitoring platform, the people on those teams that have traditionally focused on monitoring will be able to spend more of their time focusing on their team's normal workload. Some examples of where the second objective has been possible are:
 Coupled with using SMS for deploying agents, Microsoft IT uses Active Directory integration so that agents can automatically discover their monitoring configuration when the agent health service starts.
 Microsoft IT is using an agent health remediation management pack to automate repairs on unhealthy agents.
 Microsoft IT is using the OpsMgr 2007 SDK as an interface for other forms of custom-built automation.
Designing the System Center Operations Manager 2007 Infrastructure
As with previous Operations Manager deployments, Microsoft IT built two distinct infrastructures, one for the pre-production environment and another for the production environment. The following sections describe the server roles, the details about each environment, the hardware specification that Microsoft IT developed for each server role, and the measures that Microsoft IT has implemented for redundancy and disaster recovery.

Server Roles in System Center Operations Manager 2007
The various OpsMgr 2007 server roles are:
 The Operational database server. The Operational database is a Structured Query Language (SQL) database hosted on a Microsoft SQL Server 2005 Service Pack 1 (SP1) server. This database houses all of the configuration data for a management group and contains all data generated by the systems in the management group. The database must be online at all times for the management group to function.
 The root management server. The root management server is a new server role in the OpsMgr 2007 infrastructure. The root management server is responsible for hosting the SDK and configuration services and is the central coordinator of health monitoring. The SDK service is the single point in a management group that responds to SDK requests. This includes all operations consoles and connectors. The Config service is the source of authority for all configurations in a management group. All OpsMgr 2007 servers and agents ultimately get their configuration from the Config service.
 Management servers. The management server role has not changed significantly from MOM 2005. These servers are responsible for management of agents, which includes sending configuration data to the agents, and receiving operational data from the agents and forwarding that data to the Operational database. In some instances, the management servers are also responsible for writing data to a data warehouse.
 Gateway servers. The gateway server is a new role in OpsMgr 2007 that functions as a proxy between one or more management servers and a set of agents. In the presence of a firewall or the lack of a two-way trust between a set of agents and one or more management servers, a gateway server can be deployed to act as an intermediary.
 Audit collection servers. OpsMgr 2007 introduces the Audit Collection feature set. These features focus specifically on collecting security events from monitored computers and storing that data in a central repository for alerting and forensics purposes. Three distinct roles are related to OpsMgr 2007 Audit Collection: the forwarder, the collector, and the database. The forwarder, a component of the OpsMgr 2007 agent, provides real-time monitoring of the security log on managed computers. The forwarder encrypts and transmits security events to the collector, a component of the OpsMgr 2007 management server. The collector makes security events available to Windows Management Instrumentation (WMI) subscribers in real time and maintains that data in the audit collection database.
 Data warehouse and reporting server. The data warehouse and reporting server hosts a database that the reporting component of OpsMgr 2007 uses.
Pre-production Environment
The purpose of Microsoft IT's pre-production environment is to act as a complete (as possible), yet scaled-down version of the production environment. Microsoft IT uses the pre-production environment primarily for evaluation purposes. Evaluations range from working with the Operations Manager product group for testing new builds or features, to stressing product scale, to staging a proposed change prior to its official release into production. Figure 1 shows the design of a pre-production management group for Microsoft IT.