Active Directory Performance Counter Error

Sorry about the lack of new posts lately!  I was troubleshooting an alert in Operations Manager 2007 from one of the many servers we manage.  Research on the Internet did not reveal any resolution to this specific alert although I did find a couple of postings that were somewhat similar and led me down the correct path to resolution.  Hopefully this post will help others that are troubleshooting this same alert.

Event ID:  1228
Source:  ADAM
Type:  Error
Description:  System Monitor was unable to open Active Directory performance counters.  An attempt to query the following performance counter registry key failed.
Registry key:
SYSTEM\CurrentControlSet\Services\ADAM\Performance\First Counter
Additional Data
Error value:  2 The system cannot find the file specified.

Sure enough, on my problematic server there was not a First Counter key in the registry at the location listed in the alert.  The similar posts that I came across during my research indicated that there might be an .INI file for the performance counters located at C:\WINDOWS\System32.  I had already noticed that the registry location above contained a PerfIniFile string value with a value of adamctrs.ini.  I checked the System32 directory and found no such file.  A search of the C: drive located it at located it at C:\WINDOWS\System32\ADAM.

I opened a command prompt and sourced from the C:\WINDOWS\System32\ADAM directory and ran the following command:

lodctr adamctrs.ini

I then returned to the Registry Editor and, from the menu, selected View > Refresh.  Four new DWORD values, as shown above, had been created – First Counter (the one causing my alert), First Help, Last Counter, and Last Help.  Alert resolved!  Finding the resolution to the problem took longer than the actual correction of the problem.  Again, I hope this post will help someone else resolve this error in a timelier manner with less research involved.

EMP MAPI Logon Failure

Exchange Management Pack synthetic transactions can generate a lot of noise in the Operations Manager console because there are numerous components that can contribute to the problem (i.e, Exchange, Active Directory, IIS, etc.).  Once it has been verified that the problem is an Operations Manager Agent issue rather than an actual user-impacting issue, the alerts are usually disregarded until someone has a chance to dig in and do some troubleshooting.  Today is one of those days for me.

We have been receiving the following alert from a client Exchange server for a number of days and it was initially diagnosed as an agent issue:

MAPI Logon Failure
MAPI logon verification: Problem in accessing Directory
Service caused Logon failure Exchange Server: “SERVERNAME“
MDB:”First Storage Group\Mailbox Store (SERVERNAME)”
Mailbox:”SERVERNAMEMOM” Error: The information store could not
be opened. [MAPI 1.0 - [MAPI_E_LOGON_FAILED
(80040111)]]

I checked the usual settings – mailbox rights, mailbox security, mailbox access account credentials, etc. – and everything was configured properly.  However, one thing caught my eye while checking the properties of the test mailbox user account.  On the Exchange Advanced tab of the Properties dialog box (below), the Hide from Exchange address lists option was selected.

Exchange Advanced tab

This did not seem quite right to me, so I checked test mailbox account properties in a couple of other client domains.  This option was deselected on every account that I checked.  Apparently, at some point, someone had selected this option so the account, which is disabled by default when created by the Exchange Management Pack Configuration Wizard, would not appear in the Outlook Global Address List. I promptly deselected the option and monitored the Operations Manager event log on the server.  Within minutes Event ID 19980 was logged indicating that all attempted MAPI logins of test mailboxes on the server had succeeded.

As is so often the case, an overlooked checkbox or radio button was the root of the problem and an alert that had repeated hundreds of times was finally closed.  During my troubleshooting, I did not come across anything that referenced the Hide from Exchange address lists option, although it surely exists somewhere in cyberspace, so I hope this post is beneficial to others trying to resolve a similar issue.

SharePoint Execute Method Exception (Event ID 6398)

I spent some time today troubleshooting an alert that we received in Operations Manager 2007 from one of the client SharePoint servers that we manage in our Operations Center.  The alert referenced Event ID 6398 which indicates that the execute method of a SharePoint timer job definition had thrown an exception.  I logged onto the client SharePoint server and looked in the Application event log and, sure enough, it was full of the following 6398 event:

Event Type:    Error
Event Source:    Windows SharePoint Services 3
Event Category:    Timer
Event ID:    6398
Date:        11/19/2008
Time:        1:05:53 PM
User:        N/A
Computer:    <ServerName>
Description:
The Execute method of job definition Microsoft.Office.Server.Search.Administration.IndexingScheduleJobDefinition (ID 03d6f1b3-879c-42d9-8921-3f14b9e5678f) threw an exception. More information is included below.

Retrieving the COM class factory for component with CLSID {3D42CCB1-4665-4620-92A3-478F47389230} failed due to the following error: 80070005.

The System event log contained numerous instances of the following DCOM error as well:

Event Type:    Error
Event Source:    DCOM
Event Category:    None
Event ID:    10016
Date:        11/19/2008
Time:        1:06:26 PM
User:        NT AUTHORITY\NETWORK SERVICE
Computer:    ATI-SHAREPOINT
Description:
The application-specific permission settings do not grant Local Activation permission for the COM Server application with CLSID
{3D42CCB1-4665-4620-92A3-478F47389230}
to the user NT AUTHORITY\NETWORK SERVICE SID (S-1-5-20).  This security permission can be modified using the Component Services administrative tool.

This problem can be resolved by accessing the Component Services applet by way of Start > Administrative Tools > Component Services.  In the task pane expand Component Services, expand Computers, and expand My Computer, and click on DCOM Config.

In the details pane on right, locate and right-click on the OSearch application and select Properties from the menu.  On the OSearch Properties dialog box, click on the Security tab.  In the Launch and Activation Permissions section, select the Customize option and then click the Edit button.

OSearch Launch Permissions

On the Launch Permissions dialog box, add the NETWORK SERVICE user and grant it Local Launch and Local Activation permissions as shown above.  Click the OK button twice and then close the Component Services applet.  You should now have a much quieter and happier event log!

ADMP Replication Monitor and Unmanaged DC

Operation Manager 2007’s Active Directory Management Pack (ADMP) is designed for managing all domain controllers within a given site or organization.  However, we recently encountered one of the rare instances when not all DCs are managed – a client was in the process of retiring one of three DCs in their domain and wanted to remove it from our Managed Services offering.

We went through the normal steps for removing a server from Operations Manager 2007 – uninstalled the OM Agent, deleted the server from the OM database, deleted automated reports, etc.  Within 24 hours, however, the Active Directory Replication Monitor began alerting our Operations Center that a domain controller had failed to update its MOMLatencyMonitor object within the specified time period (24 hours by default).  The retired server’s MOMLatencyMonitor object had been deleted from Active Directory during its removal from Managed Services so how was the ADMP still aware of the unmanaged, albeit still running, domain controller?

Windows Server 2003 adds two additional Active Directory application naming contexts, DomainDNSZones and ForestDNSZones, when you Active Directory Integrate a DNS zone on a domain controller.  We use ADSI Edit to modify the objects and attributes of these naming contexts as follows:

1. Click Start > Run, type adsiedit.msc in the textbox, and click the OK button.
2. In the ADSI Edit task pane, right-click ADSI Edit and select Connect to from the menu.
3. In the Connection Point section of the Connection Settings dialog box shown below, select the Select or type a Distinguished Name or Naming Context option, type the following in the textbox, substituting the appropriate domain and extension values, and then click the OK button:

   DC=DomainDNSZones,DC=Domain,DC=Domain_extension

   

   ADSI Edit Connection Settings

4. In the ADSI Edit task pane, locate and click on the CN=MOMLatencyMonitors container as shown below.
   

   ADSI Edit

5. In the details pane to the right, locate and delete the directory for the server that is no longer managed.
6. Repeat steps 3-5 and specify ForestDNSZones in the Connection Point string rather than DomainDNSZones.
7. Close the ADSI Edit window.

Now the ADMP Replication Monitor will only monitor and alert on the remaining managed domain controllers.

ADMP Replication Failure Verification

A common alert generated by the Active Directory Management Pack (ADMP) indicates that there is a problem with Active Directory replication:

The following DCs have not updated their MOMLatencyMonitor objects within the specified time period (8 hours).  This is probably caused by either replication not occurring, or because the ‘AD Replication Monitoring’ script is not running on the DC.

Format: DC, Naming Context, Hours since last update

MySite
PEGASUS, NDNC:DC=DomainDnsZones,DC=myDomain,DC=com, 11

This alert indicates that my domain controller PEGASUS has not replicated the required value to the adminDescription attribute under the MOMLatencyMonitor container in Active Directory for eleven hours or more.  But how do we know if the alert is due to actual AD replication issues or a problematic Operations Manager script?

We can validate the problem by using the repadmin utility, which is included in the Windows Server 2003 Support Tools.  At a command prompt, enter the following command (substitute the appropriate servername):

repadmin.exe /showrepl PEGASUS

The output of the repadmin utility should be similar to the following:

DC=ForestDnsZones,DC=myDomain,DC=com
mySite\Odyssey via RPC
DC object GUID: 45b3241y-z849-48rs-n76u-6×09bv432ih6
Last attempt @ 2008-10-28 08:34:55 was successful.

According to the output above, the last replication attempt was very close to the current time and was successful.  This indicates that the problem is likely with the Operations Manager script.  The dsquery command line utility can be used to troubleshoot further:

dsquery * CN=MomLatencyMonitors,DC=ForestDnsZones, DC=myDomain,DC=com -scope onelevel -attr name AdminDescription

The output of dsquery should be similar to the following:

name                                   admindescription
PEGASUS                           20081028.0200
ODYSSEY                           20081028.1330

Based on the timestamp values in these results, we are able to detemine that this particular alert is actually an Operations Manager script error.

MachineKeys Directory Permissions

Today I was troubleshooting a problematic remotely managed server that was not reporting into our Operations Center.  The OpsMgr Health Service service was running; however, we were not receiving a “heartbeat” from the server.  The Operations Manager event log contained error events that indicated a problem involving the certificate private key used to unencrypt the package received from our Root Management Server (RMS).

When I accessed the System event log on the server, I also discovered that the following event was being logged several times every hour:

Event Type:    Error
Event Source:    Schannel
Event Category:    None
Event ID:    36870
Date:        10/28/2008
Time:        9:36:08 AM
User:        N/A
Computer:    Servername
Description:
A fatal error occurred when attempting to access the SSL client credential private key. The error code returned from the cryptographic module is 0xffffffff.

My research indicated that this event is the result of improper NTFS permissions being set on the MachineKeys directory, which is located under the All Users Profile\Application Data\Microsoft\Crypto\RSA directory.  This directory is utilized by both Certificate services and Internet Explorer.  The default permissions for the MachineKeys directory are as follows:

Administrator (Full Control) This folder only
Everyone (Special) This folder, subfolders, and files
SYSTEM (Full Control) This folder, subfolders, and files

Although resetting the permissions on the MachineKeys directory did not totally resolve the Operations Manager agent heartbeat problem (I also had to uninstall and reinstall the agent), it did eliminate the Schannel events in the System event log.

Addtional information can be found in Microsoft Knowledge Base Article 278381.

Exchange Management Pack MAPI Logon Alert

The Operations Manager management packs for Exchange Server are usually the noisiest of all of the management packs.  As a result, they also require considerable attention from an Operations Manager administrator.  In our Managed Services operation at ClearPointe, which manages many Exchange organizations of various size, the synthetic logons (OWA, OAS, and EAS) and MAPI logon verifications seem to generate the most alert traffic and are forwarded to me for resolution.

There are several MAPI logon verification alerts but the one shown below that includes the error text of [Collaboration Data Objects – [MAPI_E_NOT_INITIALIZED(80040605)]] was initially a little perplexing because the canned knowledge information listed with the alert in no way addresses this particular error.  Once I discovered the cause of the problem, however, it has become one of the easiest Exchange Management Pack alerts to resolve.

[MAPI_E_NOT_INITIALIZED(80040605)] indicates that there are conflicting versions of the dynamic link library file CDO.DLL on the server.  From my experience, this almost always means that Microsoft Outlook is installed on the Exchange server, which goes against Microsoft best practices but appears to be somewhat commonplace unfortunately.  Outlook and Exchange each have their own version of CDO.DLL and they butt heads when installed on the same computer.  To resolve the error, we must unregister the Outlook version of CDO.DLL and register the Exchange version from the command prompt.

You can use the Search feature within Windows Explorer to locate the two DLL files.  The Outlook version is likely somewhere in a “Common Files” directory such as C:\Program Files\Common Files\System\MSMAPI\1033.  The Exchange version will be located in the Exchsrvr\bin directory.  At the command prompt, navigate to the directory that contains the Outlook version of CDO.DLL, type regsvr32.exe /u cdo.dll, and press [Enter] to unregister the DLL file.  Now navigate to the directory that contains the Exchange version of CDO.DLL, type regsvr32.exe cdo.dll, and press [Enter] to register this DLL with the system.  Your [MAPI_E_NOT_INITIALIZED(80040605)] alerts should now be a thing of the past.

DHCP Management Pack Bug

Our Operations Center asked me to investigate an alert that they had received from a client’s managed server.  The associated event initially seemed to indicate a problem with the System event log:

Type:  Error
Source:  Health Service Modules
Category:  N/A
Event ID:  25004
Date:  2008-09-09
Time:  09:13:47
User:  N/A
Computer:  CLIENTDC1
Description:
The Windows Event Log Provider is still unable to open the System event log on computer ‘{BE6AEC47-3H7F-97C4-20G4-D37GB234BC98}’.  The Provider has been unable to open the System event log for 4390 seconds.

Most recent error details:  The RPC server is unavailable.

One or more workflows were affected by this.

Workflow name:
Microsoft.Windows.DHCPServer.Library.Server.UnitMonitor.DependentServiceHealth
Instance Name:  clientdc1.somedomain.com
Instance ID:  {BE6AEC47-3H7F-97C4-20G4-D37GB234BC98}
Management Group:  MGMTGP1

During my troubleshooting, I came across a recent posting on the Microsoft Connect web site that indicated that the problem was due to an apparent bug in the current DHCP Management Pack rather than a System event log problem.  The admin that submitted the bug uninstalled the management pack and the alerts stopped.  I did not want to uninstall the entire management pack so I began searching through the monitors looking for the culprit that was causing the noise.

From the Authoring space of the System Center Operations Manager 2007 Console, I selected Monitors in the Authoring pane, located and expanded Microsoft Windows DHCP Server in the Monitors pane and expanded the Entity Health and Availability aggregate rollups.  I then right-clicked on the DHCP Dependent Service Health Monitor and selected Overrides > Override the Monitor > For all objects of type: Microsoft Windows DHCP Server to configure the override of the monitor.

The monitor triggers the 25004 event every 12 minutes.  Once I had created the override, I monitored the System event log for approximately 30 minutes and saw no repeat alerts.

Remember, when working with management packs, it is considered a best practice to create overrides to rules and monitors rather than disable them.  Also, do not save overrides to the Default Management Pack – create a new management pack for the override with a descriptive name.

MS Exchange MTA Service and Event 2000

Earlier this week I encountered an alert in Operations Manager 2007 from a managed client’s Exchange server with Event ID 2000 which stated:

Verify that the Microsoft Exchange MTA service has started.  Consecutive ma-open calls are failing with error 3051.

I knew that the Microsoft Exchange MTA service was not in use in the client environment as it is not necessary in a single-Exchange server organization.  The service was in a ”Disabled” state but this did not prevent the alerts.

My research led me to Microsoft Knowledge Base article 810489  which confirmed that stopping and disabling the service was not enough to resolve the problem.  Two registry entries must be created on the Exchange server for each public or private database as follows:

1. Click on Start > Run and type regedit in the textbox and click the OK button.
2. In Registry Editor, navigate to the following key:  HKLM\SYSTEM\CurrentControlSet\Services\MSExchangeIS\ServerName.  Take the following actions on each private or public database that is listed under this key.
3. Right-click on the database and select New > DWord Value from the pop-up menu.  Name the value Gateway In Threads.
4. Set the Gateway In Threads value to 0 (zero).
5. Right-click on the database and select New > DWord Value from the pop-up menu.  Name the value Gateway Out Threads.
6. Set the Gateway Out Threads value to 0 (zero).
7. Restart the Microsoft Exchange Information Store service to enable the changes.

The Gateway In Threads and Gateway Out Threads values should also be set on any new databases created on the server.

Identify Agent with PowerShell Script

Earlier this week, one of the analysts in our Operations Center informed me about an alert received in System Center Operations Manager 2007 from one of our managed servers.  The alert was regarding the need for agent proxying to be enabled on a server and was similar to the following:

Source:  Servername.contoso.com
Path:  Servername.contoso.com
Last modified by:  System
Last modified time:  8/5/2008 2:36:47 PM
Alert description:  Details: Health service (9AAFF032-4567-2270-B09A-A34025D969F3) could not generate data about this managed object (EC7903B7-72FC-BFD0-3CA2-473774E60AD4).

So how in the world do we translate the GUID into a recognizable computer name?  The answer is Windows PowerShell.  Save the following script with a .ps1 file extension (for example, AgentProxy.ps1):

Param([string]$MonitoringObjectID)
Get-MonitoringObject -id $MonitoringObjectID | select name

Run the script from the command shell console as follows:

C:\MyWorkspace\AgentProxy.ps1 EC7903B7-72FC-BFD0-3CA2-473774E60AD4

The script will return the name of the computer in question and then you can enable agent proxying from within the Administration space of the Operations Manager console.

Resolving ASP Performance Counter Warnings

Recently I was troubleshooting some ASP performance counter warnings that were appearing in the Application event log of one of the servers we manage with System Center Operations Manager 2007.  The events were similar to the following:

Event ID: 35
Source: WinMgmt
Description: WMI ADAP was unable to load the ASP.NET_64_2.0.50727 performance library because it returned invalid data: 0×0

Event ID: 35
Source: WinMgmt
Description: WMI ADAP was unable to load the ASP.NET_64 performance library because it returned invalid data: 0×0

Event ID: 40
Source: WinMgmt
Description: WMI ADAP was unable to create the object Win32_PerfRawData_ASPNET_64_2050727_ASPNETAppsv2050727 for Performance Library ASP.NET_64_2.0.50727 because error 0×80041001 was returned

Event ID: 40
Source: WinMgmt
Description: WMI ADAP was unable to create the object Win32_PerfRawData_ASPNET_64_ASPNETApplications for Performance Library ASP.NET_64 because error 0×80041001 was returned

I came across a posting at MCPGuides.com that explained that these events can occur frequently and, in a large environment, can be quite bothersome.  The errors are related to a known Microsoft bug in the way WMI attempts to query ASP.NET performance counters.  As of the posting on MSPGuides.com (May 23, 2008), no hotfix currently exists.  The posting offered two possible workarounds provided that you do not need to monitor ASP.NET performance counters outside of System Center Operations Manager 2007:

1. Download the Windows 2000 Resource Kit tool “Extensible Performance Counter List” (exctrlst.exe).  Install and launch the tool.  Once open, uncheck the Performance Counters Enabled checkbox for all versions of ASP.NET that are installed.The checkbox is below the Extensible Performance Counters listbox.
2. If your environement is large and installing the tool on each impacted computer is not an option, you can script the registry key that exctrlst.exe modifies.  The net impact of unchecking the Performace Counters Enabled option in the tool is that the DWORD value of Disable Performance Counters is set to 1.  The registry key is located at HKLM\SYSTEM\CurrentControlSet\Services\%Insert ASP.NET node%\Performance.  If the key does not exist, it must be manually created.

Once either of these options have been implemented, you should have a clean Application log on the impacted server(s).  It worked for me!

Exchange Management Pack: Custom URLs for OWA, OMA, and EAS

The Exchange Management Pack used by Operations Manager 2007 automatically determines the URL used to monitor the front-end Exchange services. It does this by using a combination of the localhost/network card IP address and the virtual server and virtual directory information in the IIS metabase.  This URL is the local monitoring URL because the logon request is submitted to the local front-end server, where the logon request is generated.

You can also supply a custom URL to monitor the public address that is used by Web and mobile devices.  To enable monitoring of a custom URL, you must edit the registry on each server where the monitoring is to be enabled.

Configuring a Custom URL for Outlook Web Access (OWA)

To edit the registry, from the Windows Desktop, select Start > Run and type regedit in the textbox.  Click the OK button to open the registry editor.  Locate the \\HKLM\Software\Exchange MOM\FEMonitoring\<FE servername>\ key and create a registry value of type string named CustomURLs.  The custom URL(s) can be entered as a comma-delimited list in this value as follows:

• https://www.mydomain.com/exchange, https://www.mydomain.com/mail

Also, you must set the “CustomURL” override on the Outlook Web Access Logon Monitor to True, which causes the monitor to use the value specified in the new registry key rather than the default values.

Configuring a Custom URL for Outlook Mobile Access (OMA)

Navigate to the same registry key listed above for configuring a custom URL for Outlook Web Access (OWA).  Create a registry value of type string named CustomOmaUrls.  The custom URL(s) can be entered as a comma-delimited list in this value as follows:

• https://www.mydomain.com/oma, https://www.mydomain.com/mobile

Also, you must set the “CustomURL” override on the Outlook Mobile Access Monitor to True, which causes the monitor to use the value specified in the new registry key rather than the default values.

Configuring a Custom URL for Exchange ActiveSync

Navigate to the same registry key listed above for configuring a custom URL for Outlook Web Access (OWA).  Create a registry value of type string named CustomEasUrls.  Enter the custom URL in this value as follows:

• https://www.mydomain.com/Microsoft-Server-ActiveSync

Also, you must set the “CustomURL” override on the Exchange Active Sync Monitor to True, which causes the monitor to use the value specified in the new registry key rather than the default values.

This information reproduced from Microsoft TechNet and placed here for my convenience.

Logical Disk Free Space Override in OM 2007

In Microsoft Operations Manager 2005 (MOM), it was difficult to monitor the logical free disk space on managed servers, especially when the environment included a multitude of disk sizes.  Script parameters could be modified but there might still be disks that always exceeded the alert thresholds.

Operations Manager 2007 allows you greater flexibility by allowing you to override the monitor for a specific object.  In the navigation pane of the Operations Manager console, navigate to Windows Server 2003 Logical Disk > Entity Health > Availability.  Right-click on the Logical Disk Free Space monitor and select Overrides > Override the Monitor > For a specific object of type: Windows Server 2003 Logical Disk.  This will display a dialog box that lists every logical disk instance.  Select the drive that you want to apply the override to and set the properties of the override.

Alternatively, you can create a new group in the Authoring space of the console that contains the specific disks that are to have their thresholds overriden. An override can then be applied to the group by following the steps previously outlined except you should select the For a specific group option when configuring the override.

DCDIAG userAccountControl Issue

While recently troubleshooting some replication alerts in Operations Manager 2007 from a couple of managed domain controllers, I ran DCDIAG and there was one response that indicated a possible issue.  Internet research revealed a posting from Jorge de Almeida Pinto on the PCreview Web site.  I am posting here for my personal convenience and, hopefully, to assist others that might experience the same problem.

In the DCDIAG results, the userAccountControl attribute for one of the domain controllers was 0×82020 (532512) rather than 0×82000 (532480), which is the default value for a domain controller.  Apparently, the precreation of the computer account is one possible cause.  To restore the default domain controller (DC) userAccountControl attribute, use ADSIEDIT as follows:

1. Click on Start > Run, type adsiedit.msc in the Open textbox, and click the OK button.
2. In ADSIEDIT, connect to the domain.
3. Navigate to the Domain Controllers OU.
4. Right-click on the domain controller for which you want to change the userAccountControl attribute and select Properties from the pop-up menu.
5. Locate the userAccountControl attribute.  The value of the attribute should be something other than 532480.  I my case, the value was 532512.
6. Change the value to 532480 and exit ADSIEDIT.

This eliminated the replication alerts that we were receiving from this particular domain controller.  Thanks to Jorge de Almeida Pinto for his insight!

Exchange 2007 Outlook Web Access Connectivity Monitoring

External Outlook Web Access (OWA) connectivity monitoring is enabled by default in the Exchange 2007 Management Pack for Operations Manager 2007; however you will likely receive alerts initially because you must set an external URL on the OWA virtual directory.  To set an external URL, you need to open the Exchange Management Shell and run the Set-OwaVirtualDirectory cmdlet.  The syntax of the cmdlet is as follows:

set-owavirtualdirectory “<Server name>\owa (Default Web Site)” -externalurl:https://<Fully Qualified Domain Name>/owa

For example, if the name of the server is PEGASUS and the domain is named atlanta.contoso.com, the proper syntax of the cmdlet would be as follows:

set-owavirtualdirectory “PEGASUS\owa (Default Web Site)” -externalurl:https://pegasus.atlanta.contoso.com/owa

Note that the examples listed above assume that you have SSL enabled on the OWA virtual directory.  If SSL is not enabled in your domain (and it should be), substitute http:// in place of https://.