ADMP Replication Failure Verification

A common alert generated by the Active Directory Management Pack (ADMP) indicates that there is a problem with Active Directory replication:

The following DCs have not updated their MOMLatencyMonitor objects within the specified time period (8 hours).  This is probably caused by either replication not occurring, or because the ‘AD Replication Monitoring’ script is not running on the DC.

Format: DC, Naming Context, Hours since last update

MySite
PEGASUS, NDNC:DC=DomainDnsZones,DC=myDomain,DC=com, 11

This alert indicates that my domain controller PEGASUS has not replicated the required value to the adminDescription attribute under the MOMLatencyMonitor container in Active Directory for eleven hours or more.  But how do we know if the alert is due to actual AD replication issues or a problematic Operations Manager script?

We can validate the problem by using the repadmin utility, which is included in the Windows Server 2003 Support Tools.  At a command prompt, enter the following command (substitute the appropriate servername):

repadmin.exe /showrepl PEGASUS

The output of the repadmin utility should be similar to the following:

DC=ForestDnsZones,DC=myDomain,DC=com
mySite\Odyssey via RPC
DC object GUID: 45b3241y-z849-48rs-n76u-6×09bv432ih6
Last attempt @ 2008-10-28 08:34:55 was successful.

According to the output above, the last replication attempt was very close to the current time and was successful.  This indicates that the problem is likely with the Operations Manager script.  The dsquery command line utility can be used to troubleshoot further:

dsquery * CN=MomLatencyMonitors,DC=ForestDnsZones, DC=myDomain,DC=com -scope onelevel -attr name AdminDescription

The output of dsquery should be similar to the following:

name                                   admindescription
PEGASUS                           20081028.0200
ODYSSEY                           20081028.1330

Based on the timestamp values in these results, we are able to detemine that this particular alert is actually an Operations Manager script error.

MachineKeys Directory Permissions

Today I was troubleshooting a problematic remotely managed server that was not reporting into our Operations Center.  The OpsMgr Health Service service was running; however, we were not receiving a “heartbeat” from the server.  The Operations Manager event log contained error events that indicated a problem involving the certificate private key used to unencrypt the package received from our Root Management Server (RMS).

When I accessed the System event log on the server, I also discovered that the following event was being logged several times every hour:

Event Type:    Error
Event Source:    Schannel
Event Category:    None
Event ID:    36870
Date:        10/28/2008
Time:        9:36:08 AM
User:        N/A
Computer:    Servername
Description:
A fatal error occurred when attempting to access the SSL client credential private key. The error code returned from the cryptographic module is 0xffffffff.

My research indicated that this event is the result of improper NTFS permissions being set on the MachineKeys directory, which is located under the All Users Profile\Application Data\Microsoft\Crypto\RSA directory.  This directory is utilized by both Certificate services and Internet Explorer.  The default permissions for the MachineKeys directory are as follows:

Administrator (Full Control) This folder only
Everyone (Special) This folder, subfolders, and files
SYSTEM (Full Control) This folder, subfolders, and files

Although resetting the permissions on the MachineKeys directory did not totally resolve the Operations Manager agent heartbeat problem (I also had to uninstall and reinstall the agent), it did eliminate the Schannel events in the System event log.

Addtional information can be found in Microsoft Knowledge Base Article 278381.

Exchange Management Pack MAPI Logon Alert

The Operations Manager management packs for Exchange Server are usually the noisiest of all of the management packs.  As a result, they also require considerable attention from an Operations Manager administrator.  In our Managed Services operation at ClearPointe, which manages many Exchange organizations of various size, the synthetic logons (OWA, OAS, and EAS) and MAPI logon verifications seem to generate the most alert traffic and are forwarded to me for resolution.

There are several MAPI logon verification alerts but the one shown below that includes the error text of [Collaboration Data Objects – [MAPI_E_NOT_INITIALIZED(80040605)]] was initially a little perplexing because the canned knowledge information listed with the alert in no way addresses this particular error.  Once I discovered the cause of the problem, however, it has become one of the easiest Exchange Management Pack alerts to resolve.

[MAPI_E_NOT_INITIALIZED(80040605)] indicates that there are conflicting versions of the dynamic link library file CDO.DLL on the server.  From my experience, this almost always means that Microsoft Outlook is installed on the Exchange server, which goes against Microsoft best practices but appears to be somewhat commonplace unfortunately.  Outlook and Exchange each have their own version of CDO.DLL and they butt heads when installed on the same computer.  To resolve the error, we must unregister the Outlook version of CDO.DLL and register the Exchange version from the command prompt.

You can use the Search feature within Windows Explorer to locate the two DLL files.  The Outlook version is likely somewhere in a “Common Files” directory such as C:\Program Files\Common Files\System\MSMAPI\1033.  The Exchange version will be located in the Exchsrvr\bin directory.  At the command prompt, navigate to the directory that contains the Outlook version of CDO.DLL, type regsvr32.exe /u cdo.dll, and press [Enter] to unregister the DLL file.  Now navigate to the directory that contains the Exchange version of CDO.DLL, type regsvr32.exe cdo.dll, and press [Enter] to register this DLL with the system.  Your [MAPI_E_NOT_INITIALIZED(80040605)] alerts should now be a thing of the past.

Group Policy Replication and DNS

I ran into a problem today while trying to get a remotely managed server to receive new Group Policy Objects (GPOs).  In the Group Policy Management Console (GPMC), the GPO was linked at the domain level, the Authenticated Users group was removed from the security filtering section of the policy, and the computer’s machine account was explicitly added along with the machine accounts of other servers.  From a command prompt on the destination server, I ran gpupdate /force to force the server to update its group policy settings.  The event logs indicated that the server had successfully updated its policies; however, I could immediately tell that the server did not receive the new GPOs by checking a couple of registry keys.

I ran the Resultant Set of Policy (RSOP) wizard in the GPMC and it indicated that the server was being denied the GPOs due to security filtering.  This did not make sense as I had explicitly added the server to the GPO’s security filtering.  I checked the advanced permissions to ensure that the server did not have a Deny permission buried somewhere.  Everything was configured just as it should be!  I then checked the DNS Event Log and noticed the 4515 events shown below (the computer name and domain name have been blurred to protect the guilty).

I checked the %systemroot%\dns directory on the server and discovered a static DNS zone file, which also happened to include an incorrect IP address for the server.   DNS in the domain is Active Directory-Integrated so I knew the zone file was not needed but was unsure whether it could be the cause of the GPO problem.  I deleted the zone file and forced another group policy update from the command prompt.  The server successfully received the new GPOs.   The moral of the story is, if all else fails when troubleshooting group policy issues, check the DNS event log!