SharePoint Execute Method Exception (Event ID 6398)

I spent some time today troubleshooting an alert that we received in Operations Manager 2007 from one of the client SharePoint servers that we manage in our Operations Center.  The alert referenced Event ID 6398 which indicates that the execute method of a SharePoint timer job definition had thrown an exception.  I logged onto the client SharePoint server and looked in the Application event log and, sure enough, it was full of the following 6398 event:

Event Type:    Error
Event Source:    Windows SharePoint Services 3
Event Category:    Timer
Event ID:    6398
Date:        11/19/2008
Time:        1:05:53 PM
User:        N/A
Computer:    <ServerName>
Description:
The Execute method of job definition Microsoft.Office.Server.Search.Administration.IndexingScheduleJobDefinition (ID 03d6f1b3-879c-42d9-8921-3f14b9e5678f) threw an exception. More information is included below.

Retrieving the COM class factory for component with CLSID {3D42CCB1-4665-4620-92A3-478F47389230} failed due to the following error: 80070005.

The System event log contained numerous instances of the following DCOM error as well:

Event Type:    Error
Event Source:    DCOM
Event Category:    None
Event ID:    10016
Date:        11/19/2008
Time:        1:06:26 PM
User:        NT AUTHORITY\NETWORK SERVICE
Computer:    ATI-SHAREPOINT
Description:
The application-specific permission settings do not grant Local Activation permission for the COM Server application with CLSID
{3D42CCB1-4665-4620-92A3-478F47389230}
to the user NT AUTHORITY\NETWORK SERVICE SID (S-1-5-20).  This security permission can be modified using the Component Services administrative tool.

This problem can be resolved by accessing the Component Services applet by way of Start > Administrative Tools > Component Services.  In the task pane expand Component Services, expand Computers, and expand My Computer, and click on DCOM Config.

In the details pane on right, locate and right-click on the OSearch application and select Properties from the menu.  On the OSearch Properties dialog box, click on the Security tab.  In the Launch and Activation Permissions section, select the Customize option and then click the Edit button.

OSearch Launch Permissions

On the Launch Permissions dialog box, add the NETWORK SERVICE user and grant it Local Launch and Local Activation permissions as shown above.  Click the OK button twice and then close the Component Services applet.  You should now have a much quieter and happier event log!

ADMP Replication Monitor and Unmanaged DC

Operation Manager 2007’s Active Directory Management Pack (ADMP) is designed for managing all domain controllers within a given site or organization.  However, we recently encountered one of the rare instances when not all DCs are managed – a client was in the process of retiring one of three DCs in their domain and wanted to remove it from our Managed Services offering.

We went through the normal steps for removing a server from Operations Manager 2007 – uninstalled the OM Agent, deleted the server from the OM database, deleted automated reports, etc.  Within 24 hours, however, the Active Directory Replication Monitor began alerting our Operations Center that a domain controller had failed to update its MOMLatencyMonitor object within the specified time period (24 hours by default).  The retired server’s MOMLatencyMonitor object had been deleted from Active Directory during its removal from Managed Services so how was the ADMP still aware of the unmanaged, albeit still running, domain controller?

Windows Server 2003 adds two additional Active Directory application naming contexts, DomainDNSZones and ForestDNSZones, when you Active Directory Integrate a DNS zone on a domain controller.  We use ADSI Edit to modify the objects and attributes of these naming contexts as follows:

1. Click Start > Run, type adsiedit.msc in the textbox, and click the OK button.
2. In the ADSI Edit task pane, right-click ADSI Edit and select Connect to from the menu.
3. In the Connection Point section of the Connection Settings dialog box shown below, select the Select or type a Distinguished Name or Naming Context option, type the following in the textbox, substituting the appropriate domain and extension values, and then click the OK button:

   DC=DomainDNSZones,DC=Domain,DC=Domain_extension

   

   ADSI Edit Connection Settings

4. In the ADSI Edit task pane, locate and click on the CN=MOMLatencyMonitors container as shown below.
   

   ADSI Edit

5. In the details pane to the right, locate and delete the directory for the server that is no longer managed.
6. Repeat steps 3-5 and specify ForestDNSZones in the Connection Point string rather than DomainDNSZones.
7. Close the ADSI Edit window.

Now the ADMP Replication Monitor will only monitor and alert on the remaining managed domain controllers.

Odd (or even) NIC Teaming Problem

We recently encountered a situation that apparently seldom occurs or, at the very least, is not well documented on the Internet.  Our Operations Center informed me about a remote client server located in the eastern United States that was not reporting correctly into all of our management and monitoring tools.

My initial investigation revealed, for example, that the server was not communicating with our server that hosts an SNMP-based performance application.  However, the Operations Manager agent on the server was communicating with our Root Management Server (RMS), albeit with the secondary node of the cluster rather than the desired primary node.  ICMP pings from the remote server to the primary RMS node and the server hosting the SNMP application failed, while pings to the secondary RMS node succeeded.  I also observed that the server could not ping one of the client’s domain controllers located in a colocation facility in the south central United States.

I turned to Google and began trying to find a solution to the problem.  I came across two newsgroup postings, one involving Linux Ubuntu and the other Windows 2000 Server, that described an odd behavior where a computer would only receive ICMP replies from odd or even addressed hosts.  If the fourth octet of the server’s IP address was an odd number, it would only receive ICMP replies from hosts with an even-numbered fourth octet, or vice versa.  This sounded totally bizarre, but I started pinging our servers from the remote server which, by the way has an even-numbered fourth octet.  Sure enough!  Every server with an odd-numbered fourth octet replied while even-numbered servers did not!  Even our senior-most engineers had never heard of, much less encountered, such a phenomenon.

According each posting, the problem was eventually isolated to NIC teaming and, as it turned out, our problem server had a pair of NICs teamed as well.  Ultimately, we dissolved the NIC team, assigned the IP addressing of the team to one of the physical NICs, and disabled the other NIC.  This may not be the optimal solution to the problem, but it got our client’s problem server to communicate properly again.

If you have encountered this odd/even addressing issue with NIC teaming and resolved your problem in a different manner, please provide a comment!