I am also in the process of migrating most of the content from my long-standing Web site, jimdoylemcse.com, to the new site as well.  The original site has already been taken down.  It will take some time to get “moved in” completely, so check in often and follow the progress!  Thank you for supporting this site and I hope  your support will continue at http://www.jim-doyle.com.

From within an RDP (Remote Desktop Protocol) session on the remote server, I accessed our server in Windows Explorer via UNC path, located the file that I needed to copy to the remote server, right-clicked and selected the Copy option from the menu.  I then navigated to the root of the C: drive on the remote server, which was where I wanted to paste the file.  I right-clicked and selected Paste from the menu…but received the warning dialog box below.

Now, I understand that it can be dangerous to copy and paste certain remote files to a server and such action is oftentimes a sign of malicious activity, but where is the “security” in telling me that I do not have permission to copy files to a particular location over the network but can paste into the Documents folder and then move it to the original desired location?  I guess it could deter a malicious individual if said individual just could not wait the additional 10 seconds that it would take to get his package to the desired location!

Peter Frampton

Before Frampton took the stage, a band called The Elms did a thirty-minute set.  This band was founded in 2001 and hails from Seymour, IN.  The Elms have become one of the most respected and consistent young rock & roll bands from the midwestern U.S., both on record and on stage.  Katie and I were quite impressed with these four young lads.  The band did a very respectable job of getting the crowd ready for Frampton.

At 8:00 sharp, after a 30-minute break while the roadies shuffled equipment, Frampton and his band took the stage.  Their set included all of the Frampton classics as well as a couple of tracks from his latest CD Fingerprints, which was released in 2006, and a couple of tracks from his upcoming CD which, according to Frampton, is currently about 80% complete.  The band included long-time bassist John Regan, who has been making music with Frampton for 30 years.  Rob Arthur manned the keyboards and did a great job of filling the big shoes of the late, great Bob Mayo.  Adam Lester joined Frampton on guitar, a position also formerly occupied by Mayo.  For drummer Dan Wojciechowski, the band’s newest member, the evening was a homecoming of sorts because he is originally from Fayetteville.

This was truly one of the best, if not the absolute best, concert that I have ever attended and having my daughter along made it even more special.  Frampton has a number of concert dates scheduled through the end of September.  Should he come to a venue near you, I highly recommend that you make every effort to attend.  I have posted photographs and a couple of brief videos from the show on my Facebook page and my website.  Feel free to take a look.

After the upgrade it was discovered that the tasks were no longer running.  I began to troubleshoot the problem.  From within the application, I could right-click on a task and select to run a test job from the pop-up menu.  The task completed successfully.  If I selected the option to start the job in the context that it was to run each night, it failed consistently.  So why were the test jobs succeeding when the actual tasks were failing?  I turned to good, old Google and quicky found the root cause of the problem.

I discovered that tasks run in Windows Server 2008′s Task Scheduler, by default, run only when the user is logged onto the server.  This was not the case with Scheduled Tasks in Windows Server 2003.  To remedy the problem, I accessed Task Scheduler on the new server, right-clicked on each task and selected Properties from the pop-up menu.  On the General tab of the task’s properties, I selected the Run whether user is logged on or not option as shown below, and then clicked the OK button.  Now the tasks run nightly as expected.

Task Properties in Windows Server 2008

Although this subtle change in the way tasks are run in Windows Server 2008 is an improvement from a security standpoint, it can cause some head scratching if you are unaware of the change.  I hope this post saves saves some administrators a little head scratching.

Event ID:  1228
Source:  ADAM
Type:  Error
Description:  System Monitor was unable to open Active Directory performance counters.  An attempt to query the following performance counter registry key failed.
Registry key:
SYSTEM\CurrentControlSet\Services\ADAM\Performance\First Counter
Additional Data
Error value:  2 The system cannot find the file specified.

Sure enough, on my problematic server there was not a First Counter key in the registry at the location listed in the alert.  The similar posts that I came across during my research indicated that there might be an .INI file for the performance counters located at C:\WINDOWS\System32.  I had already noticed that the registry location above contained a PerfIniFile string value with a value of adamctrs.ini.  I checked the System32 directory and found no such file.  A search of the C: drive located it at located it at C:\WINDOWS\System32\ADAM.

I opened a command prompt and sourced from the C:\WINDOWS\System32\ADAM directory and ran the following command:

lodctr adamctrs.ini

I then returned to the Registry Editor and, from the menu, selected View > Refresh.  Four new DWORD values, as shown above, had been created – First Counter (the one causing my alert), First Help, Last Counter, and Last Help.  Alert resolved!  Finding the resolution to the problem took longer than the actual correction of the problem.  Again, I hope this post will help someone else resolve this error in a timelier manner with less research involved.

We have been receiving the following alert from a client Exchange server for a number of days and it was initially diagnosed as an agent issue:

MAPI Logon Failure
MAPI logon verification: Problem in accessing Directory
Service caused Logon failure Exchange Server: “SERVERNAME“
MDB:”First Storage Group\Mailbox Store (SERVERNAME)”
Mailbox:”SERVERNAMEMOM” Error: The information store could not
be opened. [MAPI 1.0 - [MAPI_E_LOGON_FAILED
(80040111)]]

I checked the usual settings – mailbox rights, mailbox security, mailbox access account credentials, etc. – and everything was configured properly.  However, one thing caught my eye while checking the properties of the test mailbox user account.  On the Exchange Advanced tab of the Properties dialog box (below), the Hide from Exchange address lists option was selected.

Exchange Advanced tab

This did not seem quite right to me, so I checked test mailbox account properties in a couple of other client domains.  This option was deselected on every account that I checked.  Apparently, at some point, someone had selected this option so the account, which is disabled by default when created by the Exchange Management Pack Configuration Wizard, would not appear in the Outlook Global Address List. I promptly deselected the option and monitored the Operations Manager event log on the server.  Within minutes Event ID 19980 was logged indicating that all attempted MAPI logins of test mailboxes on the server had succeeded.

As is so often the case, an overlooked checkbox or radio button was the root of the problem and an alert that had repeated hundreds of times was finally closed.  During my troubleshooting, I did not come across anything that referenced the Hide from Exchange address lists option, although it surely exists somewhere in cyberspace, so I hope this post is beneficial to others trying to resolve a similar issue.

Event Type:    Error
Event Source:    Windows SharePoint Services 3
Event Category:    Timer
Event ID:    6398
Date:        11/19/2008
Time:        1:05:53 PM
User:        N/A
Computer:    <ServerName>
Description:
The Execute method of job definition Microsoft.Office.Server.Search.Administration.IndexingScheduleJobDefinition (ID 03d6f1b3-879c-42d9-8921-3f14b9e5678f) threw an exception. More information is included below.

Retrieving the COM class factory for component with CLSID {3D42CCB1-4665-4620-92A3-478F47389230} failed due to the following error: 80070005.

The System event log contained numerous instances of the following DCOM error as well:

Event Type:    Error
Event Source:    DCOM
Event Category:    None
Event ID:    10016
Date:        11/19/2008
Time:        1:06:26 PM
User:        NT AUTHORITY\NETWORK SERVICE
Computer:    ATI-SHAREPOINT
Description:
The application-specific permission settings do not grant Local Activation permission for the COM Server application with CLSID
{3D42CCB1-4665-4620-92A3-478F47389230}
to the user NT AUTHORITY\NETWORK SERVICE SID (S-1-5-20).  This security permission can be modified using the Component Services administrative tool.

This problem can be resolved by accessing the Component Services applet by way of Start > Administrative Tools > Component Services.  In the task pane expand Component Services, expand Computers, and expand My Computer, and click on DCOM Config.

In the details pane on right, locate and right-click on the OSearch application and select Properties from the menu.  On the OSearch Properties dialog box, click on the Security tab.  In the Launch and Activation Permissions section, select the Customize option and then click the Edit button.

OSearch Launch Permissions

On the Launch Permissions dialog box, add the NETWORK SERVICE user and grant it Local Launch and Local Activation permissions as shown above.  Click the OK button twice and then close the Component Services applet.  You should now have a much quieter and happier event log!

We went through the normal steps for removing a server from Operations Manager 2007 – uninstalled the OM Agent, deleted the server from the OM database, deleted automated reports, etc.  Within 24 hours, however, the Active Directory Replication Monitor began alerting our Operations Center that a domain controller had failed to update its MOMLatencyMonitor object within the specified time period (24 hours by default).  The retired server’s MOMLatencyMonitor object had been deleted from Active Directory during its removal from Managed Services so how was the ADMP still aware of the unmanaged, albeit still running, domain controller?

Windows Server 2003 adds two additional Active Directory application naming contexts, DomainDNSZones and ForestDNSZones, when you Active Directory Integrate a DNS zone on a domain controller.  We use ADSI Edit to modify the objects and attributes of these naming contexts as follows:

1. Click Start > Run, type adsiedit.msc in the textbox, and click the OK button.
2. In the ADSI Edit task pane, right-click ADSI Edit and select Connect to from the menu.
3. In the Connection Point section of the Connection Settings dialog box shown below, select the Select or type a Distinguished Name or Naming Context option, type the following in the textbox, substituting the appropriate domain and extension values, and then click the OK button:

   DC=DomainDNSZones,DC=Domain,DC=Domain_extension

   

   ADSI Edit Connection Settings

4. In the ADSI Edit task pane, locate and click on the CN=MOMLatencyMonitors container as shown below.
   

   ADSI Edit

5. In the details pane to the right, locate and delete the directory for the server that is no longer managed.
6. Repeat steps 3-5 and specify ForestDNSZones in the Connection Point string rather than DomainDNSZones.
7. Close the ADSI Edit window.

Now the ADMP Replication Monitor will only monitor and alert on the remaining managed domain controllers.

My initial investigation revealed, for example, that the server was not communicating with our server that hosts an SNMP-based performance application.  However, the Operations Manager agent on the server was communicating with our Root Management Server (RMS), albeit with the secondary node of the cluster rather than the desired primary node.  ICMP pings from the remote server to the primary RMS node and the server hosting the SNMP application failed, while pings to the secondary RMS node succeeded.  I also observed that the server could not ping one of the client’s domain controllers located in a colocation facility in the south central United States.

I turned to Google and began trying to find a solution to the problem.  I came across two newsgroup postings, one involving Linux Ubuntu and the other Windows 2000 Server, that described an odd behavior where a computer would only receive ICMP replies from odd or even addressed hosts.  If the fourth octet of the server’s IP address was an odd number, it would only receive ICMP replies from hosts with an even-numbered fourth octet, or vice versa.  This sounded totally bizarre, but I started pinging our servers from the remote server which, by the way has an even-numbered fourth octet.  Sure enough!  Every server with an odd-numbered fourth octet replied while even-numbered servers did not!  Even our senior-most engineers had never heard of, much less encountered, such a phenomenon.

According each posting, the problem was eventually isolated to NIC teaming and, as it turned out, our problem server had a pair of NICs teamed as well.  Ultimately, we dissolved the NIC team, assigned the IP addressing of the team to one of the physical NICs, and disabled the other NIC.  This may not be the optimal solution to the problem, but it got our client’s problem server to communicate properly again.

If you have encountered this odd/even addressing issue with NIC teaming and resolved your problem in a different manner, please provide a comment!

The following DCs have not updated their MOMLatencyMonitor objects within the specified time period (8 hours).  This is probably caused by either replication not occurring, or because the ‘AD Replication Monitoring’ script is not running on the DC.

Format: DC, Naming Context, Hours since last update

MySite
PEGASUS, NDNC:DC=DomainDnsZones,DC=myDomain,DC=com, 11

This alert indicates that my domain controller PEGASUS has not replicated the required value to the adminDescription attribute under the MOMLatencyMonitor container in Active Directory for eleven hours or more.  But how do we know if the alert is due to actual AD replication issues or a problematic Operations Manager script?

We can validate the problem by using the repadmin utility, which is included in the Windows Server 2003 Support Tools.  At a command prompt, enter the following command (substitute the appropriate servername):

repadmin.exe /showrepl PEGASUS

The output of the repadmin utility should be similar to the following:

DC=ForestDnsZones,DC=myDomain,DC=com
mySite\Odyssey via RPC
DC object GUID: 45b3241y-z849-48rs-n76u-6x09bv432ih6
Last attempt @ 2008-10-28 08:34:55 was successful.

According to the output above, the last replication attempt was very close to the current time and was successful.  This indicates that the problem is likely with the Operations Manager script.  The dsquery command line utility can be used to troubleshoot further:

dsquery * CN=MomLatencyMonitors,DC=ForestDnsZones, DC=myDomain,DC=com -scope onelevel -attr name AdminDescription

The output of dsquery should be similar to the following:

name                                   admindescription
PEGASUS                           20081028.0200
ODYSSEY                           20081028.1330

Based on the timestamp values in these results, we are able to detemine that this particular alert is actually an Operations Manager script error.