Active Directory Performance Counter Error

Sorry about the lack of new posts lately!  I was troubleshooting an alert in Operations Manager 2007 from one of the many servers we manage.  Research on the Internet did not reveal any resolution to this specific alert although I did find a couple of postings that were somewhat similar and led me down the correct path to resolution.  Hopefully this post will help others that are troubleshooting this same alert.

Event ID:  1228
Source:  ADAM
Type:  Error
Description:  System Monitor was unable to open Active Directory performance counters.  An attempt to query the following performance counter registry key failed.
Registry key:
SYSTEM\CurrentControlSet\Services\ADAM\Performance\First Counter
Additional Data
Error value:  2 The system cannot find the file specified.

Sure enough, on my problematic server there was not a First Counter key in the registry at the location listed in the alert.  The similar posts that I came across during my research indicated that there might be an .INI file for the performance counters located at C:\WINDOWS\System32.  I had already noticed that the registry location above contained a PerfIniFile string value with a value of adamctrs.ini.  I checked the System32 directory and found no such file.  A search of the C: drive located it at located it at C:\WINDOWS\System32\ADAM.

I opened a command prompt and sourced from the C:\WINDOWS\System32\ADAM directory and ran the following command:

lodctr adamctrs.ini

I then returned to the Registry Editor and, from the menu, selected View > Refresh.  Four new DWORD values, as shown above, had been created – First Counter (the one causing my alert), First Help, Last Counter, and Last Help.  Alert resolved!  Finding the resolution to the problem took longer than the actual correction of the problem.  Again, I hope this post will help someone else resolve this error in a timelier manner with less research involved.

Odd (or even) NIC Teaming Problem

We recently encountered a situation that apparently seldom occurs or, at the very least, is not well documented on the Internet.  Our Operations Center informed me about a remote client server located in the eastern United States that was not reporting correctly into all of our management and monitoring tools.

My initial investigation revealed, for example, that the server was not communicating with our server that hosts an SNMP-based performance application.  However, the Operations Manager agent on the server was communicating with our Root Management Server (RMS), albeit with the secondary node of the cluster rather than the desired primary node.  ICMP pings from the remote server to the primary RMS node and the server hosting the SNMP application failed, while pings to the secondary RMS node succeeded.  I also observed that the server could not ping one of the client’s domain controllers located in a colocation facility in the south central United States.

I turned to Google and began trying to find a solution to the problem.  I came across two newsgroup postings, one involving Linux Ubuntu and the other Windows 2000 Server, that described an odd behavior where a computer would only receive ICMP replies from odd or even addressed hosts.  If the fourth octet of the server’s IP address was an odd number, it would only receive ICMP replies from hosts with an even-numbered fourth octet, or vice versa.  This sounded totally bizarre, but I started pinging our servers from the remote server which, by the way has an even-numbered fourth octet.  Sure enough!  Every server with an odd-numbered fourth octet replied while even-numbered servers did not!  Even our senior-most engineers had never heard of, much less encountered, such a phenomenon.

According each posting, the problem was eventually isolated to NIC teaming and, as it turned out, our problem server had a pair of NICs teamed as well.  Ultimately, we dissolved the NIC team, assigned the IP addressing of the team to one of the physical NICs, and disabled the other NIC.  This may not be the optimal solution to the problem, but it got our client’s problem server to communicate properly again.

If you have encountered this odd/even addressing issue with NIC teaming and resolved your problem in a different manner, please provide a comment!

MachineKeys Directory Permissions

Today I was troubleshooting a problematic remotely managed server that was not reporting into our Operations Center.  The OpsMgr Health Service service was running; however, we were not receiving a “heartbeat” from the server.  The Operations Manager event log contained error events that indicated a problem involving the certificate private key used to unencrypt the package received from our Root Management Server (RMS).

When I accessed the System event log on the server, I also discovered that the following event was being logged several times every hour:

Event Type:    Error
Event Source:    Schannel
Event Category:    None
Event ID:    36870
Date:        10/28/2008
Time:        9:36:08 AM
User:        N/A
Computer:    Servername
Description:
A fatal error occurred when attempting to access the SSL client credential private key. The error code returned from the cryptographic module is 0xffffffff.

My research indicated that this event is the result of improper NTFS permissions being set on the MachineKeys directory, which is located under the All Users Profile\Application Data\Microsoft\Crypto\RSA directory.  This directory is utilized by both Certificate services and Internet Explorer.  The default permissions for the MachineKeys directory are as follows:

Administrator (Full Control) This folder only
Everyone (Special) This folder, subfolders, and files
SYSTEM (Full Control) This folder, subfolders, and files

Although resetting the permissions on the MachineKeys directory did not totally resolve the Operations Manager agent heartbeat problem (I also had to uninstall and reinstall the agent), it did eliminate the Schannel events in the System event log.

Addtional information can be found in Microsoft Knowledge Base Article 278381.

Group Policy Replication and DNS

I ran into a problem today while trying to get a remotely managed server to receive new Group Policy Objects (GPOs).  In the Group Policy Management Console (GPMC), the GPO was linked at the domain level, the Authenticated Users group was removed from the security filtering section of the policy, and the computer’s machine account was explicitly added along with the machine accounts of other servers.  From a command prompt on the destination server, I ran gpupdate /force to force the server to update its group policy settings.  The event logs indicated that the server had successfully updated its policies; however, I could immediately tell that the server did not receive the new GPOs by checking a couple of registry keys.

I ran the Resultant Set of Policy (RSOP) wizard in the GPMC and it indicated that the server was being denied the GPOs due to security filtering.  This did not make sense as I had explicitly added the server to the GPO’s security filtering.  I checked the advanced permissions to ensure that the server did not have a Deny permission buried somewhere.  Everything was configured just as it should be!  I then checked the DNS Event Log and noticed the 4515 events shown below (the computer name and domain name have been blurred to protect the guilty).

I checked the %systemroot%\dns directory on the server and discovered a static DNS zone file, which also happened to include an incorrect IP address for the server.   DNS in the domain is Active Directory-Integrated so I knew the zone file was not needed but was unsure whether it could be the cause of the GPO problem.  I deleted the zone file and forced another group policy update from the command prompt.  The server successfully received the new GPOs.   The moral of the story is, if all else fails when troubleshooting group policy issues, check the DNS event log!

Performance Issues & TCP Chimney

I was recently troubleshooting a performance issue on one of the servers we manage and I came across a posting on Greg Galipeau’s Weblog that covered problems that a couple of network adapters have with TCP Chimney.  TCP Chimney is a concept introduced with Service Pack 2 for Windows Server 2003 and is supposed to increase a server’s performance.  However, two network adapters have a known issue with TCP Chimney:

• BroadCom NetXtreme II
• Hewlett-Packard NC373i Multifunction Gigabit Server Adapter

As it turned out, the server I was troubleshooting had the HP adapter installed so I followed Greg’s instructions, which are outlined below:

1. Disable TCP Chimney from a command prompt by typing Netsh int ip set chimney DISABLED
2. Make a few registry changes to make sure that performance is not hindered by disabling TCP Chimney.  Click Start, click Run, type regedit in the textbox, and then click the OK button.
3. Locate the following registry subkey:  HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
4. Double-click the EnableTCPChimney registry entry.
5. In the Edit DWORD Value dialog box, type 0 in the Value data textbox and then click OK.
6. Double-click the EnableRSS registry entry.
7. In the Edit DWORD Value dialog box, type 0 in the Value data textbox and then click OK.
8. Double-click the EnableTCPA registry entry.
9. In the Edit DWORD Value dialog box, type 0 in the Value data textbox and then click OK.
10. Restart the server.

For additional information regarding TCP Chimney issues, please review Microsoft Knowledge Base article 942861.

Security Vulnerability Research & Defense Blog

Yesterday Microsoft launched a blog (http://blogs.technet.com/swi/) about the internals of their IT security research and patch development process. There are already some posts that you will not find in the official security bulletins or KB articles. One of the posts says, ‘We periodically identify workarounds or mitigations like this that we can’t use for official guidance because they’re either too nuanced or have some exception cases. When we discover something potentially useful but are uncomfortable listing it in the bulletin, we’ll do our best to describe it here in this blog.’ It looks like Microsoft is making an effort to become more ‘open’ in the area of security research and communication.

HyperTerminal and Vista

Microsoft chose not to include HyperTerminal in the Vista operating system.  For most users it is probably no big deal and hasn’t been a problem for me up until now.  I am getting more involved in router configuration and run Windows Vista on my workstation at ClearPointe.  I did a quick search on the Internet and discovered that there are several free applications available for download to overcome Vista’s lack of HyperTerminal.  However, I also discovered that HyperTerminal can be ported from a computer running Windows XP.  Here are the steps:

1. Map a drive between your Windows XP computer and your Windows Vista computer. 
2. On the Windows XP box, navigate to C:\Program Files\Windows NT and copy hypertrm.exe.
3. Paste hypertrm.exe in the C:\Program Files\Windows NT directory on the Vista machine.
4. Back on the Windows XP computer, navigate to C:\WINDOWS\System32 and copy hypertrm.dll.
5. Paste hypertrm.dll in the C:\Windows\System32 directory on the Vista box.

You now have HyperTerminal installed on your Windows Vista system!

Display Computername on Desktop

We monitor and manage hundreds of servers across the country from our Operations Center.  It is not uncommon to have RDP sessions open to several remote servers at any given time.  In order to more easily identify each server and reduce the possibility of mistakes, we implemented a registry hack that adds the computername to the My Computer icon on the Desktop.

1. In the registry, navigate to HKEY_CLASSES_ROOT\CLSID\{20D04FE0-3AEA-1069-A2D8-08002B30309D}
2. Rename the LocalizedString value to LocalizedString.old
3. Create a new REG_EXPAND_SZ value named LocalizedString and set the value to My Computer %COMPUTERNAME%
4. Close the registry editor, right-click on an empty area of the Desktop and select Refresh from the pop-up menu. 

The My Computer icon should now reflect the computer’s name.Alternatively, if you want to also add the logged on user’s name to the My Computer icon, set the  value of LocalizedString to %USERNAME on %COMPUTERNAME%.

On pre-Windows 2000 SP3 servers it might be necessary to copy the contents of the LocalizedString.old value, @C:\WINNT\system32\shell32.dll,-9217@1033,My Computer, and substitute %USERNAME% at %COMPUTERNAME% for My Computer in the value of the new LocalizedString value.