Perfmon and Procmon
If you are a Windows or Citrix administrator, you might have heard the terms Perfmon and Procmon before. These are Microsoft tools that have been around from the NT era and have been used by all administrators for their daily troubleshooting issues.
Perfmon or Performance Monitor is used to monitor system performance such as CPU, memory, and network bandwidth being used by an operating system, applications, services, and so on. Perfmon is installed by default during Windows installation.
Procmon or Process Monitor is a free tool provided by Microsoft Sysinternals. The tool is used to monitor real-time activity of all the processes on the Windows platform. It combines two old tools called RegMon and FileMon, used to track registry and file access activities.
Procmon can be used to track failed attempts against accessing the registry, file shares, detect file errors, critical system processes, and many more.
Configuring Perfmon
Perfmon is installed by default on all Windows servers. To configure Perfmon, you can launch it by going to All Programs | Administrative Tools | Performance Monitor or simply by typing perfmon in the Run window.
Perfmon comes with a default system performance counter set to monitor CPU utilization, as shown in the following screenshot:
To configure Perfmon to capture all relevant performance parameters, we need to configure the data collector set to add all the relevant performance counters. Microsoft provides a built-in user level data collector set that comes with Windows 2008 Server as Server Manager Performance monitor.
If you just want to monitor CPU and memory performance, this data collector set will provide you with sufficient information:
However, to configure advanced data collector sets to monitor performance counters that are not limited to CPU, memory, or network, you need to add relevant counters to your customized data collector set.
Please refer to the following screenshot of a customized data collector set to be used to monitor CPU, memory, paging file utilization, ICA connections, ICA session latency, Citrix Profile Management logon/logoff duration, HDX Flash input/output data bandwidth, and more:
The data collector set can capture all the parameters for server performance, logon/logoff duration, HDX MediaStream bandwidth and much more.
In today's world, this tool is rarely used by Citrix administrators to get performance-related data, as Citrix Director can provide you with the relevant data and you don't even need to spend time in configuring these data sets.
However, I still recommend this tool to advanced administrators who want to explore all the built-in counters provided by Microsoft and Citrix to play around. Sometime, the advanced tools such as Citrix Director or EdgeSight don't provide you with all the details you need for troubleshooting; configuring the right counters with this tool can point you in the right direction.
XenDesktop 7.x has a sizable number of built-in performance counters that can serve as a great tool for troubleshooting. Please refer to the following list of XenDesktop Perfmon counters that you can use on the XenDesktop environment for troubleshooting.
- Citrix Broker Service: Brokered Sessions
- Citrix Broker Service: Database Avg. Transaction Time
- Citrix Broker Service: Database Connected
- Citrix Broker Service: Database Transaction Errors/sec
- Citrix Broker Service: Database Transactions/sec
- Citrix Broker Service: Deregistration Requests
- Citrix Broker Service: Expired Launches/sec
- Citrix Broker Service: Expired Registrations/sec
- Citrix Broker Service: Ping Requests
- Citrix Broker Service: Hard Registrations/sec
- Citrix Broker Service: Registration Avg. Request Time
- Citrix Broker Service: Registration Rejects/sec
- Citrix Broker Service: Registration Requests/sec
- Citrix Broker Service: Soft Registrations/sec
- Citrix ADIdentity Service: Database Connected
- Citrix ADIdentity Service: Database Transaction Errors/sec
- Citrix Configuration Logging: Database Connected
- Citrix Configuration Logging: Database Transaction Errors/sec
- Citrix Configuration Service: Database Connected
- Citrix Configuration Service: Database Transaction Errors/sec
- Citrix Delegated Admin: Database Connected
- Citrix Delegated Admin: Database Transaction Errors/sec
- Citrix Environment Test: Database Connected
- Citrix Environment Test: Database Transaction Errors/sec
- Citrix Host Service: Database Connected
- Citrix Host Service: Database Transaction Errors/sec
- Citrix Machine Creation Service: Database Connected
- Citrix Machine Creation Service: Database Transaction Errors/sec
- Citrix Monitor: Database Connected
- Citrix Monitor: Database Transaction Errors/sec
- Citrix Storefront: Database Connected
- Citrix Storefront: Database Transaction Errors/sec
The counter details have been fetched directly from Perfmon available on the XenDesktop Controller server. To read more about XenApp/XenDesktop performance counters, please download the article at http://support.citrix.com/servlet/KbServlet/download/30998-102-686964/Operations%20Guide%20-%20Monitoring.pdf.
The document is outdated and won't provide you with much detail on recent releases of the XenDesktop/XenApp product line. However, the document is still considered as a good read for the information it contains on the performance monitors required to monitor Citrix infrastructure.
Configuring Procmon
Process Monitor is a free tool from Microsoft and can be easily downloaded from http://download.sysinternals.com/files/ProcessMonitor.zip.
This tool serves as a great asset in terms of troubleshooting application failures, file system issues, registry issues, boot logging, reading the application stack, and so on.
Once you have downloaded the tool, you will have three files named Procmon.exe, procmon.chm, and Eula.
Procmon.exe: This is the utility setup fileprocmon.chm: This is the software helper fileEula: This is the Microsoft End User License Agreement
Procmon comes as a portable utility that doesn't require installation. You can run Procmon.exe directly by accepting the Eula screen and it will start capturing, as shown in the following screenshot for reference:
We have now downloaded Procmon and it is ready to take our first capture. Before we actually start looking at the capture log, we should have a basic understanding of the Procmon toolbar. Please have a look at the following Procmon toolbar screenshot to understand the basic toolbar features that can help us in analyzing the Procmon logs:
To run this tool, you should be a part of the local administrator group on the local machine. Once the tool is launched, it immediately starts capturing three system-level activities: filesystem, registry, and process.
- Filesystem: Procmon captures the filesystem activity for all Windows filesystems whether it's a local storage or remote file share. The activity is monitored relative to the logged-on user session. You can deselect the filesystem activity by clicking on the filesystem activity button on the toolbar.
- Registry: Procmon captures all registry operations and the information is displayed using the Windows registry conventional path (HKLM:
HKEY_LOCAL_MACHINE). You can deselect the registry activity by clicking on the registry activity button on the toolbar. - Process: Procmon captures all process-related information starting from the process and thread monitoring subsystem to creation and exit operations including DLL and device driver load operations. You can deselect the process activity by clicking on the process activity button on the toolbar.
We have gained a basic understanding of the Procmon tool, toolbar features, and relevant data that the Procmon log provides.
To understand the importance of this tool with respect to troubleshooting of Citrix environment, we are going to do a case study that is focused on troubleshooting slow logons of Citrix published desktops based on XenApp.
Problem description:
Users have been complaining about slow logons while logging to Citrix desktops. The desktops logins were taking approximately 3-5 minutes to show up on the desktop interface of the users.
Environment:
XenApp 6.5, Citrix Profile Manager 3.2.2, and ICA Online Plug-in 12.1.44.1
Troubleshooting analysis:
On initial diagnosis, we found that the slowness was experienced by the users intermittently. Some users were able to login within 30-50 seconds and some users were experiencing a delay of 3-4 minutes.
We started with checking the event viewer logs for the machines that were experiencing delays, but couldn't find any relevant error that could lead us to the cause of delay. We enabled UPM logging and GPO logging to understand the delay but there were no errors or warnings that were causing this delay.
Note
To enable Citrix User Profile Manager (UPM), please visit the Citrix Knowledge Center article at http://support.citrix.com/article/CTX126723.
To enable GPO logging, please refer to the MS article at http://blogs.technet.com/b/csstwplatform/archive/2010/11/09/how-to-enable-gpo-logging-on-windows-7-2008-r2.aspx.
As the next troubleshooting step, before involving Citrix, we went ahead and used the Procmon tool to capture the entire logon process for impacted users. However, we had to run this tool multiple times to capture a valid trace for impacted users.
While analyzing the logon process, we found that the freeze happens among 2-3 process threads, which indicated that the thread was waiting for network restoration. It led us to find out whether any network drives were mapped for users that were no longer accessible or if it was waiting for any network printers to be mapped.
I checked and found that the issue is with the network printer's mapping while the user is logging on to the Citrix desktop.
Resolution:
We changed the published desktop application setting from the Citrix AppCenter console so that it doesn't wait for the network printers to be mapped while logging in, which resolved the issue.
After modifying this application setting, the login time reduced from 3-5 minutes to just 15-20 seconds.
Similarly, Procmon can be used while troubleshooting slow logons, application failures, file access permission issues, and more.