Problem
Desktops fail to start with a 1030 error.
Background
In this scenario, when the user clicks on their desktop group, the desktop viewer starts, but then disconnects giving an error 1030:
“The connection to ‘mydesktop’ failed with status 1030”
This issue was not fixed by reinstalling the Virtual Desktop Agent (VDA) software and was not seen with a fresh vanilla OS Image, however the customer required that their organization use the image approved by their security department.
Environment
- Citrix Product: XenDesktop
- Hosting Infrastructure: VMware 5
- Citrix Servers: XenDesktop 5.5,VDA 5.5
- Server OS: 2008r2 64-bit
- Target Device OS: Windows 7
- SQL server: 2008
- Client version: 13.1.0.89
Troubleshooting Methodology
- Collect good traces and bad traces:
- Configuration setting of the Citrix agent
- Third party
- Applied GPO policies.
- Attempt to match the images as closely as possible by removing irrelevant software that might add extra log events or affect the deployment
- Compare good traces to the bad traces and compare the configuration as well
Citrix Technical Support Engineers noticed the following message in Portica good traces: “SendMessageToGina” This message was followed by a message for communication to a named pipe CTX_PICA_SVC_TO_GINA_2.
In the bad traces, this sequence was very difference. The engineers started with the same initial message “SendMessageToGina”, but this was followed by an error that “The GINA notification event does not exist”. This error was followed by an alert “Failed to open PicaGina event” and that the function would wait and try again. This loop continued a number of iteration until the engineers noticed a message for the function surrender “Failed to open PicaGina event. Give up.”
Following is a sample of the Good Traces:
Entry5: Citrix.Portica.GinaServer.SendMessageToGina Entry5: Citrix.Utils.Kernel32.NamedPipeClient.Open Name:CTX_PICA_SVC_TO_GINA_2 Entry9: Citrix.Utils.Kernel32.NamedPipe.ctor Exit9: Citrix.Utils.Kernel32.NamedPipe.ctor Exit5: Citrix.Utils.Kernel32.NamedPipeClient.Open Handle set=True Entry9: Citrix.Utils.Kernel32.NamedPipe.Write Entry9: Citrix.Utils.Kernel32.SafeOverlapped.Wait Exit9: Citrix.Utils.Kernel32.SafeOverlapped.Wait Entry9: Citrix.Utils.Kernel32.SafeOverlapped.GetResult Exit9: Citrix.Utils.Kernel32.NamedPipe.Write Return:True
Following is a sample of the Bad Traces:
Entry5: Citrix.Portica.GinaServer.SendMessageToGina Trace: Citrix.Portica.GinaServer.SendMessageToGina The GINA notification event does not exist. Trace5: Citrix.Portica.GinaServer.SendMessageToGina Failed to open PicaGina event. Wait and try again. Trace: Citrix.Portica.GinaServer.SendMessageToGina The GINA notification event does not exist. Trace5: Citrix.Portica.GinaServer.SendMessageToGina Failed to open PicaGina event. Wait and try again. . <Repeated Messages removed > . Trace5: Citrix.Portica.GinaServer.SendMessageToGina Failed to open PicaGina event. Wait and try again. Trace: Citrix.Portica.GinaServer.SendMessageToGina The GINA notification event does not exist. Trace5: Citrix.Portica.GinaServer.SendMessageToGina Failed to open PicaGina event. Give up.
After identifying this, the engineers reviewed the good image with pipelist from sysinternals to list the named pipes and compared this to the bad image. In the Restricted image, the following pipes are not being loaded:
- CTX_PICA_SVC_TO_GINA_1
- CTX_PICA_CP_TO_SESSION_1
The engineers had previously collected traces CDF and ProcMon traces from the bad VDA and requested that the customer provide a set of parallel traces from the good vanilla image to compare against locked down image.
Good Procmon (tree view):
For the good traces, the engineers noticed that the OS loaded the session manager through the Winlogon Process and then this passes to the LogonUI Process that manages the password handover:
Bad Trace View (tree view):
In bad traces, the winlogon process transitions to LogonUI process without ever calling our process resulting in the missing Citrix process and pipe not being loading.
With this analysis, the engineers understood the failure as the data pointed to your Citrix Session Manager (PicaSessionMgr.exe) not being loaded. This process was responsible for loading the pipes that the Gina was waiting on in the logs. The engineers then reviewed the code and a working system to determine how this process is loaded by the agent.
The Citrix Session Manager gets loaded through one of the core dll’s mfaphook.dll which does API hooking. When they filtered on the loading of this process, they found that the path was malformed and library failed to load. The OS was splitting the path ‘C:Program Files (x86)CitrixSystem32MFAPHOOK64.DLL’ into three entities at the following spaces:
- C:Program Files (x86)CitrixSystem32MFAPHOOK64.DLL
On the Good System, the path used was the shortname for the loaction that did not contain any spaces:
C:PROGRA~1CitrixSystem32MFAPHOOK64.DLL
The engineers attempted to access this path on the bad system and could not access the path. Further investigation revealed that the short name for the path was missing. C:>dir program*. /x Volume in drive C is DDC1
Directory of C:
06/05/2012 10:41 AM <DIR> PROGRA~1 Program Files 06/05/2012 04:49 PM <DIR> PROGRA~2 Program Files (x86)
Resolution
The data showed that the root cause of issue was the OS not loading the mfaphook64.dll, thereby disabling API hooking in the Image.
The folder location was missing its short-name label and the agent installer added the long-name which included spaces.
The resolution was to remove the full path from the key Key Location: HKEY_LOCAL_MACHINESOFTWAREMicrosoftWindows NTCurrentVersionWindows Entry Name: AppInit_DLLs Entry type: String Entry Value: mfaphook64.dll
Key Location: HKEY_LOCAL_MACHINESOFTWAREWow6432NodeMicrosoftWindows NTCurrentVersionWinlogon Entry Name: AppInit_DLLs Entry type: String Entry Value: mfaphook.dll
Read the entire article on the knowledgebase here.