MQ Woes Part 999: Error encountered on Queue.Get Queue name = Reason code = 2354. hresult = 0XC0C1100C
At my current client we have a large BizTalk solution using IBM Websphere MQ to talk between systems. The development environment is a nightmare as we're not allowed actual servers or proper operating systems; we're running crappy business desktops with Windows XP - and they're subject to the same kinds of abuse (maintenance) that, for example, the project manager's laptop receives. On Friday, they pushed Windows XP SP2 onto the (shared) MQ machine. As you can imagine this killed it as the MQ adapter uses COM+ to do the network hop onto the remote MQ box and MSDTC to roll the transaction onto it and SP2 screws MSDTC.
After lots of fun with DTCPing, I re-established happy DTC communications between the two machines. This involved setting the RestrictRemoteClients registry key to 0, and re-enabling network MSDTC (info from Florin Lazar) - in the Component Services applet, right click My Computer, Properties, MSDTC, Security Configuration: check Network DTC Access, Allow Remote Clients, Allow Remote Administration, Allow Inbound, Allow Outbound, No authentication required + Enable XA Transactions.
This left me with the following errors:
On the MQ server: Error encountered as BizTalk attempts to retrive a message from MQSeries, description = Error encountered on Queue.Get Queue name = Reason code = 2354. hresult = 0XC0C1100C. This error will also be reported on the BizTalk server.
On the BizTalk Server: The adapter "MQSeries" raised an error message. Details "Exception from HRESULT: 0xC0C1100C."
A Google search on the above gives about 3 results all of which point to Florin's blog entry which I'd already been through. The error 2354 means "Enlistment in global unit of work failed" - following this up led me finally to this page on the IBM website - the Product Readme for V6.0.0.0 which I'm not using (using 5.3). This instructs me to enable XA transactions, but also has another error described:
"You have XA transactions enabled, but you still see a 2354 reason code when you try to get or put a message in a COM+ transaction or using a .NET application that inherits from ServicedComponent"
Quite why mqmtop is standing in for "where MQ is installed" I don't know. However my registry had the following in it "\bin\amqmtsxatm.dll". Changing this to "C:\Program Files\IBM\WebSphere MQ\bin\amqmtsxatm.dll" finally fixed that issue. In subsequent searching I actually found details of this fix in a KB article for BizTalk 2002 - good to know IBM haven't fixed this issue after all these years!
Anyway, this allowed me to receive messages from MQ into BizTalk but not send them. This was failing with the following error:
The adapter failed to transmit message going to send port "MQS://Server/QManager/QName". It will be retransmitted after the retry interval specified for this Send Port. Details:"Access is denied.".
In the process of trying to fix this I added these registry settings to the MQ box: The Enterprise Single Sign-On Service and associated BizTalk Server 2004 services fail after you install Windows XP Service Pack 2 (SP2), although this had no effect. I was also getting the following logon failures in the security log:
Where <MQAgent Identity> is the username of the local account the MQAgent COM+ application is running as. That's local to the remote MQ box! I'm not sure why this is required to do a send from BizTalk to MQ but it appears it is. There was already a local account with the same username and password as this remote (local) user - a hangover from some experiment long ago. Giving this user permissions to log on to the BizTalk machine finally solved the problem. I don't have access to view or edit group policy here, and a network admin ran the necessary stuff from the reskit for me:
net user <MQAgent Identity> /active:yes
ntrights +r SeNetworkLogonRight -u <MQAgent Identity>
ntrights -r SeDenyNetworkLogonRight -u <MQAgent Identity>
Anyway. Thank god that's over...
After lots of fun with DTCPing, I re-established happy DTC communications between the two machines. This involved setting the RestrictRemoteClients registry key to 0, and re-enabling network MSDTC (info from Florin Lazar) - in the Component Services applet, right click My Computer, Properties, MSDTC, Security Configuration: check Network DTC Access, Allow Remote Clients, Allow Remote Administration, Allow Inbound, Allow Outbound, No authentication required + Enable XA Transactions.
This left me with the following errors:
On the MQ server: Error encountered as BizTalk attempts to retrive a message from MQSeries, description = Error encountered on Queue.Get Queue name =
On the BizTalk Server: The adapter "MQSeries" raised an error message. Details "Exception from HRESULT: 0xC0C1100C."
A Google search on the above gives about 3 results all of which point to Florin's blog entry which I'd already been through. The error 2354 means "Enlistment in global unit of work failed" - following this up led me finally to this page on the IBM website - the Product Readme for V6.0.0.0 which I'm not using (using 5.3). This instructs me to enable XA transactions, but also has another error described:
"You have XA transactions enabled, but you still see a 2354 reason code when you try to get or put a message in a COM+ transaction or using a .NET application that inherits from ServicedComponent"
Solution: Add the WebSphere MQ XA resource manager DLL to the list of DLLs in the XADLL key of the registry:
HKLM\SOFTWARE\Microsoft\MSDTC\XADLL
The DLL name is amqmtsxatm.dll. It has a value type of REG_SZ. The value of the installed location of the DLL is <mqmtop>\bin\amqmtsxatm.dll.
Quite why mqmtop is standing in for "where MQ is installed" I don't know. However my registry had the following in it "\bin\amqmtsxatm.dll". Changing this to "C:\Program Files\IBM\WebSphere MQ\bin\amqmtsxatm.dll" finally fixed that issue. In subsequent searching I actually found details of this fix in a KB article for BizTalk 2002 - good to know IBM haven't fixed this issue after all these years!
Anyway, this allowed me to receive messages from MQ into BizTalk but not send them. This was failing with the following error:
The adapter failed to transmit message going to send port "MQS://Server/QManager/QName". It will be retransmitted after the retry interval specified for this Send Port. Details:"Access is denied.".
In the process of trying to fix this I added these registry settings to the MQ box: The Enterprise Single Sign-On Service and associated BizTalk Server 2004 services fail after you install Windows XP Service Pack 2 (SP2), although this had no effect. I was also getting the following logon failures in the security log:
Logon Failure:
Reason: The user has not been granted the requested
logon type at this machine
User Name: <MQAgent Identity>
Where <MQAgent Identity> is the username of the local account the MQAgent COM+ application is running as. That's local to the remote MQ box! I'm not sure why this is required to do a send from BizTalk to MQ but it appears it is. There was already a local account with the same username and password as this remote (local) user - a hangover from some experiment long ago. Giving this user permissions to log on to the BizTalk machine finally solved the problem. I don't have access to view or edit group policy here, and a network admin ran the necessary stuff from the reskit for me:
net user <MQAgent Identity> /active:yes
ntrights +r SeNetworkLogonRight -u <MQAgent Identity>
ntrights -r SeDenyNetworkLogonRight -u <MQAgent Identity>
Anyway. Thank god that's over...
posted by Ben Cops at
5:18 pm
2 Comments:
I have to say that NT security and MQ can be icky. I've found that once a particular username gets into the registry for the MQ service to run as, it can be hard to change it.
By
Anonymous, at 6:20 am
Ya, its really weird. Looks like we are exploiting a loop hole in windows authentication mechanism. May be when windows checks the user at adapter end, it verifies it against the credentials stored at MQ server end for that local user.
Anyways, finally got through it. Really helpful blog.
Suneet :)
By
Anonymous, at 3:37 pm
Post a Comment
<< Home