Kerberos and IIS hosted WCF : 'useAppPoolCredentials', 'Kernel mode authentication' and 'etc.'

Having invested quite a deal of time in troubleshooting a WCF authentication issue this week, I thought of sharing the experience, as this can perhaps save some man hours in the future. Though I worked my way through multiple issues in course of this exercie, but at a higher level, this post can come handy if you are investigating an authentication failure of the type below of a WCF service hosted in IIS, with windows authentication enabled:

System.ServiceModel.Security.MessageSecurityException: The HTTP request is unauthorized with client authentication scheme 'Negotiate'. The authentication header received from the server was 'Negotiate, NTLM'

Or

If you are trying to get your service being authenticated as a custom domain identity.

I also hope for this post to provide some directions to go around troubleshooting WCF authentication related issue in general as this is what I ended up doing in spirit.

Goal:

Aim of my exercise is to successfully use Integrated Windows Authentication (Kerberos and not NTLM) for the custom identity account under which the service is running. Why Kerberos and not NTLM?? In my case, I want to delegate client credentials from the middle tier WCF service to the backend Database and this is only possible with Kerberos (NTLM does not support delegation of credentials)

It’s worth mentioning that IIS 7 onwards, kernel mode authentication is enabled by default in IIS. Though Microsoft recommends this to remain enabled for reasons of performance and avoiding authentication related issues, there may be situations where someone really wants to authenticate against the custom identity and not the machine account and do it outside of Kernel mode (I was doing it to mimic IIS 6 days where kernel mode authentication was not an option). IIS 7 onwards, you can configure ‘Kernel-mode authentication’ under 'Advanced Settings' after enabling 'Windows Authentication' authentication type (Authentication feature of IIS, available at the server, site and application level)

Environment & Setup

I have a very simple WCF service configured with a single contract, basicHttpBinding and chose security mode 'TransportCredentialOnly' so as to avoid mixing it up with SSL and learning too much too soon. This is the relevant part of my service's web config :

  
   <system.serviceModel>
  
     <bindings>
       <basicHttpBinding>
         <binding name="binding1">
           <security mode="TransportCredentialOnly">
             <transport clientCredentialType="Windows"></transport>
           </security>
         </binding>
       </basicHttpBinding>
     </bindings>
     
     <services>
       <service name="MyService" behaviorConfiguration="Behavior">
         <endpoint name="endpoint1" binding="basicHttpBinding" bindingConfiguration="binding1" contract="MyContract">
           <identity>
             <servicePrincipalName value="MyService/MyMachine" />
           </identity>
         </endpoint>
       </service>
     </services>
  
 </system.serviceModel>

The service is hosted in IIS 7.5 (Windows 2008 R2) and running as identity 'EMEADS\SvcOwner'

If SPN in above config amazes you, with WCF client and service, you are free to venture out of SPN naming conventions (so forget about Service class, port etc.)

I find this MSDN magazine article quite useful to understand SPNs in general as well as wrt WCF : https://msdn.microsoft.com/en-us/magazine/cc163570.aspx

This is how I set the SPN for my service:

setspn -a MyService/MyMachine EMEADS\SvcOwner

Action time:

I disabled all other authentication schemes, keeping Windows Authentication only in IIS, right from the server level down through web site and up to my WCF application. I also disabled Kernel mode authentication at all levels under 'Advance Settings' for 'Windows Authentication'.

With SPN in place, I was all set to get pass the authentication when inevitable happened :

 Unhandled Exception: System.ServiceModel.Security.MessageSecurityException: The HTTP request is unauthorized with client authentication scheme 'Negotiate'. The authentication header received from the server was 'Negotiate oWwwaqADCgEBomMEYWBfBgkqhkiG9xIBAgIDAH5QME6gAwIBBaEDAgEepBEYDzIwMTIwNDIwMDkzODU4WqUFAgMOUo6mAwIBKakOGwxFTUVBRFMuTE9DQUyqEzARoAMCAQGhCjAIGwZua3NuMiQ='. ---> System.Net.WebException: The remote server returned an error: (401) Unauthorized. ---> System.ComponentModel.Win32Exception: The target principal name is incorrect

Troubleshooting:

Win32Exception inner exception message often helps to indicate what is wrong here. "The target principal name is incorrect" suggest an issue with service Identity (SPN in this case). (For a quick experiment, I disabled the Active Directory account under which my client is running and though WCF MessageSecurityException remained same but inner Win32Exception message changed to "The logon attempt failed". Self-explanatory)

With the 'AllowNtlm’ property of WindowsClientCredentials set to ‘false’ (note: This property is now marked as obsolete-local machine policy should be used instead: https://msdn.microsoft.com/en-us/library/system.servicemodel.security.windowsclientcredential.allowntlm.aspx ) and 'Negotiate' provider enabled on WCF app, I knew I am troubleshooting Kerberos already. The first troubleshooting step was to enable Kerberos logging on my client box: https://support.microsoft.com/kb/262177

Reproducing the issue again, the following event was logged in the System event log:

-Log Name: System

Source: Microsoft-Windows-Security-Kerberos

Date: 20/04/2012 12:03:31

Event ID: 3

Task Category: None

Level: Error

Keywords: Classic

User: N/A

Computer: NKSN1.EMEADS.local

Description:

A Kerberos Error Message was received:

on logon session

Client Time:

Server Time: 11:3:31.0000 4/20/2012 Z

Error Code: 0x29 KRB_AP_ERR_MODIFIED

Extended Error:

Client Realm:

Client Name:

Server Realm: EMEADS.LOCAL

Server Name: nksn2$

Target Name:

Error Text:

File: 3

Line: 576

Error Data is in record data.

At this time, I noticed, even when my Kerberos logging was not enabled, there were these events in event log from earlier attempts of troubleshooting:

Kerberos event 4 : The Kerberos client received a KRB_AP_ERR_MODIFIED error from the server nksn2$. The target name used was HTTP/nksn2.emeads.local. This indicates that the target server failed to decrypt the ticket provided by the client. This can occur when the target server principal name (SPN) is registered on an account other than the account the target service is using. Please ensure that the target SPN is registered on, and only registered on, the account used by the server. This error can also happen when the target service is using a different password for the target service account than what the Kerberos Key Distribution Center (KDC) has for the target service account. Please ensure that the service on the server and the KDC are both updated to use the current password. If the server name is not fully qualified, and the target domain (EMEADS.LOCAL) is different from the client domain (EMEADS.LOCAL), check if there are identically named server accounts in these two domains, or use the fully-qualified name to identify the server.

That was quite a wealth of info to give me directions.

setspn –X’ is the easiest way to trace duplicate SPNs ('duplicate SPN' is when the same SPN gets assigned to more than one accounts. 'Duplicate SPNs' is just one possible SPN related issue. Go through link in 'References' section below for complete SPN checklist for Kerberos). For some reason, I used an old school way to find which account(s) is (are) using this SPN: https://technet.microsoft.com/en-us/library/cc772897(v=WS.10).aspx

And I got this output from the LDP tool:

***Searching...

ldap_search_s(ld, "DC=EMEADS,DC=local", 2, "serviceprincipalname=MyService/MyMachine", attrList, 0, &msg)

Getting 2 entries:

Dn: CN=SvcOwner,CN=Users,DC=EMEADS,DC=local

canonicalName: EMEADS.local/Users/svcowner;

name: SvcOwner;

objectClass (4): top; person; organizationalPerson; user;

Dn: CN=iismanaged,CN=Managed Service Accounts,DC=EMEADS,DC=local

canonicalName: EMEADS.local/Managed Service Accounts/iismanaged;

name: iismanaged;

objectClass (4): top; person; organizationalPerson; user;

This confirms that duplicate SPNs are getting in my path. To start from a clean slate, I to deleted both the SPNs and created a new one, only for the custom identity the WCF service is running as (you need to be a domain administrator to run this):

Delete both SPNs:

setspn -d MyService/MyMachine emeads\iismanaged

setspn -d MyService/MyMachine emeads\svcowner

Create the SPN as I want it to be (for domain idenity under which the service is running)

setspn -a MyService/MyMachine emeads\svcowner

I thought I am close. Another run and again :

Unhandled Exception: System.ServiceModel.Security.MessageSecurityException: The HTTP request is unauthorized with client authentication scheme 'Negotiate'. The authentication header received from the server was 'Negotiate oWwwaqADCgEBomMEYW BfBgkqhkiG9xIBAgIDAH5QME6gAwIBBaEDAgEepBEYDzIwMTIwNDIwMDkzODU4WqUFAgMOUo6mAwIBKakOGwxFTUVBRFMuTE9DQUyqEzARoAMCAQGhCjAIGwZua3NuMiQ='. ---> System.Net.WebException: The remote server returned an error: (401) Unauthorized. ---> System.ComponentModel.Win32Exception: The target principal name is incorrect

Well, remember your client system caches SPNs for target identity, hence always run 'klist purge' after any SPN changes. Unfortunately, in my case, even 'klist purge' is of no consequence and I still run into the same exception. Perhaps because I was playing around with 'useAppPoolCredentials' setting in applicationHost.config: https://technet.microsoft.com/en-us/library/dd759186.aspx . I reverted everything to default on IIS (useAppPoolCredentials = false) and did an iisreset. The WCF client successfully negotiated credentials with service now and all is well.

Now is the time to demystify links between 'kernel mode authentication’, 'useAppPoolCredentials' and 'service SPN identity'.

After investing another great deal with various experiments and readings, these are the scenarios and respective implications:

Scenario 1. useAppPoolCredentials=false

1a. Kernel mode disabled. Client explicitly specifies service SPN Identity (Service identity can be of various types like Certificate, UPN etc) :

 <client>
   <endpoint....>
      <identity>
          <servicePrincipalName value="MyService/MyMachine">
      </identity>
   </endpoint>
 </client>

This succeeds with custom Identity credentials. But as already warned, this is non-performant as Kernel is not handling authentication

1b. Kernel mode enabled. Client explicitly specifies SPN Identity as above. As expected, authentication fails since the server is expecting host/http SPN.

1c. Kernel mode enabled. Client does not specify service SPN Identity. Authentication succeeds but with default host/http SPN. This is what we always recommend. Performant and (almost) no authetication issues.

Scenario 2. useAppPoolCredentials=true

2a. Kernel mode disabled. (With kernel mode disabled, useAppPoolCredentials is of no consequence as authentication is now onus of application host) Client explicitly specifies service SPN Identity. This succeeds with custom Identity credentials but non performant like 1a.

2b. Kernel mode enabled. Client explicitly specifies service SPN Identity. Authentication succeeds. Kernel authenticating against appPool identity. More performant and recommended for custom identity scenarios.

2c. Kernel mode enabled. Client does not specifies service SPN Identity. Authentication FAILS as client provided ticket for host/http SPN whereas Kernel mode authentication at server is configured to authenticate against appPool credentials.

Putting all of above in nice tabular form:

The learning is that kernel mode authentication in IIS doesn't necessarily mean machine account. With 'useAppPoolCredentials=true', we are telling IIS to do kernel mode authentication with AppPool identity account (and not machine account). To be 100% sure, I verified network traffic using Netmon 3.4 and indeed kerberos ticket was issued for 'MyService/MyMachine' SPN and used successfully for service authentication with useAppPoolCredentials=true.

Some other tips:

- An easy way to rule out any SPN related issues is by running the service as ‘Network Services’ account where default host SPNs are used. That means, remove any ‘Identity’ related configs from client and service and test again.

- When Kernel-mode authentication is used and useAppPoolCredentials=’false’ (default), server ticket will be issued and decrypted with machine account . In this case, server Identity will be inferred from service URI and not required to be explicitly mentioned in client or service config. Refer : https://blogs.msdn.com/b/webtopics/archive/2009/01/19/service-principal-name-spn-checklist-for-kerberos-authentication-with-iis-7-0.aspx 

-Remember to 'klist purge' after any SPN related change

-if duplicate SPNs are such a hassle, why are they even permitted. I hope to get an answer to this soon and update the post.

-For simple scenarios, your Authentication types should be coherent right from server to application level in IIS unless there are other considerations

-Remember to do iisreset after any change in applicationhost.config

How to force Kerberos with WCF:

Binding should specify clientCredentialType="Windows"

If your service is hosted in IIS 7.5, there is a new authentication provider 'Negotiate:Kerberos', which doesn't fall back to NTLM

How to force NTLM with WCF:

Binding should specify clientCredentialType="Ntlm"

WCF service host (IIS in most cases) should have NTLM as one of authentication provider in the list.

References:

https://blogs.msdn.com/b/asiatech/archive/2011/10/26/iis-7-kerberos-authentication-failure-krb-ap-err-modified.aspx

https://blogs.msdn.com/b/friis/archive/2009/12/31/things-to-check-when-kerberos-authentication-fails-using-iis-ie.aspx?wa=wsignin1.0

Comments

• Anonymous
  December 07, 2012
  Thanks for this valuable information..