I came across an issue with a clients Exchange Server deployment today that is not well documented – or rather it is, but you need to know where to look. So I thought I would document the troubleshooting steps and the fix here.
We specifically came across this error when testing Free/Busy for an Office 365 migration, though it could happen for a variety of reasons. Free/Busy and other lookups in a cross-forest Exchange Server deployment require a working organization configuration and this was failing. Running Test-FederationTrust (a prerequisite of the organization relationship) in verbose mode (add -Verbose to the end) returned the following:
Unable to retrieve federation metadata from the security token
service. Reason: Microsoft.Exchange.Management.FederationProvisioning.FederationMetadataException: Unable to access the
Federation Metadata document from the federation partner. Detailed information: “The underlying connection was closed:
Could not establish trust relationship for the SSL/TLS secure channel.”.
The final result of the test will also show two errors for “Unable to retrieve federation metadata from the security token service.” and “Failed to request delegation token.”
The last part of the verbose error is the clue here. The server in question is unable to make an SSL/TLS connection to the endpoint that the federation trust needs to reach to get the federation trust metadata. That endpoint is listed right at the start of the Verbose output. It reads:
VERBOSE: [16:53:08.306 GMT] Test-FederationTrust : Requesting Federation Metadata from
Now that we have a URL and an error message, check that the URL is reachable from each of your Exchange Servers. At my client today we found one server could not successfully reach this endpoint without an SSL error turning up in the browser. The problem was that the certificate that the endpoint is secure with is issued by the Baltimore Cybertrust Root Certificate – one that Microsoft uses for lots of services, but the root certificate was not installed on that machine. Lots of root certs where missing from that machine as it had never had a root certificate update applied to it.
We installed the latest Root Certificate Update and then the federation trust worked and free/busy etc. (mail tips, cross-forest message tracking etc.) all worked fine.
Update: 10 Dec 2014: It is reported that this issue is fixed in CU7 for Exchange Server 2013
OAuth authentication is a new server to server authentication model available in Exchange 2013 SP1 and later and Exchange Online (Office 365). With OAuth enabled and Exchange hybrid in place and where you have multiple endpoints of Exchange Server on-premises and those on-premises Exchange Servers are different versions then you might have issues getting Exchange Online to On-Premises free/busy lookups to work.
Here is the scenario:
Company with Exchange 2010 servers in multiple internet connected sites, going hybrid to Exchange Online.
Exchange Online tenant created and hybrid mode put in place between Exchange Online and Exchange Server 2013 on-premises. In the site where the Exchange 2013 hybrid servers are located there are Exchange 2010 SP3 servers. As hybrid mode was set up with SP1, OAuth was enabled.
Exchange 2010 in the remote sites is configured with an ExternalURL for EWS. Therefore a free/busy lookup from an Office 365 user to a mailbox in one of these remote sites goes direct to the EWS endpoint on Exchange 2010 – it is not proxied via the 2013 hybrid server.
With OAuth enabled this configuration will fail as Exchange Online will use OAuth to authenticate to Exchange 2010 on-premises and fail. The IIS logs will contain entries such as this:
2014-07-22 19:39:34 10.100.28.73 POST /ews/exchange.asmx – 443 – 10.100.28.220 ASProxy/CrossForest/EmailDomain//15.00.0985.008 401 0 0 0
Where the 401 indicates authentication failed and the path ASProxy/CrossForest/EmailDomain indicating OAuth in use. There will be no entries in the IIS log for the Federation Org type of authentication.
If the EWS connection for free/busy goes via the 2013 hybrid server (ExternalURL for the remote site is null) then the free/busy lookup works, or if the OAuth connector in Exchange Online is disabled (Get-IntraOrganizationConnector | Set-IntraOrganizationConnector -Enabled $false from Exchange Online remote PowerShell session) and EWS lookup for free/busy goes direct to the remote Exchange 2010 server then free/busy lookups work.
So if you want OAuth and direct EWS connections to remote sites for free/busy you need Exchange 2013 at those remote sites. If you want to have Exchange 2013 hybrid servers only at your primary site (for mail flow) and OAuth as well (for eDiscovery cross-forest) then you need to proxy your EWS free/busy requests via the Exchange 2013 hybrid server.
This is a known issue in Exchange and may be fixed in the future.