Those in the Exchange Server space for a number of years know all about AutoDiscover and the many ways it can be configured and misconfigured – if even configured at all. Often misconfiguration is to do with certificates or it is not configured at all because it involves certificates and I thought I was aware of all the problems I could see with this and then I came across one more!
Its all to do with the AutoDetect Service that runs in the Microsoft Cloud. AutoDetect is like an “AutoDiscover of AutoDiscovers”. The Exchange Server/Online AutoDiscover service finds your mailbox only if it is on the Exchange platform and the Outlook desktop product has a feature called GuessSmart to detect your email settings if you are not using Exchange (i.e. Gmail, Yahoo etc.). Outlook Mobile, and now Outlook desktop for the last few years, uses a cloud service called AutoDetect. Originally this was only used by Outlook Mobile when it was a seperate product called Acompli but it now used instead of GuessSmart in the mobile and desktop versions of Outlook.
Here we will look at a problem in AutoDiscover and Exchange Server configuration that breaks AutoDetect and as a side effect breaks Hybrid Modern Auth (HMA) in Outlook Mobile as well. AutoDetect is called when you enter your email address in Outlook Mobile and click Add Account.
The AutoDetect service is called with the following web service call and sending a header called x-email that matches your email address
This can be tested from PowerShell using:
Invoke-WebRequest -Uri 'https://prod-autodetect.outlookmobile.com/detect?services=office365,outlook,google,icloud,yahoo&protocols=rest-cloud,rest-outlook,rest-office365,eas,imap,smtp' -Headers @{'x-email'="your@email.com"} | ConvertFrom-Json
When this call is made to AutoDetect, the AutoDetect service attempts to work out what email service you are using and this is run from a Kubernetes container app. For example if the domain MX points to Gmail, it will start a Gmail discovery process. In this case its Exchange Server running Hybrid Modern Auth, so AutoDiscover Version 2 is called from the container. Microsoft publish the IP addresses of this AutoDetect service and the AutoDiscover and ActiveSync service endpoints need to be open to these IP addresses.
This should reliably return mailbox info to the app and then to Outlook Mobile so it can be configured. In an example with a client of mine, this was failing and so though we have Hybrid Modern Auth working for Outlook, we did not have it working for Outlook Mobile.
In Outlook Mobile we where seeing a Select your email provider page as shown:
This shows that AutoDetect is not working, because we knew that AutoDiscover was working (and tests at https://exrca.com showed that it was) but we would expect to see Microsoft Authenticator and a sign-in prompt at this point.
So calling the above PowerShell Invoke-WebRequest I saw that about 1 in 20 attempts would successfully return the expected AutoDiscover information. Mostly I was seeing this:
The response for “services” and “protocols” was blank when one or both of these values should be populated. “Services” should return data indicating the location of your mailbox (for example, in Exchange Online or if accessed via Hybrid Modern Auth to Exchange Server or Outlook or Google or Yahoo or other) and “Protocols” should contain data if if your mailbox is in Exchange Online or Basic Auth and located on-premises. Here I see neither, but occasionally I would see a working response:
This needs to be 100% successful, but we needed to find out the issue so we could resolve the problem. To get the error message back I had a number of options. First I used the Advanced REST client Chrome add-in – you enter the URL and headers as above, and all the responses are shown, or I could use PowerShell and look for the Headers rather than the JSON response as above. So I ran the following:
(Invoke-WebRequest -Uri 'https://prod-autodetect.outlookmobile.com/detect?services=office365,outlook,google,icloud,yahoo&protocols=rest-cloud,rest-outlook,rest-office365,eas,imap,smtp' -Headers @{'x-email'="your@email.com"} -UseBasicParsing).Headers
This returned the following:
Key Value
--- -----
X-Request-Id prod5-1-5a58808b-15c1-4ba4-95e3-1feb61be5c2c
X-Instance-Id prod5-autodetect-1
X-DNS-Prefetch-Control off
X-Frame-Options SAMEORIGIN
Strict-Transport-Security max-age=15552000; includeSubDomains
X-Download-Options noopen
X-Content-Type-Options nosniff
X-XSS-Protection 1; mode=block
X-OLM-Source-Endpoint https://prod5-api.acompli.net/autodetect/detect
X-Provider-Id Unknown
X-Debug-Support eyJkZWNpc2lvbiI6ImF1ahchzsfasfwlklsfiudfzrseceZWQgZGIgcHJvdmlkZXIgPiBmaXhlZCBkYiBkb21haW...
X-AutoDv2-Error UNABLE_TO_VERIFY_LEAF_SIGNATURE
X-Feedback-Token eyJhbGciOifsfsersfrvrtgfxkpXVCJ9.eyJkIjoiZwfefwefzsdefdasesdffiLCJwSSI6bnVsbCwicyI6W10sI...
Vary Accept-Encoding
X-Response-Time 822ms
Content-Length 64
Content-Type application/json; charset=utf-8
Date Sat, 15 May 2021 16:46:21 GMT
Specifically of interest is the X-AutoDv2-Error key which reads “UNABLE_TO_VERIFY_LEAF_SIGNATURE”. This is the source of my issues with my clients Exchange Server deployment. Specifically the Kubernetes container app that is AutoDetect does not have its own pool of certificates to compare the Exchange Servers certificate too – specifically the Intermediate certificate and possibly the Root certificates are not available to this service. It requires these certificates to be provided by Exchange Server or by any SSL inspection device inline to the server (such as a load balancer).
Note that the X-Debug-Support header contains some Base 64 encoded text. Copy that to a Base64 decoder (online) and you will get more information about your connection as well.
You can check your certificates and see if the intermediate certificate is being offered by your Exchange Server by visiting https://www.sslshopper.com/ssl-checker.html and entering your autodiscover hostname, as that is where AutoDetect is going. I was seeing this result for my clients URL:
The Intermediate certificate was missing in two places – first it was not loaded into the Intermediate certificate store in Windows Server Certificates and also missing on the load balancer.
Once the correct certificate was published and intermediate added to the load balancer the above check succeeded, the PowerShell Invoke-WebRequest returned AutoDetect returned “services” data 100% of the time (and this includes the ActiveSync URL for the user as the “onprem” value within the services array). At this time we also checked if the on-premises ActiveSync URL that was returned has the same issue with broken certificates and resolved that as well. So it is not just autodiscover.domain.com that you need to check, but maybe mail.domain.com or whatever your ActiveSync URL is.
Once this was working, back in the Outlook Mobile client when adding a new mailbox we say that it would go straight to Microsoft Authenticator (as Hybrid Modern Auth was in use here), but if you are doing Basic Auth to on-premises, AutoDiscover will run and the mailbox added automatically as well. With AutoDetect working the prompt for “Select your email service provider” shown above never appeared.
And once the mailbox was added to Outlook Mobile, the Exchange Online mailbox that caches the on-premises mailbox (as this is how Hybrid Modern Auth works with Outlook Mobile) was also successfully provisioned. On the 1 in 20 successes I was having this was probably due to one load balancer was working and all the rest were not. If no load balancer/server was publishing the intermediate you would get 100% fails and if one of two load balancers/servers was publishing the intermediate I would expect 50% success!
As an aside issue, I was also finding that mailbox provisioning was not working – Outlook Mobile would ask Exchange Online to get the email from Exchange Server, which requires Exchange Online to create a “shard mailbox” or ghost mailbox for the user and populate it with 4 weeks of data. If your intermediate/root certs are not published correctly, Exchange Online cannot succeed here as well and this is either due to the certificates for AutoDiscover V2 being unpublished or ActiveSync.
This overall took about a week to work out, with thanks to Ross and Rob at Microsoft for guidance as well.
Photo by Nubia Navarro (nubikini) from Pexels
Leave a Reply