Capax Zantas EAS EWS 501 Errors

Posted on Leave a commentPosted in EAS, Exchange Server, Kemp, Load Master, Zantas

Whilst load balancing Exchange 2013 with a Capax Zantas EAS deployment in place I was reminded of an issue I experienced a few years ago with a Kemp load balancer. The EAS app for OWA was failing with various errors such as:

image

Communication with service failed. The service might be down or misconfigured. Error code: 500

As well as the search functionality never returning results.

And in the EASOWABackend log file we were seeing the following:

w3wp.exe Error: 0 : 22 : [09/23/2016 10:13:22.880 AM] User username@domain.com: EWS Call failed Microsoft.Exchange.WebServices.Data.ServiceRequestException: The request failed. The remote server returned an error: (501) Not Implemented. —> System.Net.WebException: The remote server returned an error: (501) Not Implemented.
   at System.Net.HttpWebRequest.GetResponse()
   at Microsoft.Exchange.WebServices.Data.EwsHttpWebRequest.Microsoft.Exchange.WebServices.Data.IEwsHttpWebRequest.GetResponse()
   at Microsoft.Exchange.WebServices.Data.ServiceRequestBase.GetEwsHttpWebResponse(IEwsHttpWebRequest request)
   — End of inner exception stack trace —
   at Microsoft.Exchange.WebServices.Data.ServiceRequestBase.GetEwsHttpWebResponse(IEwsHttpWebRequest request)
   at Microsoft.Exchange.WebServices.Data.ServiceRequestBase.ValidateAndEmitRequest(IEwsHttpWebRequest& request)
   at Microsoft.Exchange.WebServices.Data.MultiResponseServiceRequest`1.Execute()
   at Microsoft.Exchange.WebServices.Data.ExchangeService.InternalConvertIds(IEnumerable`1 ids, IdFormat destinationFormat, ServiceErrorHandling errorHandling)
   at Microsoft.Exchange.WebServices.Data.ExchangeService.ConvertId(AlternateIdBase id, IdFormat destinationFormat)
   at EASOWA2013.EASOWABackend.OpenMessage(String owaID, String userEmail, String authToken, String pageURL)

This error contained a 501 Not Implemented message, and this message was getting exposed in OWA. But this error was not present in any of the Exchange Server logs, so it was not being caused by Exchange. This led us to looking at the load balancer, and testing EAS when bypassing the load balancer. To bypass the load balancer requires host file changes and changing the EAS configuration, so it is not a trivial change. When the load balancer was out of the traffic path everything worked fine.

Under the Kemp system logging (Logging Options > System Log Files > System Message File) we were seeing:

Sep 22 16:39:09 cwh-kemp01 kernel: L7: badrequest-client_read [192.168.x.y:44987->192.168.lb.vip:443] (-501): 192.168.lb.vip:443] (-501): 192.168.lb.vip:443] (-501): 192.168.lb.vip:443] (-501): 192.168.lb.vip:443] (-501): 192.168.lb.vip:443] (-501): 192.168.lb.vip:443] (-501): 192.168.lb.vip:443] (-501): 192.168.lb.vip:443] (-501): 192.168.lb.vip:443] (-501): 192.168.lb.vip:443] (-501):

Where 192.168.x.y was the client IP and 192.168.lb.vip was the load balanced IP for Exchange.

With reference to the above linked article, the fix was to change the Kemp’s handling of the 100-Continue feature and by changing the setting to RFC-7231 Compliant EAS in OWA started working:

image

SSL and Exchange Server

Posted on 2 CommentsPosted in 2008 R2, 2012 R2, 2013, certificates, exchange, https, IAmMEC, JetNexus, load balancer, Load Master, loadbalancer, mobile phones, SSL, TLS, windows server, xp

In October 2014 or thereabouts it became known that the SSL protocol (specifically SSL v3) was broken and decryption of the encrypted data was possible. This blog post sets out the steps to protect your Exchange Server organization regardless of whether you have one server or many, or whether or not you use a load balancer or not. As load balancers can terminate the SSL session and recreate it, it might be that changes are needed on your load balancer or maybe directly on the servers that run the CAS role. This blog post will cover both options and looks at the settings for a Kemp load balancer and a JetNexus load balancer.

Of course being an Exchange Server MVP, I tend to blog about Exchange related stuff, but actually this is valid for any server that you publish to the internet and probably valid of any internal server that you encrypt traffic to via the SSL suite of protocols. Microsoft outline the below configuration at https://technet.microsoft.com/en-us/library/security/3009008.aspx.

The steps in this blog will look at turning off the SSL protocol in Windows Server and turning on the TLS protocol (which does the same thing as SSL and is interchangeable for SSL, but more secure at the time of writing – Jan 2015). Some clients do not support TLS (such as Internet Explorer on Windows XP Service Pack 2 or earlier, so securing your servers as you need to do may stop some home users connecting to your Exchange Servers, but as XP SP2 should not be in use in any business now, these changes should not affect desktops. You could always use a different browser on XP as that might mitigate this issue, but using XP is a security risk in an of itself anyway! To disable clients from connecting to SSL v3 sites requires a client or GPO setting and this can be found via your favourite search engine.

Note that the registry settings and updates for the load balancers in this blog post will restrict client access to your servers if your client cannot negotiate a mutual cipher and secure channel protocol. Therefore care and testing are strongly advised.

Testing and checking your changes

Before you make any changes to your servers, especially internet facing ones, check and document what you have in place at the moment using https://www.ssllabs.com/ssltest. This service will connect to an SSL/TLS protected web site and report back on the issues found. Before running any of the changes below see what overall rating you get and document the following:

  • Authentication section: record the signature algorithm. For the signature algorithm its possible the certificate authority signature will be marked “SHA1withRSA WEAK SIGNATURE”. This certificate, if rekeyed and issued again by your certificate authority might be replaced with a SHA-2 certificate. The Google Chrome browser from September 2014 will report sites secured with this SHA-1 certificate as not fully trustworthy based on the expiry date of the certificate. If your certificate expires after Jan 1st 2017 then get it rekeyed as soon as possible. As 2015 goes on, this date will move closer in time. From early 2015 this cut off date becomes June 1st 2016 and so on. Details on the dates for this impact are in http://googleonlinesecurity.blogspot.co.uk/2014/09/gradually-sunsetting-sha-1.html. You can also use https://shaaaaaaaaaaaaa.com/ to test your certificate if the site is public facing, and this website gives details on who is now issuing SHA-2 keyed certificates. You can examine your external servers for SHA-1 certificates and the impact in Chrome (and later IE and Firefox) at https://www.digicert.com/sha1-sunset/. To do the same internally, use the DigiCert Certificate Inspector at https://www.digicert.com/cert-inspector.htm.
  • Authentication section: record the path values. Ensure that each certificate is either in the trust store or sent by the server and not an extra download.
  • Configuration section: document the cipher suites that are provided by your server
  • Handshake simulation section: Here it will list browsers and other devices (mobile phones) and what their default cipher is. If you do not support the cipher they support then you cannot communicate. Note that you typically support more than one cipher and the client will often support more than one cipher to, so though it is shown here as a mismatch this does not mean that it will not work and if this client is used by your users then click the link for the client and ensure that the server offers at least one of the the ciphers required by the client – unless all the ciphers are insecure in which case do not use that client!

Once you have a document on your current configuration, and a list of the clients you need to support and the ciphers they need you to support, you can go about removing SSL v3 and insecure ciphers.

Disabling SSL v3 on the server

To disable SSL v3 on a Windows Server (2008 or later) you need to set the Enabled registry value at “HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\Schannel\Protocols\SSL 3.0\Server” to 0. If this value does not exist, the create a DWORD value called “Enabled” and leave it at 0. You then need to reboot the server.

If you are using Windows 2008 R2 or earlier you should enable TLS v1.1 and v1.2 at the same time. Those versions of Windows Server support TLS v1.1 and v1.2 but it is not enabled (only TLS v1.0 is enabled). To enable TLS v1.1 and v1.2 use set the Enabled value at “HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\Schannel\Protocols\TLS 1.1\Server” to 1. Change the path to “HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\Schannel\Protocols\TLS 1.2\Server” and the same setting to support TLS v1.2. If these keys do not exist, create them. It is also documented that the “DisabledByDefault” key is required, but I have seen this noted as being the same as the “Enabled” key – just the opposite value. Therefore as I have not actually checked, I set both Enabled to 1 and DisabledByDefault to 0.

To do both the disabling of SSL v2 and v3 (v2 can be enabled on older versions of Windows and should be disabled as well) I place the following in a .reg file and double click it on each server, followed by a reboot for it to take effect. This .reg file contents also disables the RC4 ciphers. These ciphers have been considered insecure for a few years and when I configure my servers not to support SSL v3 I also disable the RC4 ciphers as well.

Windows Registry Editor Version 5.00

 

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\Schannel\Protocols\SSL 2.0]

 

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\Schannel\Protocols\SSL 2.0\Client]

"Enabled"=dword:00000000

 

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\Schannel\Protocols\SSL 2.0\Server]

"Enabled"=dword:00000000

 

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\Schannel\Protocols\SSL 3.0]

 

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\Schannel\Protocols\SSL 3.0\Client]

"Enabled"=dword:00000000

 

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\Schannel\Protocols\SSL 3.0\Server]

"Enabled"=dword:00000000

 

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\Schannel\Ciphers\RC4 128/128]

"Enabled"=dword:00000000

 

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\Schannel\Ciphers\RC4 40/128]

"Enabled"=dword:00000000

 

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\Schannel\Ciphers\RC4 56/128]

"Enabled"=dword:00000000

Then I use the following .reg file to enabled TLS v1.1 and TLS v1.2

Windows Registry Editor Version 5.00

 

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\Schannel\Protocols\TLS 1.1]

 

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\Schannel\Protocols\TLS 1.1\Client]

"Enabled"=dword:00000001

"DisabledByDefault"=dword:00000000

 

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\Schannel\Protocols\TLS 1.1\Server]

"Enabled"=dword:00000001

"DisabledByDefault"=dword:00000000

 

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\Schannel\Protocols\TLS 1.2]

 

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\Schannel\Protocols\TLS 1.2\Client]

"Enabled"=dword:00000001

"DisabledByDefault"=dword:00000000

 

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\Schannel\Protocols\TLS 1.2\Server]

"Enabled"=dword:00000001

"DisabledByDefault"=dword:00000000

 

Once you have applied both of the above sets of registry keys you can reboot the server at your convenience. Note that the regkeys may set values that are already set, for example TLS v1.1 and v1.2 are enabled on Exchange 2013 CAS servers and SSL v2 is disabled. For example the first of the below graphics comes from a test environment of mine that is running Windows Server 2012 R2 without any of the above registry keys set on them. You can see that Windows Server 2012 R2 is vulnerable to the POODLE attack and supports the RC4 cipher which is weak.

image

The F grade comes from patched but un-configured with regards to SSL Windows Server 2008 R2 server

image

After setting the above registry keys and rebooting, the test at https://www.ssllabs.com/ssltest then showed the following for 2012 R2 on the left (A grade) and Windows Server 2008 R2 on the right (A- grade):

image image

Disabling SSL v3 on a Kemp LoadMaster load balancer

If you protect your servers with a load balancer, which is common in the Exchange Server world, then you need to set your SSL and cipher settings on the load balancer, unless you are only balancing at TCP layer 4 and doing SSL pass through. Therefore even for clients that have a load balancer, you might not need to make the changes on the load balancer, but on the server via the above section instead. If you do SSL termination on the load balancer (TCP layer 7 load balancing) then I recommend setting the registry keys on the Exchange servers anyway to avoid security issues if you need to connect to the server directly and if you are going to disable SSL v3 in one location (the load balancer) there is no problem in disabling it on the server as well.

For a Kemp load balancer you need to be running version 7.1-20b to be able to do the following, and to ensure that the SSL code on the load balancer is not susceptible to issues such as heartbleed as well. To configure your load balancer to disable SSL v3 you need to modify the SSL properties of the virtual server and check the “Support TLS Only” option.

To disable the RC4 weak ciphers then there are a few choices, but the easiest I have seen to do is to select “Perfect Forward Secrecy Only” under Selection Filters and then add all the listed filters. Then from this list remove the three RC4 ciphers that are in the list.

If you do not select “Support TLS Only” and leave the ciphers at the default level then your load balancer will get an C grade at the test at https://www.ssllabs.com/ssltest because it is vulnerable to the POODLE attack. Setting just the “Support TLS Only” option and leaving the default ciphers in place will result in a B grade, as RC4 is still supported. Removing the RC4 ciphers (by following the instructions above to add the perfect forward secrecy ciphers and remove the RC4 ciphers from this list) as well as allowing only the TLS protocol will result in an A grade.

image

Kemp 7.1-22b does not support SSL v3 for the API and web interface as well as completing the above to protect the virtual services that the load balancer offers.

Kemp Technologies document the above steps at https://support.kemptechnologies.com/hc/en-us/articles/201995869, and point out the unobvious setting that if you filter the cipher list with the “TLS 1.x Ciphers Only” setting then it will only show you the TLS 1.2 ciphers and not any TLS 1.1 or TLS 1.0 ciphers. THerefore selecting “TLS 1.x Ciphers Only” rather than filtering using “Perfect Forward Secrecy Only” will result in a reduced client list, which may be an issue.

I was able to achieve an A grade on the SSL Labs test site. My certificate uses SHA-1, but expires in 2015 so by the time SHA-1 is reported an issue in the browser I will have changed it anyway.

image

Disabling SSL v3 on a JetNexus ALB-X load balancer

If you protect your servers with a load balancer, which is common in the Exchange Server world, then you need to set your SSL and cipher settings on the load balancer, unless you are only balancing at TCP layer 4 and doing SSL pass through. Therefore even for clients that have a load balancer, you might not need to make the changes on the load balancer, but on the server via the above section instead. If you do SSL termination on the load balancer (TCP layer 7 load balancing) then I recommend setting the registry keys on the Exchange servers anyway to avoid security issues if you need to connect to the server directly and if you are going to disable SSL v3 in one location (the load balancer) there is no problem in disabling it on the server as well.

For a JetNexus ALB-X load balancer you need to be running build 1553 or later. Build 1553 is a version 3 build, so any version 4 build is of a higher, and therefore valid build. This build (version 3.54.3) or later is needed to ensure Heartbleed mitigation and to allow the following configuration changes to be applied.

To configure the JetNexus  you need to upload a config file to turn off SSL v3 and RC4 ciphers. The config file is .txt file that is uploaded to the load balancer. In version 4, the primary cluster node can have the file uploaded to it, and the changes are replicated to the second node in the cluster automatically.

Before you upload a config file to make the changes required, ensure that you backup the current configuration from Advanced >> Update Software and click the button next to Download Current Configuration to save the configuration locally. Ensure you backup all nodes in a v4 cluster is appropriate.

Then select one of the three config file settings below and copy it to a text file and upload it from Advanced >> Update Software and use the Upload New Configuration option to install the file. The upload will reset all connections, do do this at during a quiet period of time.

The three configs are to reset the default ciphers, to disable SSL v3 and RC4, and to disable TLS v1.0 and SSL v3 and RC4

JetNexus protocol and cipher defaults:

#!update

 

[jetnexusdaemon]

Cipher004="ECDHE-RSA-AES128-SHA256:AES128-GCM-SHA256:RC4:HIGH:!MD5:!aNULL:!EDH"

Cipher1=""

Cipher2=""

CipherOptions="CIPHER_SERVER_PREFERENCE"

JetNexus protocol and cipher changes to disable SSL v3 and disable RC4 ciphers:

#!update

 

[jetnexusdaemon]

Cipher004="ECDHE-RSA-AES128-SHA256:AES128-GCM-SHA256:HIGH:!MD5:!aNULL:!EDH:!RC4"

Cipher1=""

Cipher2=""

CipherOptions="NO_SSLv3,CIPHER_SERVER_PREFERENCE"

JetNexus protocol and cipher changes to disable TLS v1.0, SSL v3 and disable RC4 ciphers:

#!update

 

[jetnexusdaemon]

Cipher004="ECDHE-RSA-AES128-SHA256:AES128-GCM-SHA256:HIGH:!MD5:!aNULL:!EDH:!RC4"

Cipher1=""

Cipher2=""

CipherOptions="NO_SSLv3,NO_TLSv1,CIPHER_SERVER_PREFERENCE"

On my test environment I was able to achieve an A- grade with the SSL Test website and the config to disable TLS 1.0, SSL3 and RC4 enabled. The A- is because of a lack of support for Forward Secrecy with the reference browsers used by the test site.

image

Browsers and Other Clients

There too much to discuss with regards to clients, apart from they need to support the same ciphers as mentioned above. A good guide to clients can be found at https://www.howsmyssl.com/s/about.html and from there you can test your client as well.

Additional comment 23/1/15 : One important comment to make though comes courtesy of Ingo Gegenwarth at https://ingogegenwarth.wordpress.com/2015/01/20/hardening-ssltls-and-outlook-for-mac/. This post discusses the TLS Renegotiation Indication Extension update at RFC 5746. It is possible to use the AllowInsecureRenegoClients registry key at HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\SecurityProviders\SCHANNEL to ensure that only clients with the update mentioned at http://support.microsoft.com/kb/980436 are allowed to connect. If this is enabled (set to Strict Mode) and the above to disable SSL 2 and 3 is done then Outlook for Mac clients cannot connect to your Exchange Server. If this regkey is deleted or has a non-zero value then connections to SSL 2 and 3 can be made, but only for a renegotiation to TLS. Therefore ensure that you allow Compatibility Mode (which is the default) when you disable SSL 2 and 3, as Outlook for Mac and Outlook for Mac for Office 365 both require SSL support to then be able to start a TLS session.

Exchange Web Services (EWS) and 501 Error

Posted on 11 CommentsPosted in 2010, 2013, EWS, exchange, exchange online, hybrid, IAmMEC, iis, Kemp, Load Master, Office 365, tmg

As is common with a lot that I write in this blog, it is based on noting down the answers to stuff I could not find online. For this issue, I did find something online by Michael Van “Hybrid”, but finding it was the challenge.

So rather than detailing the issue and the reason (you can read that on Michael’s blog) this talks about the steps to troubleshoot this issue.

So first the issue. When starting a migration test from an Exchange 2010 mailbox with an Exchange 2013 hybrid server (running the mailbox and CAS roles) behind a Kemp load balancing (running 7.16 – the latest release at the time of writing, but recently upgraded from version 5 through 6 to 7) I got the following error:

image

The server name will be different (thanks Michael for the screenshot). In my case this was my clients UK datacentre. My clients Hong Kong datacentre was behind a Kemp load balancer as well, but is only running Exchange 2010 and the New York datacentre has an F5 load balancer. Moves from HK worked, but UK and NY failed for different reasons.

The issue shown above is not easy to solve as the migration dialog tells you nothing. In my case it was also telling me the wrong server name. It should have been returning the External EWSUrl from Autodiscover for the mailbox I was trying to move, instead it was returning the Outlook Anywhere external URL from the New York site (as the UK is proxied via NY for the OA connections). For moves to the cloud, we added the External URL for EWS to each site directly so we would move direct and not via the only site that offered internet connected email.

So troubleshooting started with exrca.com – the Microsoft Connectivity Analyser. Autodiscover worked most of the time in the UK but Synchronization, Notification, Availability, and Automatic Replies tests (to test EWS) always failed after six and a half seconds.

Autodiscover returned the following error:

A Web exception occurred because an HTTP 400 – BadRequest response was received from Unknown.
HTTP Response Headers:
Connection: close
Content-Length: 87
Content-Type: text/html
Date: Tue, 24 Jun 2014 09:03:40 GMT

Elapsed Time: 108 ms.

And EWS, when AutoDiscover was returning data correctly, was as follows:

Creating a temporary folder to perform synchronization tests.

Failed to create temporary folder for performing tests.

Additional Details

Exception details:
Message: The request failed. The remote server returned an error: (501) Not Implemented.
Type: Microsoft.Exchange.WebServices.Data.ServiceRequestException
Stack trace:
at Microsoft.Exchange.WebServices.Data.ServiceRequestBase.GetEwsHttpWebResponse(IEwsHttpWebRequest request)
at Microsoft.Exchange.WebServices.Data.ServiceRequestBase.ValidateAndEmitRequest(IEwsHttpWebRequest& request)
at Microsoft.Exchange.WebServices.Data.ExchangeService.InternalFindFolders(IEnumerable`1 parentFolderIds, SearchFilter searchFilter, FolderView view, ServiceErrorHandling errorHandlingMode)
at Microsoft.Exchange.WebServices.Data.ExchangeService.FindFolders(FolderId parentFolderId, SearchFilter searchFilter, FolderView view)
at Microsoft.Exchange.WebServices.Data.Folder.FindFolders(SearchFilter searchFilter, FolderView view)
at Microsoft.Exchange.Tools.ExRca.Tests.GetOrCreateSyncFolderTest.PerformTestReally()
Exception details:
Message: The remote server returned an error: (501) Not Implemented.
Type: System.Net.WebException
Stack trace:
at System.Net.HttpWebRequest.GetResponse()
at Microsoft.Exchange.WebServices.Data.EwsHttpWebRequest.Microsoft.Exchange.WebServices.Data.IEwsHttpWebRequest.GetResponse()
at Microsoft.Exchange.WebServices.Data.ServiceRequestBase.GetEwsHttpWebResponse(IEwsHttpWebRequest request)

Elapsed Time: 6249 ms.

What was interesting here was the 501 and that it was always approx. 6 seconds before it failed.

Looking in the IIS logs from the 2010 servers that hold the UK mailboxes there were no 501 errors logged. The same was true for the EWS logs as well. So where is the 501 coming from. I decided to bypass Exchange 2013 for the exrca.com test (as my system is not yet live and that is easy to do) and so in Kemp pointed the EWS SubVDir directly to a specific Exchange 2010 server. Everything worked. So I decided it was an Exchange 2013 issue, apart from the fact I have lab environments the same as this (without Kemp) and it works fine there. So I decided to search for “Kemp EWS 501” and that was the bingo keyword combination. EWS and 501 or Exchange EWS and 501 got nothing at all.

With my environment back to Kemp >  2013 >  2010 I looked at Michaels suggestions. The first was to run Test-MigrationServerAvailability –ExchangeRemoteMove –RemoteServer servername.domain.com. I changed this slightly, as I was not convinced that I was connecting to the correct endpoint. The migration reported the wrong server name and the Exrca tests do not tell you what endpoint they are connecting to. So I tried Test-MigrationServerAvailability –ExchangeRemoteMove –Autodiscover –EmailAddress user-on-premises@domain.com

As AutoDiscover is reporting errors at times, the second of these cmdlets sometimes reported the following:

RunspaceId         : a711bdd3-b6a1-4fb8-96b8-f669239ea534
Result             : Failed
Message            : AutoDiscover failed with a configuration error: The migration service failed to detect the
migration endpoint using the Autodiscover service. Please enter the migration endpoint settings
or go back to the first step and retry using the Autodiscover service. Consider using the
Exchange Remote Connectivity Analyzer (
https://testexchangeconnectivity.com) to diagnose the
connectivity issues.
ConnectionSettings :
SupportsCutover    : False
ErrorDetail        : internal error:Microsoft.Exchange.Migration.AutoDiscoverFailedConfigurationErrorException:
AutoDiscover failed with a configuration error: The migration service failed to detect the
migration endpoint using the Autodiscover service. Please enter the migration endpoint settings
or go back to the first step and retry using the Autodiscover service. Consider using the
Exchange Remote Connectivity Analyzer (
https://testexchangeconnectivity.com) to diagnose the
connectivity issues.
at Microsoft.Exchange.Migration.DataAccessLayer.MigrationEndpointBase.InitializeFromAutoDiscove
r(SmtpAddress emailAddress, PSCredential credentials)
at Microsoft.Exchange.Management.Migration.TestMigrationServerAvailability.InternalProcessExcha
ngeRemoteMoveAutoDiscover()
IsValid            : True
Identity           :
ObjectState        : New

And when AutoDiscover was working (as it was random) I would get this:

RunspaceId         : a711bdd3-b6a1-4fb8-96b8-f669239ea534
Result             : Failed
Message            : The ExchangeRemote endpoint settings could not be determined from the autodiscover response. No
MRSProxy was found running at ‘outlook.domain.com’.
ConnectionSettings :
SupportsCutover    : False
ErrorDetail        : internal error:Microsoft.Exchange.Migration.MigrationRemoteEndpointSettingsCouldNotBeAutodiscovere
dException: The ExchangeRemote endpoint settings could not be determined from the autodiscover
response. No MRSProxy was found running at ‘outlook.domain.com’. —>
Microsoft.Exchange.Migration.MigrationServerConnectionFailedException: The connection to the
server ‘outlook.domain.com’ could not be completed. —>
Microsoft.Exchange.MailboxReplicationService.RemoteTransientException: The call to
https://outlook.domain.com/EWS/mrsproxy.svc’ failed. Error details: The HTTP request is
unauthorized with client authentication scheme ‘Negotiate’. The authentication header received
from the server was ‘Basic Realm=”outlook.domain.com”‘. –> The remote server returned an
error: (401) Unauthorized.. —>
Microsoft.Exchange.MailboxReplicationService.RemotePermanentException: The HTTP request is
unauthorized with client authentication scheme ‘Negotiate’. The authentication header received
from the server was ‘Basic Realm=”outlook.domain.com”‘. —>
Microsoft.Exchange.MailboxReplicationService.RemotePermanentException: The remote server returned
an error: (401) Unauthorized.
— End of inner exception stack trace —
— End of inner exception stack trace —
at Microsoft.Exchange.MailboxReplicationService.MailboxReplicationServiceFault.<>c__DisplayClas
s1.<ReconstructAndThrow>b__0()
at Microsoft.Exchange.MailboxReplicationService.ExecutionContext.Execute(Action operation)
at Microsoft.Exchange.MailboxReplicationService.MailboxReplicationServiceFault.ReconstructAndTh
row(String serverName, VersionInformation serverVersion)
at Microsoft.Exchange.MailboxReplicationService.WcfClientWithFaultHandling`2.<>c__DisplayClass1
.<CallService>b__0()
at Microsoft.Exchange.Net.WcfClientBase`1.CallService(Action serviceCall, String context)
at Microsoft.Exchange.Migration.MigrationExchangeProxyRpcClient.CanConnectToMrsProxy(Fqdn
serverName, Guid mbxGuid, NetworkCredential credentials, LocalizedException& error)
— End of inner exception stack trace —
at Microsoft.Exchange.Migration.DataAccessLayer.ExchangeRemoteMoveEndpoint.VerifyConnectivity()
at Microsoft.Exchange.Management.Migration.TestMigrationServerAvailability.InternalProcessEndpo
int(Boolean fromAutoDiscover)
— End of inner exception stack trace —
IsValid            : True
Identity           :
ObjectState        : New

This was returning the URL https://outlook.domain.com/EWS/mrsproxy.svc’ which is not correct for this mailbox (this was the OA endpoint in a different datacentre) and external Outlook access is not allowed at this company and so the TMG server in front of the F5 load balancer in the NY datacentre was not configured for OA anyway and browsing the the above URL returned the following picture, which is a well broken scenario but not the issue at hand here!

image

If OA (Outlook Anywhere) was available for this company, this is not what I would expect to see when browsing to the External EWS URL. To that end we have EWS URL’s are bypass TMG and go direct to the load balancer.

So now we have either no valid AutoDiscover response or EWS using the wrong URL. Back to the version of the cmdlet Michael was using as that ignores AutoDiscover: Test-MigrationServerAvailability –ExchangeRemoteMove –RemoteServer servername.domain.com

RunspaceId         : 5874c796-54ce-420f-950b-1d300cf0a64a
Result             : Failed
Message            : The connection to the server ‘ewsinukdatacentre.domain.com’ could not be completed.
ConnectionSettings :
SupportsCutover    : False
ErrorDetail        : Microsoft.Exchange.Migration.MigrationServerConnectionFailedException: The connection to the
server ‘ewsinukdatacentre.domain.com’ could not be completed. —>
Microsoft.Exchange.MailboxReplicationService.RemoteTransientException: The call to
https://ewsinukdatacentre.domain.com/EWS/mrsproxy.svc’ failed. Error details: The remote server returned
an unexpected response: (501) Invalid Request. –> The remote server returned an error: (501) Not
                     Implemented.. —> Microsoft.Exchange.MailboxReplicationService.RemotePermanentException: The
remote server returned an unexpected response: (501) Invalid Request. —>
Microsoft.Exchange.MailboxReplicationService.RemotePermanentException: The remote server returned
an error: (501) Not Implemented.
— End of inner exception stack trace —
— End of inner exception stack trace —
at Microsoft.Exchange.MailboxReplicationService.MailboxReplicationServiceFault.<>c__DisplayClas
s1.<ReconstructAndThrow>b__0()
at Microsoft.Exchange.MailboxReplicationService.ExecutionContext.Execute(Action operation)
at Microsoft.Exchange.MailboxReplicationService.MailboxReplicationServiceFault.ReconstructAndTh
row(String serverName, VersionInformation serverVersion)
at Microsoft.Exchange.MailboxReplicationService.WcfClientWithFaultHandling`2.<>c__DisplayClass1
.<CallService>b__0()
at Microsoft.Exchange.Net.WcfClientBase`1.CallService(Action serviceCall, String context)
at Microsoft.Exchange.Migration.MigrationExchangeProxyRpcClient.CanConnectToMrsProxy(Fqdn
serverName, Guid mbxGuid, NetworkCredential credentials, LocalizedException& error)
— End of inner exception stack trace —
at Microsoft.Exchange.Migration.DataAccessLayer.ExchangeRemoteMoveEndpoint.VerifyConnectivity()
at Microsoft.Exchange.Management.Migration.TestMigrationServerAvailability.InternalProcessEndpo
int(Boolean fromAutoDiscover)
IsValid            : True
Identity           :
ObjectState        : New

Now we can see the 501 error that exrca was returning. It would seem that the 501 is coming from the Kemp and not from the endpoint servers, which is why I could not located it in IIS or EWS logs and so in the Kemp System Message File (logging options > system log files) I found the 501 error:

kernel: L7: badrequest-client_read [157.56.251.92:61541->192.168.1.2:443] (-501): <s:Envelope ? , 0 [hlen 1270, nhdrs 8]

Where the first IP address was a Microsoft datacentre IP and the second was the Kemp listener IP.

It turns out this is due to the Kemp load balancer not returning to Microsoft the Continue-100 status that it should get. Microsoft waits 5 seconds and then sends the data it would have done if it got the response back. At this point the Kemp blocks this data with a 501 error.

It is possible to turn off Kemp’s processing of Continue-100 HTTP packets so it lets them through and this is covered in http://blog.masteringmsuc.com/2013/10/kemp-load-balancer-and-lync-unified.html:

 

In the version at my clients which was upgraded to version 7.0-16 from a v5 to v6 to v7 it was defaulting to RFC Conformant, but needs to be Ignore Continue-100 to work with Office 365. In versions of Kemp 7.1 and later the value needs to be changed to RFC-7231 Compliant from the default of “RFC-2616 Compliant”. Now EWS works, hybrid mailbox moves work, and AutoDiscover is always working on that server – so a mix of issues caused by differing interpretations of an RFC. To cover all these issues the Kemp load balancer started to include this 100-Continue Handling setting. We as ITPro’s need to ensure that we set the correct setting for our environment based on the software we use.