Capax Zantas EAS EWS 501 Errors

Posted on Leave a commentPosted in EAS, Exchange Server, Kemp, Load Master, Zantas

Whilst load balancing Exchange 2013 with a Capax Zantas EAS deployment in place I was reminded of an issue I experienced a few years ago with a Kemp load balancer. The EAS app for OWA was failing with various errors such as:


Communication with service failed. The service might be down or misconfigured. Error code: 500

As well as the search functionality never returning results.

And in the EASOWABackend log file we were seeing the following:

w3wp.exe Error: 0 : 22 : [09/23/2016 10:13:22.880 AM] User EWS Call failed Microsoft.Exchange.WebServices.Data.ServiceRequestException: The request failed. The remote server returned an error: (501) Not Implemented. —> System.Net.WebException: The remote server returned an error: (501) Not Implemented.
   at System.Net.HttpWebRequest.GetResponse()
   at Microsoft.Exchange.WebServices.Data.EwsHttpWebRequest.Microsoft.Exchange.WebServices.Data.IEwsHttpWebRequest.GetResponse()
   at Microsoft.Exchange.WebServices.Data.ServiceRequestBase.GetEwsHttpWebResponse(IEwsHttpWebRequest request)
   — End of inner exception stack trace —
   at Microsoft.Exchange.WebServices.Data.ServiceRequestBase.GetEwsHttpWebResponse(IEwsHttpWebRequest request)
   at Microsoft.Exchange.WebServices.Data.ServiceRequestBase.ValidateAndEmitRequest(IEwsHttpWebRequest& request)
   at Microsoft.Exchange.WebServices.Data.MultiResponseServiceRequest`1.Execute()
   at Microsoft.Exchange.WebServices.Data.ExchangeService.InternalConvertIds(IEnumerable`1 ids, IdFormat destinationFormat, ServiceErrorHandling errorHandling)
   at Microsoft.Exchange.WebServices.Data.ExchangeService.ConvertId(AlternateIdBase id, IdFormat destinationFormat)
   at EASOWA2013.EASOWABackend.OpenMessage(String owaID, String userEmail, String authToken, String pageURL)

This error contained a 501 Not Implemented message, and this message was getting exposed in OWA. But this error was not present in any of the Exchange Server logs, so it was not being caused by Exchange. This led us to looking at the load balancer, and testing EAS when bypassing the load balancer. To bypass the load balancer requires host file changes and changing the EAS configuration, so it is not a trivial change. When the load balancer was out of the traffic path everything worked fine.

Under the Kemp system logging (Logging Options > System Log Files > System Message File) we were seeing:

Sep 22 16:39:09 cwh-kemp01 kernel: L7: badrequest-client_read [192.168.x.y:44987->] (-501):] (-501):] (-501):] (-501):] (-501):] (-501):] (-501):] (-501):] (-501):] (-501):] (-501):

Where 192.168.x.y was the client IP and was the load balanced IP for Exchange.

With reference to the above linked article, the fix was to change the Kemp’s handling of the 100-Continue feature and by changing the setting to RFC-7231 Compliant EAS in OWA started working:


Photos, Exchange, And The File System

Posted on 1 CommentPosted in 2013, 2016, Exchange Server, Office 365, owa

On an Exchange 2013 and later server this is a folder called photos that gets created after installation and can contain a couple of user photos for some of your users. How does it get there and what does it contain?

The photos folder appears (on 2016 anyway) when the user uploads a photo (via OWA). Two images are created one 96square and the other 648square. They are made in a folder unique to the user and on the mailbox server that contains their active mailbox at the time of upload.

To reproduce this, login to OWA. Determine which server is currently the active server for that mailbox and then access the file system of that server. You are looking for “C:\Program Files\Microsoft\Exchange Server\V15\ClientAccess” though it will be wherever Exchange Server was installed if not the C: drive. If anyone has uploaded photos already via this server then you will see a folder named photo. You can delete this folder without impact (unless someone is actively uploading a photo at that exact time).

In OWA, click the photo icon top right and then click Change:



Click Upload photo and select a photo. I’ve used the sample pictures that are installed on Windows 7 in this example:


At this point a copy of the photo is uploaded to a web service on Exchange Server. Click Save above your chosen photo. At this point the photo folder in the ClientAccess folder on the server that is active for your mailbox is created. Inside this folder you will see a subfolder called Inside this folder will be two subfolders called HR96x96 and HR648x648. Inside each of these will be the JPEG file that was created at the time of saving the upload. The size of each will match the folder name and the name of the file will be _Alias-UNIQUEID. If the user deletes their photo then a 0 byte JPEG file will be created in the folder.

Note that these two photos are not a cache of the photo for the Exchange Server to download to other users. They are just used during uploading the photos. Once uploaded they are resized using this file system location and then stored in their respective locations. The 96×96 photo (at less than 100Kb) is stored in the Active Directory and the 648×648 image is stored in the Exchange 2013 or later mailbox for use by Exchange, Skype for Business and SharePoint.

If there are policies and privacy laws that state the caching of images on the file system must be avoided, then you should be able to delete the photo upload cache at your convenience.

The photo folder does not appear on another server when viewing that user with a photo in their mailbox. Requesting the photo is done via owa/service.svc and not AFAIK from a file on the file system.

Deleting the folder after the fact did not impact my test users photo (as its now in the mailbox and not read from the file system). If this mailbox is later migrated to Office 365, then the photo will migrate with the mailbox as it is part of the mailbox. If the photo stored in AD is less than 100kb then it will be synced to Azure AD.

Exchange Online Archive–Counting Archives

Posted on Leave a commentPosted in archive, exchange, exchange online, Exchange Server, EXO, IAmMEC, Office 365

If you are using Exchange Online Archive and what to get a count of the number of users with an archive, or a list of the users with an archive, then the following PowerShell scripts will give you this info:

List all users with an Exchange Online Archive:

Get-MailUser -ResultSize Unlimited | where {$_.ArchiveName -ilike “In-Place Archive*”}

Count all users with an Exchange Online Archive:

(Get-MailUser -ResultSize Unlimited | where {$_.ArchiveName -ilike “In-Place Archive*”}).Count

Both of these PowerShell cmdlets need to be run in Exchange Online via Remote PowerShell.

Exchange Server and Missing Root Certificates

Posted on Leave a commentPosted in 2007, 2010, 2013, exchange, exchange online, Exchange Server, federation, Free/Busy

I came across an issue with a clients Exchange Server deployment today that is not well documented – or rather it is, but you need to know where to look. So I thought I would document the troubleshooting steps and the fix here.

We specifically came across this error when testing Free/Busy for an Office 365 migration, though it could happen for a variety of reasons. Free/Busy and other lookups in a cross-forest Exchange Server deployment require a working organization configuration and this was failing. Running Test-FederationTrust (a prerequisite of the organization relationship) in verbose mode (add -Verbose to the end) returned the following:

Unable to retrieve federation metadata from the security token
service. Reason: Microsoft.Exchange.Management.FederationProvisioning.FederationMetadataException: Unable to access the
Federation Metadata document from the federation partner. Detailed information: “The underlying connection was closed:
Could not establish trust relationship for the SSL/TLS secure channel.”.

The final result of the test will also show two errors for “Unable to retrieve federation metadata from the security token service.” and “Failed to request delegation token.”

The last part of the verbose error is the clue here. The server in question is unable to make an SSL/TLS connection to the endpoint that the federation trust needs to reach to get the federation trust metadata. That endpoint is listed right at the start of the Verbose output. It reads:

VERBOSE: [16:53:08.306 GMT] Test-FederationTrust : Requesting Federation Metadata from

Now that we have a URL and an error message, check that the URL is reachable from each of your Exchange Servers. At my client today we found one server could not successfully reach this endpoint without an SSL error turning up in the browser. The problem was that the certificate that the endpoint is secure with is issued by the Baltimore Cybertrust Root Certificate – one that Microsoft uses for lots of services, but the root certificate was not installed on that machine. Lots of root certs where missing from that machine as it had never had a root certificate update applied to it.

We installed the latest Root Certificate Update and then the federation trust worked and free/busy etc. (mail tips, cross-forest message tracking etc.) all worked fine.

Exchange 2013 Partner Applications and Error 2008

Posted on 2 CommentsPosted in Exchange Server, Lync Server, OAuth, Skype for Business Server

When Exchange Server 2013 is configured to connect to Lync / Skype for Business Server one of the steps is to create a partner application. When this is first run the partner application stores the certificate presented by Lync Server in the Active Directory configuration partition. If the certificate changes on the Lync Server then the Exchange Server will start to alert about this every 15 minutes with the following warning: MSExchange OAuth and Event ID 2008.


To fix this error we need to update the Active Directory configuration where the metadata info is kept (CN=LyncEnterprise-guid,CN=Partner Applications,CN=Auth Configuration,CN=Exchange Organization Name,CN=Microsoft Exchange, CN=Services, CN=Configuration, DC=domain,DC=co,DC=uk)

To do this you need to start Exchange Management Shell on Exchange 2013 and remove the Lync partner application. Immediately after you have done this you can create the partner application again. The cmdlets for this are as follows:

is the same as you used when you first created the partner application. This is as follows:

  1. In the Exchange Management Shell confirm you have one Lync partner application with Get-PartnerApplication | FL
  2. In the Exchange Management Shell run Remove-PartnerApplication Lync* and then confirm that you want to do this.
  3. In the Exchange Management Shell, change directory to the Exchange scripts folder with cd $exscripts
  4. Then run the script to configure a new partner application. An example would be .\Configure-EnterprisePartnerApplication.ps1 -AuthMetadataUrl ‘’ -ApplicationType Lync where the URL points to the Lync server and contains a valid name for the certificate on the Lync Server.

This should return the following and report that the configuration has succeeded.


Your 2008 repeating errors in the Event Viewer will now be gone.

Qualifications in Exchange Signatures

Posted on Leave a commentPosted in 2013, active directory, exchange, Exchange Server, Global Catalog, IAmMEC, iQ.Suite

In a recent project I was working with iQ.Suite from GBS and specifically the component of this software that add signatures to emails. The client are an international organization with users in different geographies and we needed to accommodate the users qualifications in their email signature.

The problem with this is that in Germany qualifications are written in front of the name and in the USA at the end and in other countries at the start and the end. We were doing a Notes to Exchange migration and in Notes the iQ.Suite signature software read data from Notes that was originally pulled from Active Directory, and so the client had placed the qualifications in the DisplayName field in the Active Directory.

But when we migrated to Exchange Server the Global Address List listed the users DisplayName an so the German users where all listed together with “Dipl” as the first characters of their name. Also the name the email came from was written like this. The signature worked, but the other changes that became apparent meant we had to work out a different way to look at this problem.

So rather than using DisplayName for the users name and qualifications, we used personalPrefix in Active Directory to store anything needed before their name (Dipl in the above German example, and Prof or Dr being English examples) and the generationQualifier Active Directory attribute to store any string that followed the users DisplayName (such as Jr in the USA or BSc for qualifications etc.)

In iQ.Suite we created a signature that looked like the following. This has a conditional [COND] entry for personalTitle, displayName and generationQualifier. That is if each of these are present, then show the displayName with personalTitle before it and generationQualifier after it. If the user does not have values for these fields, do not show them. The [COND] control is documented in iQ.Suite.

[COND]personalTitle;[VAR]personalTitle[/VAR] [/COND][COND]displayName;[VAR]displayName[/VAR][/COND][COND]generationQualifier; [VAR]generationQualifier[/VAR][/COND]

What was not so well documented, and why I wanted to write this blog entry was that the personalTitle and generationQualifier attributes are not stored in the Global Catalog and so are missing in the users signature. In the multi-domain deployment we had at the client, iQ.Suite read the personalTitle, displayName and generationQualifier Active Directory attributes from the Global Catalog as Exchange was installed in a resource domain and the users in separate domains and so unless the attribute was pushed to the Global Catalog it was not seen by iQ.Suite.

To promote an attribute to be visible in the Global Catalog you need to open the Schema Management MMC snap-in, find the attributes of question and tick the Replicate this attribute to the Global Catalog field. This is outlined in