Exchange Edge Server and Common Attachment Blocking In Exchange Online Protection

Posted on Leave a commentPosted in 2007, 2010, 2013, 2016, Edge, EOP, exchange, exchange online, Exchange Online Protection, FOPE, IAmMEC, Office 365

Both Exchange Server Edge role and Exchange Online Protection have an attachment filtering policy. The default in Edge Server is quite long, and the default in EOP is quite short. There is also a few values that are common to both.

So, how do you merge the lists so that your Edge Server attachment filtering policy is copied to Exchange Online in advance of changing your MX record to EOP?

You run

Set-MalwareFilterPolicy Default -FileTypes ade,adp,cpl,app,bas,asx,bat,chm,cmd,com,crt,csh,exe,fxp,hlp,hta,inf,ins,isp,js,jse,ksh,lnk,mda,mdb,mde,mdt,mdw,mdz,msc,msi,msp,mst,ops,pcd,pif,prf,prg,ps1,ps11,ps11xml,ps1xml,ps2,ps2xml,psc1,psc2,reg,scf,scr,sct,shb,shs,url,vb,vbe,vbs,wsc,wsf,wsh,xnk,ace,ani,docm,jar

This takes both the Edge Server default list and the EOP default list, minus the duplicate values and adds them to EOP. If you have a different custom list then use the following PowerShell to get your two lists and then use the above (with “Default” being the name of the policy) PowerShell to update the list in the cloud

Edge Server: Get-AttachmentFilterEntry

EOP: $variable = Get-MalwareFilterPolicy Default
$variable.FileTypes

Exchange Server and Missing Root Certificates

Posted on Leave a commentPosted in 2007, 2010, 2013, exchange, exchange online, Exchange Server, federation, Free/Busy

I came across an issue with a clients Exchange Server deployment today that is not well documented – or rather it is, but you need to know where to look. So I thought I would document the troubleshooting steps and the fix here.

We specifically came across this error when testing Free/Busy for an Office 365 migration, though it could happen for a variety of reasons. Free/Busy and other lookups in a cross-forest Exchange Server deployment require a working organization configuration and this was failing. Running Test-FederationTrust (a prerequisite of the organization relationship) in verbose mode (add -Verbose to the end) returned the following:

Unable to retrieve federation metadata from the security token
service. Reason: Microsoft.Exchange.Management.FederationProvisioning.FederationMetadataException: Unable to access the
Federation Metadata document from the federation partner. Detailed information: “The underlying connection was closed:
Could not establish trust relationship for the SSL/TLS secure channel.”.

The final result of the test will also show two errors for “Unable to retrieve federation metadata from the security token service.” and “Failed to request delegation token.”

The last part of the verbose error is the clue here. The server in question is unable to make an SSL/TLS connection to the endpoint that the federation trust needs to reach to get the federation trust metadata. That endpoint is listed right at the start of the Verbose output. It reads:

VERBOSE: [16:53:08.306 GMT] Test-FederationTrust : Requesting Federation Metadata from
https://nexus.microsoftonline-p.com/FederationMetadata/2006-12/FederationMetadata.xml.

Now that we have a URL and an error message, check that the URL is reachable from each of your Exchange Servers. At my client today we found one server could not successfully reach this endpoint without an SSL error turning up in the browser. The problem was that the certificate that the endpoint is secure with is issued by the Baltimore Cybertrust Root Certificate – one that Microsoft uses for lots of services, but the root certificate was not installed on that machine. Lots of root certs where missing from that machine as it had never had a root certificate update applied to it.

We installed the latest Root Certificate Update and then the federation trust worked and free/busy etc. (mail tips, cross-forest message tracking etc.) all worked fine.

Unable To Send Exchange Quota Message

Posted on Leave a commentPosted in 2010, 2013, exchange, IAmMEC

In Exchange 2013 you can sometimes see the following event log error (MSExchange Store Driver Submission, ID 1012):

The store driver failed to submit event <id> mailbox <guid> MDB <database guid> and couldn’t generate an NDR due to exception Microsoft.Exchange.MailboxTransport.StoreDriverCommon.InvalidSenderException
   at Microsoft.Exchange.MailboxTransport.Shared.SubmissionItem.SubmissionItemUtils.CopySenderTo(SubmissionItemBase submissionItem, TransportMailItem message)
   at Microsoft.Exchange.MailboxTransport.Submission.StoreDriverSubmission.MailItemSubmitter.GenerateNdrMailItem()
   at Microsoft.Exchange.MailboxTransport.Submission.StoreDriverSubmission.MailItemSubmitter.<>c__DisplayClass1.<FailedSubmissionNdrWorker>b__0()
   at Microsoft.Exchange.MailboxTransport.StoreDriverCommon.StorageExceptionHandler.RunUnderTableBasedExceptionHandler(IMessageConverter converter, StoreDriverDelegate workerFunction).

And this will be preceded by the following event log warning (MSexchangeIS, ID 1077):

The mailbox <guid> on database <database guid> is approaching its storage limit. A notification has been sent to the user. This warning will not be sent again for at least twenty four hours.

The mailbox in both errors is the same and it occurs for mailboxes that have moved to Exchange Server 2013 from Exchange Server 2010 and are close to their mailbox quota. To fix the issue move the mailbox to a different database. The easiest way to do this is New-MoveRequest <guid> where the same GUID is used.

If you have lots of these then this is a little more time consuming, unless you get PowerShell to the rescue.

The following two cmdlets will query the last seven days of the event logs for MSexchangeIS sourced events with ID 1077, get the event log message (which contains the mailbox guid), manipulate the string containing the message and generate a text file of just the mailbox guids. The second cmdlet will run a New-MoveRequest for each mailbox listed in the text file.

Get-WinEvent -ComputerName PC1 -ProviderName MSExchangeIS | where {$_.ID -eq 1077 -AND $_.TimeCreated -gt [DateTime]::Now.AddDays(-7).Date} | select @{Name="mailbox";Expression={$_.Message.Substring(12,36)}} | ft -HideTableHeaders -AutoSize | out-file nearquota.txt

and then

Get-Content .\nearquota.txt | foreach {New-MoveRequest -Identity $_}

Make sure though that your Application event log is large enough to store more than seven days of events and then run these cmdlets, per server every seven days until the issue goes away (or over the course of say a year, move all mailboxes to different databases and that fixes it as well).

Exchange OWA and Multi-Factor Authentication

Posted on 10 CommentsPosted in 2010, 2013, Azure, exchange, IAmMEC, MFA, MVP, owa, smartphone

Multi-factor authentication (MFA), that is the need to have a username, password and something else to pass authentication is possible with on-premises servers using a service from Windows Azure and the Multi-Factor Authentication Server (an on-premises piece of software).

The Multi-Factor Authentication Server intercepts login request to OWA, if the request is valid (that is the username and password work) then the mobile phone of the user is called or texted (or an app starts automatically on the phone) and the user validates their login. This is typically done by pressing # (if a phone call) or clicking Verify in the app, but can require the entry of a PIN as well.

To configure Multi-Factor Authentication Server for OWA you need to complete the following steps:

Some of these steps are the same regardless of which service you are adding MFA to and some slightly different. I wrote a blog on MFA and VPN at http://c7solutions.com/2015/01/windows-rras-vpn-and-multi-factor-authentication and this contains the general setup steps and so these are not repeated here. Just what you need to do differently

Step 1

See http://c7solutions.com/2015/01/windows-rras-vpn-and-multi-factor-authentication

Step 2: Install MFA Server on-premises

This is covered in http://c7solutions.com/2015/01/windows-rras-vpn-and-multi-factor-authentication, but the difference with OWA is that it needs to be installed on the Exchange CAS server where the authentication takes place.

Ensure you have .NET 3.5 installed via Server Manager > Features. This will install the .NET 2.0 feature that is required by MFA server. If the installation of the download fails, this is the most likely reason for the failure, so install .NET 3.5 and then try the MFA Server install again.

The install of the MFA server does not take very long. After a few minutes the install will complete and then you need to run the Multi-Factor Authentication Server admin tools. These are on the Start Screen in More Apps or the Start Menu. Note that it will start the software itself if given time:

image

image

Do not skip the wizard, but click Next. You will be asked to activate the server. Activating the server is linking it to your Azure MFA instance. The email address and password you need are obtained from the Azure multi-factor auth provider that was configured in Step 1. Click the Generate Activation Credentials on the Downloads page of the Azure MFA provider auth management page.

image

The credentials are valid for ten minutes, so your will differ from mine. Enter them into the MFA Server configuration wizard and click Next.

MFA Server will attempt to reach Azure over TCP 443.

Select the group of servers that the configuration should replicate around. For example, if you where installing this software on each Exchange CAS server, then you might enter “Exchange Servers” as the group name in the first install and then select it during the install on the remaining servers. This config will be shared amongst all servers with the same group name. If you already have a config set up with users in it and set up a new group here, then it will be different settings for the users. For example you might have a phone call to authenticate a VPN connection but use the app for OWA logins. This would require two configs and different groups of servers. If you want the same settings for all users in the entire company, then one group (the default group) should be configured.

image

Next choose if you want to replicate your settings. If you have more than one MFA Server instance in the same group select yes.

Then choose what you want to authenticate. Here I have chosen OWA:

image

Then I need to choose the type of authentication I have in place. In my OWA installation I am using the default of Forms Based Authentication, but if you select Forms-based authentication here, the example URL for forms based authentication shown on the next page is from Exchange Server 2003 (not 2007 or later). Therefore I select HTTP authentication

Next I need to provide the URL to OWA. I can get this by browsing the OWA site over https. The MFA install will also use HTTPS, so you will need a certificate and have this trusted by a third party if you want to support user managed devices. Users managing their own MFA settings (such as telephone numbers and form of authentication) reduces the support requirement. That needs the User Portal, the SDK and the Mobile App webservice installed as well. These are outside the scope of this blog. For here I am going to use https://servername/owa.

image

Finish the installation at this time and wait for the admin application to appear.

Step 3: Configure Users for MFA

Here we need to import the users who will be authenticated with MFA. Select the Users area and click Import from Active Directory. Browse the settings to imports group members, or OUs or a search to add your user account. Once you have it working for yourself, add others. Users not listed here will not see any change in their authentication method.

Ensure that your test user has a mobile number imported from the Active Directory. If not add one, choosing the correct country code as well. The default authentication for the user is that they will get a phone call to this number and need to press # before they can be logged in. Ensure that the user is set to Enabled as well in the users area of the management program.

Step 4: Configure OWA for MFA (additional steps)

On the IIS Authentication node you can adjust the default configuration for HTTP. Here you need to set Require Multi-Factor Authentication user match. This ensures that each auth attempt is matched to a user in the users list. If the user exists and is enabled, then do MFA for them. If disabled, then the setting for Succeed Authentication on the advanced tab comes into play. If the user is not listed, authentication passes through without MFA.

image

Change to the Native Module tab and select OWA under Default Web Site only. Do not set authentication on the Backend Web Site. Also enable the native module on ECP on the Default Web Site as well:

image

Then I can attempt a login to OWA or ECP. Once I successfully authenticate my phone rings and I am prompted to press #. Once I press # I am allowed into Exchange!

Group Policy Import To Fix Google Chrome v37 Issues With Exchange Server and Microsoft CRM

Posted on 2 CommentsPosted in 2010, 2013, Chrome, crm, Dynamics, exchange, exchange online, Group Policy, IAmMEC, Office 365, owa

A recent update to Google Chrome (37.0.2062.120) removed the ability to support modal dialog boxes. This are dialogs that require your attention and stop you going back to the previous page until you have completed the info required – these are very useful in workflow type scenarios.

Google claim that as 0.004% of web sites use them (from Google anonymous statistics gathering that you can opt into in Chrome) they are justified in removing support for them – but they have not removed other things that have the same level of support!

With this version of Chrome (or the Chromium open source browser) there is a work around until April 30th 2015 that will allow modal dialogs to work again. Without this work around clicking links in OWA and ECP in Exchange 2010 and OWA and EAC in Exchange Online and Exchange 2013 will not popup. This can cause issues such as the inability to attach files in OWA and to create objects in ECP/EAC for the administrator. Popups in Microsoft CRM also do not work.

As a work around you could use a different browser, but if Chrome works for you (or does not in this case) and you are joined to a domain then you can download the following GPO export file and import it into your Active Directory to enable modal dialogs to work again in Exchange Server, Office 365 and Microsoft CRM products.

To download and import this GPO file to enable Chrome modal dialog box functionality to resume (until 30th April 2015, when Google stop allowing the work around) follow these steps:

  1. Download Google Chrome Show Modal Dialog Before 30 April 2015.zip
  2. Copy to a domain controller and expand the zip file. Ensure the contents of the zip file are not placed directly on your desktop as you cannot import from the desktop directly, so if you expand the zip to the desktop then copy the one folder that was created into a new subfolder.
  3. Start Group Policy Management MMC admin tool.
  4. Expand Forest > Domain > Your Domain > Group Policy Objects.
  5. Right click “Group Policy Objects” and choose New
  6. Create a new GPO called “Chrome and Chromium Modal Dialog Box Allow”:
    image
  7. Right click “Chrome and Chromium Modal Dialog Box Allow” GPO that you just made and choose Import Settings
  8. Proceed through the import wizard. You do not need to backup this new GPO on the second page of the dialog as the new GPO is empty.
  9. On the third page of the wizard browse to the parent folder containing the contents of the download above:
    image
  10. Click Next and you should see one backed up GPO listed:
    image
  11. Click Next to import this. If you click View Settings first a web page will open showing you that this GPO sets two registry keys for the computer and two registry keys for the user. These set SOFTWARE\Policies\Chromium\EnableDeprecatedWebPlatformFeatures and Software\Policies\Google\Chrome\EnableDeprecatedWebPlatformFeatures (for both Chromium and Chrome browsers) with a reg key (type string) 1:ShowModalDialog_EffectiveUntil20150430
  12. Proceed with Next and then Finish and the import will begin:
    image
  13. Click OK.
  14. Now link the GPO object to the root of your domain so it impacts all users and to the root of any OU that blocks inheritance. Import to other domains as above or link from this domain depending upon your current policy for managing GPO cross domains.
  15. Delete the zip and folder you downloaded. They are not needed any more.

Exchange Online Free/Busy Issues with OAuth Authentication

Posted on Leave a commentPosted in 2010, 2013, EWS, exchange, exchange online, Free/Busy, OAuth, Office 365

Update: 10 Dec 2014: It is reported that this issue is fixed in CU7 for Exchange Server 2013

OAuth authentication is a new server to server authentication model available in Exchange 2013 SP1 and later and Exchange Online (Office 365). With OAuth enabled and Exchange hybrid in place and where you have multiple endpoints of Exchange Server on-premises and those on-premises Exchange Servers are different versions then you might have issues getting Exchange Online to On-Premises free/busy lookups to work.

Here is the scenario:

Company with Exchange 2010 servers in multiple internet connected sites, going hybrid to Exchange Online.

Exchange Online tenant created and hybrid mode put in place between Exchange Online and Exchange Server 2013 on-premises. In the site where the Exchange 2013 hybrid servers are located there are Exchange 2010 SP3 servers. As hybrid mode was set up with SP1, OAuth was enabled.

Exchange 2010 in the remote sites is configured with an ExternalURL for EWS. Therefore a free/busy lookup from an Office 365 user to a mailbox in one of these remote sites goes direct to the EWS endpoint on Exchange 2010 – it is not proxied via the 2013 hybrid server.

With OAuth enabled this configuration will fail as Exchange Online will use OAuth to authenticate to Exchange 2010 on-premises and fail. The IIS logs will contain entries such as this:

2014-07-22 19:39:34 10.100.28.73 POST /ews/exchange.asmx – 443 – 10.100.28.220 ASProxy/CrossForest/EmailDomain//15.00.0985.008 401 0 0 0

Where the 401 indicates authentication failed and the path ASProxy/CrossForest/EmailDomain indicating OAuth in use. There will be no entries in the IIS log for the Federation Org type of authentication.

If the EWS connection for free/busy goes via the 2013 hybrid server (ExternalURL for the remote site is null) then the free/busy lookup works, or if the OAuth connector in Exchange Online is disabled (Get-IntraOrganizationConnector | Set-IntraOrganizationConnector -Enabled $false from Exchange Online remote PowerShell session) and EWS lookup for free/busy goes direct to the remote Exchange 2010 server then free/busy lookups work.

So if you want OAuth and direct EWS connections to remote sites for free/busy you need Exchange 2013 at those remote sites. If you want to have Exchange 2013 hybrid servers only at your primary site (for mail flow) and OAuth as well (for eDiscovery cross-forest) then you need to proxy your EWS free/busy requests via the Exchange 2013 hybrid server.

This is a known issue in Exchange and may be fixed in the future.

Exchange Web Services (EWS) and 501 Error

Posted on 11 CommentsPosted in 2010, 2013, EWS, exchange, exchange online, hybrid, IAmMEC, iis, Kemp, Load Master, Office 365, tmg

As is common with a lot that I write in this blog, it is based on noting down the answers to stuff I could not find online. For this issue, I did find something online by Michael Van “Hybrid”, but finding it was the challenge.

So rather than detailing the issue and the reason (you can read that on Michael’s blog) this talks about the steps to troubleshoot this issue.

So first the issue. When starting a migration test from an Exchange 2010 mailbox with an Exchange 2013 hybrid server (running the mailbox and CAS roles) behind a Kemp load balancing (running 7.16 – the latest release at the time of writing, but recently upgraded from version 5 through 6 to 7) I got the following error:

image

The server name will be different (thanks Michael for the screenshot). In my case this was my clients UK datacentre. My clients Hong Kong datacentre was behind a Kemp load balancer as well, but is only running Exchange 2010 and the New York datacentre has an F5 load balancer. Moves from HK worked, but UK and NY failed for different reasons.

The issue shown above is not easy to solve as the migration dialog tells you nothing. In my case it was also telling me the wrong server name. It should have been returning the External EWSUrl from Autodiscover for the mailbox I was trying to move, instead it was returning the Outlook Anywhere external URL from the New York site (as the UK is proxied via NY for the OA connections). For moves to the cloud, we added the External URL for EWS to each site directly so we would move direct and not via the only site that offered internet connected email.

So troubleshooting started with exrca.com – the Microsoft Connectivity Analyser. Autodiscover worked most of the time in the UK but Synchronization, Notification, Availability, and Automatic Replies tests (to test EWS) always failed after six and a half seconds.

Autodiscover returned the following error:

A Web exception occurred because an HTTP 400 – BadRequest response was received from Unknown.
HTTP Response Headers:
Connection: close
Content-Length: 87
Content-Type: text/html
Date: Tue, 24 Jun 2014 09:03:40 GMT

Elapsed Time: 108 ms.

And EWS, when AutoDiscover was returning data correctly, was as follows:

Creating a temporary folder to perform synchronization tests.

Failed to create temporary folder for performing tests.

Additional Details

Exception details:
Message: The request failed. The remote server returned an error: (501) Not Implemented.
Type: Microsoft.Exchange.WebServices.Data.ServiceRequestException
Stack trace:
at Microsoft.Exchange.WebServices.Data.ServiceRequestBase.GetEwsHttpWebResponse(IEwsHttpWebRequest request)
at Microsoft.Exchange.WebServices.Data.ServiceRequestBase.ValidateAndEmitRequest(IEwsHttpWebRequest& request)
at Microsoft.Exchange.WebServices.Data.ExchangeService.InternalFindFolders(IEnumerable`1 parentFolderIds, SearchFilter searchFilter, FolderView view, ServiceErrorHandling errorHandlingMode)
at Microsoft.Exchange.WebServices.Data.ExchangeService.FindFolders(FolderId parentFolderId, SearchFilter searchFilter, FolderView view)
at Microsoft.Exchange.WebServices.Data.Folder.FindFolders(SearchFilter searchFilter, FolderView view)
at Microsoft.Exchange.Tools.ExRca.Tests.GetOrCreateSyncFolderTest.PerformTestReally()
Exception details:
Message: The remote server returned an error: (501) Not Implemented.
Type: System.Net.WebException
Stack trace:
at System.Net.HttpWebRequest.GetResponse()
at Microsoft.Exchange.WebServices.Data.EwsHttpWebRequest.Microsoft.Exchange.WebServices.Data.IEwsHttpWebRequest.GetResponse()
at Microsoft.Exchange.WebServices.Data.ServiceRequestBase.GetEwsHttpWebResponse(IEwsHttpWebRequest request)

Elapsed Time: 6249 ms.

What was interesting here was the 501 and that it was always approx. 6 seconds before it failed.

Looking in the IIS logs from the 2010 servers that hold the UK mailboxes there were no 501 errors logged. The same was true for the EWS logs as well. So where is the 501 coming from. I decided to bypass Exchange 2013 for the exrca.com test (as my system is not yet live and that is easy to do) and so in Kemp pointed the EWS SubVDir directly to a specific Exchange 2010 server. Everything worked. So I decided it was an Exchange 2013 issue, apart from the fact I have lab environments the same as this (without Kemp) and it works fine there. So I decided to search for “Kemp EWS 501” and that was the bingo keyword combination. EWS and 501 or Exchange EWS and 501 got nothing at all.

With my environment back to Kemp >  2013 >  2010 I looked at Michaels suggestions. The first was to run Test-MigrationServerAvailability –ExchangeRemoteMove –RemoteServer servername.domain.com. I changed this slightly, as I was not convinced that I was connecting to the correct endpoint. The migration reported the wrong server name and the Exrca tests do not tell you what endpoint they are connecting to. So I tried Test-MigrationServerAvailability –ExchangeRemoteMove –Autodiscover –EmailAddress user-on-premises@domain.com

As AutoDiscover is reporting errors at times, the second of these cmdlets sometimes reported the following:

RunspaceId         : a711bdd3-b6a1-4fb8-96b8-f669239ea534
Result             : Failed
Message            : AutoDiscover failed with a configuration error: The migration service failed to detect the
migration endpoint using the Autodiscover service. Please enter the migration endpoint settings
or go back to the first step and retry using the Autodiscover service. Consider using the
Exchange Remote Connectivity Analyzer (
https://testexchangeconnectivity.com) to diagnose the
connectivity issues.
ConnectionSettings :
SupportsCutover    : False
ErrorDetail        : internal error:Microsoft.Exchange.Migration.AutoDiscoverFailedConfigurationErrorException:
AutoDiscover failed with a configuration error: The migration service failed to detect the
migration endpoint using the Autodiscover service. Please enter the migration endpoint settings
or go back to the first step and retry using the Autodiscover service. Consider using the
Exchange Remote Connectivity Analyzer (
https://testexchangeconnectivity.com) to diagnose the
connectivity issues.
at Microsoft.Exchange.Migration.DataAccessLayer.MigrationEndpointBase.InitializeFromAutoDiscove
r(SmtpAddress emailAddress, PSCredential credentials)
at Microsoft.Exchange.Management.Migration.TestMigrationServerAvailability.InternalProcessExcha
ngeRemoteMoveAutoDiscover()
IsValid            : True
Identity           :
ObjectState        : New

And when AutoDiscover was working (as it was random) I would get this:

RunspaceId         : a711bdd3-b6a1-4fb8-96b8-f669239ea534
Result             : Failed
Message            : The ExchangeRemote endpoint settings could not be determined from the autodiscover response. No
MRSProxy was found running at ‘outlook.domain.com’.
ConnectionSettings :
SupportsCutover    : False
ErrorDetail        : internal error:Microsoft.Exchange.Migration.MigrationRemoteEndpointSettingsCouldNotBeAutodiscovere
dException: The ExchangeRemote endpoint settings could not be determined from the autodiscover
response. No MRSProxy was found running at ‘outlook.domain.com’. —>
Microsoft.Exchange.Migration.MigrationServerConnectionFailedException: The connection to the
server ‘outlook.domain.com’ could not be completed. —>
Microsoft.Exchange.MailboxReplicationService.RemoteTransientException: The call to
https://outlook.domain.com/EWS/mrsproxy.svc’ failed. Error details: The HTTP request is
unauthorized with client authentication scheme ‘Negotiate’. The authentication header received
from the server was ‘Basic Realm=”outlook.domain.com”‘. –> The remote server returned an
error: (401) Unauthorized.. —>
Microsoft.Exchange.MailboxReplicationService.RemotePermanentException: The HTTP request is
unauthorized with client authentication scheme ‘Negotiate’. The authentication header received
from the server was ‘Basic Realm=”outlook.domain.com”‘. —>
Microsoft.Exchange.MailboxReplicationService.RemotePermanentException: The remote server returned
an error: (401) Unauthorized.
— End of inner exception stack trace —
— End of inner exception stack trace —
at Microsoft.Exchange.MailboxReplicationService.MailboxReplicationServiceFault.<>c__DisplayClas
s1.<ReconstructAndThrow>b__0()
at Microsoft.Exchange.MailboxReplicationService.ExecutionContext.Execute(Action operation)
at Microsoft.Exchange.MailboxReplicationService.MailboxReplicationServiceFault.ReconstructAndTh
row(String serverName, VersionInformation serverVersion)
at Microsoft.Exchange.MailboxReplicationService.WcfClientWithFaultHandling`2.<>c__DisplayClass1
.<CallService>b__0()
at Microsoft.Exchange.Net.WcfClientBase`1.CallService(Action serviceCall, String context)
at Microsoft.Exchange.Migration.MigrationExchangeProxyRpcClient.CanConnectToMrsProxy(Fqdn
serverName, Guid mbxGuid, NetworkCredential credentials, LocalizedException& error)
— End of inner exception stack trace —
at Microsoft.Exchange.Migration.DataAccessLayer.ExchangeRemoteMoveEndpoint.VerifyConnectivity()
at Microsoft.Exchange.Management.Migration.TestMigrationServerAvailability.InternalProcessEndpo
int(Boolean fromAutoDiscover)
— End of inner exception stack trace —
IsValid            : True
Identity           :
ObjectState        : New

This was returning the URL https://outlook.domain.com/EWS/mrsproxy.svc’ which is not correct for this mailbox (this was the OA endpoint in a different datacentre) and external Outlook access is not allowed at this company and so the TMG server in front of the F5 load balancer in the NY datacentre was not configured for OA anyway and browsing the the above URL returned the following picture, which is a well broken scenario but not the issue at hand here!

image

If OA (Outlook Anywhere) was available for this company, this is not what I would expect to see when browsing to the External EWS URL. To that end we have EWS URL’s are bypass TMG and go direct to the load balancer.

So now we have either no valid AutoDiscover response or EWS using the wrong URL. Back to the version of the cmdlet Michael was using as that ignores AutoDiscover: Test-MigrationServerAvailability –ExchangeRemoteMove –RemoteServer servername.domain.com

RunspaceId         : 5874c796-54ce-420f-950b-1d300cf0a64a
Result             : Failed
Message            : The connection to the server ‘ewsinukdatacentre.domain.com’ could not be completed.
ConnectionSettings :
SupportsCutover    : False
ErrorDetail        : Microsoft.Exchange.Migration.MigrationServerConnectionFailedException: The connection to the
server ‘ewsinukdatacentre.domain.com’ could not be completed. —>
Microsoft.Exchange.MailboxReplicationService.RemoteTransientException: The call to
https://ewsinukdatacentre.domain.com/EWS/mrsproxy.svc’ failed. Error details: The remote server returned
an unexpected response: (501) Invalid Request. –> The remote server returned an error: (501) Not
                     Implemented.. —> Microsoft.Exchange.MailboxReplicationService.RemotePermanentException: The
remote server returned an unexpected response: (501) Invalid Request. —>
Microsoft.Exchange.MailboxReplicationService.RemotePermanentException: The remote server returned
an error: (501) Not Implemented.
— End of inner exception stack trace —
— End of inner exception stack trace —
at Microsoft.Exchange.MailboxReplicationService.MailboxReplicationServiceFault.<>c__DisplayClas
s1.<ReconstructAndThrow>b__0()
at Microsoft.Exchange.MailboxReplicationService.ExecutionContext.Execute(Action operation)
at Microsoft.Exchange.MailboxReplicationService.MailboxReplicationServiceFault.ReconstructAndTh
row(String serverName, VersionInformation serverVersion)
at Microsoft.Exchange.MailboxReplicationService.WcfClientWithFaultHandling`2.<>c__DisplayClass1
.<CallService>b__0()
at Microsoft.Exchange.Net.WcfClientBase`1.CallService(Action serviceCall, String context)
at Microsoft.Exchange.Migration.MigrationExchangeProxyRpcClient.CanConnectToMrsProxy(Fqdn
serverName, Guid mbxGuid, NetworkCredential credentials, LocalizedException& error)
— End of inner exception stack trace —
at Microsoft.Exchange.Migration.DataAccessLayer.ExchangeRemoteMoveEndpoint.VerifyConnectivity()
at Microsoft.Exchange.Management.Migration.TestMigrationServerAvailability.InternalProcessEndpo
int(Boolean fromAutoDiscover)
IsValid            : True
Identity           :
ObjectState        : New

Now we can see the 501 error that exrca was returning. It would seem that the 501 is coming from the Kemp and not from the endpoint servers, which is why I could not located it in IIS or EWS logs and so in the Kemp System Message File (logging options > system log files) I found the 501 error:

kernel: L7: badrequest-client_read [157.56.251.92:61541->192.168.1.2:443] (-501): <s:Envelope ? , 0 [hlen 1270, nhdrs 8]

Where the first IP address was a Microsoft datacentre IP and the second was the Kemp listener IP.

It turns out this is due to the Kemp load balancer not returning to Microsoft the Continue-100 status that it should get. Microsoft waits 5 seconds and then sends the data it would have done if it got the response back. At this point the Kemp blocks this data with a 501 error.

It is possible to turn off Kemp’s processing of Continue-100 HTTP packets so it lets them through and this is covered in http://blog.masteringmsuc.com/2013/10/kemp-load-balancer-and-lync-unified.html:

 

In the version at my clients which was upgraded to version 7.0-16 from a v5 to v6 to v7 it was defaulting to RFC Conformant, but needs to be Ignore Continue-100 to work with Office 365. In versions of Kemp 7.1 and later the value needs to be changed to RFC-7231 Compliant from the default of “RFC-2616 Compliant”. Now EWS works, hybrid mailbox moves work, and AutoDiscover is always working on that server – so a mix of issues caused by differing interpretations of an RFC. To cover all these issues the Kemp load balancer started to include this 100-Continue Handling setting. We as ITPro’s need to ensure that we set the correct setting for our environment based on the software we use.

Configuring Exchange On-Premises to Use Azure Rights Management

Posted on 7 CommentsPosted in 2010, 2013, 64 bit, aadrm, ADFS, ADFS 2.0, DLP, DNS, exchange, exchange online, https, hybrid, IAmMEC, load balancer, loadbalancer, mcm, mcsm, MVP, Office 365, powershell, rms, sharepoint, warm

This article is the fifth in a series of posts looking at Microsoft’s new Rights Management product set. In an earlier previous post we looked at turning on the feature in Office 365 and in this post we will look at enabling on-premises Exchange Servers to use this cloud based RMS server. This means your cloud users and your on-premises users can shared encrypted content and as it is cloud based, you can send encrypted content to anyone even if you are not using an Office 365 mailbox.

In this series of articles we will look at the following:

The items above will get lit up as the articles are released – so check back or leave a comment to the first post in the series and I will let you know when new content is added.

Exchange Server integrates very nicely with on-premises RMS servers. To integrate Exchange on-premises with Windows Azure Rights Management you need to install a small service online that can connect Exchange on-premises to the cloud RMS service. On-premises file servers (classification) and SharePoint can also use this service to integrate themselves with cloud RMS.

You install this small service on-premises on servers that run Windows Server 2012 R2, Windows Server 2012, or Windows Server 2008 R2. After you install and configure the connector, it acts as a communications interface between the on-premises IRM-enabled servers and the cloud service. The service can be downloaded from http://www.microsoft.com/en-us/download/details.aspx?id=40839

From this download link there are three files to get onto the server you are going to use for the connector.

  • RMSConnectorSetup.exe (the connector server software)
  • GenConnectorConfig.ps1 (this automates the configuration of registry settings on your Exchange and SharePoint servers)
  • RMSConnectorAdminToolSetup_x86.exe (needed if you want to configure the connector from a 32bit client)

Once you have all this software (or that which you need) and you install it then IT and users can easily protect documents and pictures both inside your organization and outside, without having to install additional infrastructure or establish trust relationships with other organizations.

The overview of the structure of the link between on-premises and Windows Azure Rights Management is as follows:

IC721938

Notice therefore that there are some prerequisites needed. You need to have an Office 365 tenant and turn on Windows Azure Rights Management. Once you have this done you need the following:

  • Get your Office 365 tenant up and running
  • Configure Directory Synchronization between on-premises Active Directory and Windows Azure Active Directory (the Office 365 DirSync tool)
  • It is also recommended (but not required) to enable ADFS for Office 365 to avoid having to login to Windows Azure Rights Management when creating or opening protected content.
  • Install the connector
  • Prepare credentials for configuring the software.
  • Authorising the server for connecting to the service
  • Configuring load balancing to make this a highly available service
  • Configuring Exchange Server on-premises to use the connector

Installing the Connector Service

  1. You need to set up an RMS administrator. This administrator is either the a specific user object in Office 365 or all the members of a security group in Office 365.
    1. To do this start PowerShell and connect to the cloud RMS service by typing Import-Module aadrm and then Connect-AadrmService.
    2. Enter your Office 365 global administrator username and password
    3. Run Add-AadrmRoleBasedAdministrator -EmailAddress <email address> -Role “GlobalAdministrator” or Add-AadrmRoleBasedAdministrator -SecurityGroupDisplayName <group Name> -Role “ConnectorAdministrator”. If the administrator object does not have an email address then you can lookup the ObjectID in Get-MSOLUser and use that instead of the email address.
  2. Create a namespace for the connector on any DNS namespace that you own. This namespace needs to be reachable from your on-premises servers, so it could be your .local etc. AD domain namespace. For example rmsconnector.contoso.local and an IP address of the connector server or load balancer VIP that you will use for the connector.
  3. Run RMSConnectorSetup.exe on the server you wish to have as the service endpoint on premises. If you are going to make a highly available solutions, then this software needs installing on multiple machines and can be installed in parallel. Install a single RMS connector (potentially consisting of multiple servers for high availability) per Windows Azure RMS tenant. Unlike Active Directory RMS, you do not have to install an RMS connector in each forest. Select to install the software on this computer:
    IC001
  4. Read and accept the licence agreement!
  5. Enter your RMS administrator credentials as configured in the first step.
  6. Click Next to prepare the cloud for the installation of the connector.
  7. Once the cloud is ready, click Install. During the RMS installation process, all prerequisite software is validated and installed, Internet Information Services (IIS) is installed if not already present, and the connector software is installed and configured
    IC002
  8. If this is the last server that you are installing the connector service on (or the first if you are not building a highly available solution) then select Launch connector administrator console to authorize servers. If you are planning on installing more servers, do them now rather than authorising servers:
    IC003
  9. To validate the connector quickly, connect to http://<connectoraddress>/_wmcs/certification/servercertification.asmx, replacing <connectoraddress> with the server address or name that has the RMS connector installed. A successful connection displays a ServerCertificationWebService page.
  10. For and Exchange Server organization or SharePoint farm it is recommended to create a security group (one for each) that contains the security objects that Exchange or SharePoint is. This way the servers all get the rights needed for RMS with the minimal of administration interaction. Adding servers individually rather than to the group results in the same outcome, it just requires you to do more work. It is important that you authorize the correct object. For a server to use the connector, the account that runs the on-premises service (for example, Exchange or SharePoint) must be selected for authorization. For example, if the service is running as a configured service account, add the name of that service account to the list. If the service is running as Local System, add the name of the computer object (for example, SERVERNAME$).
    1. For servers that run Exchange: You must specify a security group and you can use the default group (DOMAIN\Exchange Servers) that Exchange automatically creates and maintains of all Exchange servers in the forest.
    2. For SharePoint you can use the SERVERNAME$ object, but the recommendation configuration is to run SharePoint by using a manually configured service account. For the steps for this see http://technet.microsoft.com/en-us/library/dn375964.aspx.
    3. For file servers that use File Classification Infrastructure, the associated services run as the Local System account, so you must authorize the computer account for the file servers (for example, SERVERNAME$) or a group that contains those computer accounts.
  11. Add all the required groups (or servers) to the authorization dialog and then click close. For Exchange Servers, they will get SuperUser rights to RMS (to decrypt content):
    image
    image
  12. If you are using a load balancer, then add all the IP addresses of the connector servers to the load balancer under a new virtual IP and publish it for TCP port 80 (and 443 if you want to configure it to use certificates) and equally distribute the data across all the servers. No affinity is required. Add a health check for the success of a HTTP or HTTPS connection to http://<connectoraddress>/_wmcs/certification/servercertification.asmx so that the load balancer fails over correctly in the event of connector server failure.
  13. To use SSL (HTTPS) to connect to the connector server, on each server that runs the RMS connector, install a server authentication certificate that contains the name that you will use for the connector. For example, if your RMS connector name that you defined in DNS is rmsconnector.contoso.com, deploy a server authentication certificate that contains rmsconnector.contoso.com in the certificate subject as the common name. Or, specify rmsconnector.contoso.com in the certificate alternative name as the DNS value. The certificate does not have to include the name of the server. Then in IIS, bind this certificate to the Default Web Site.
  14. Note that any certificate chains or CRL’s for the certificates in use must be reachable.
  15. If you use proxy servers to reach the internet then see http://technet.microsoft.com/en-us/library/dn375964.aspx for steps on configuring the connector servers to reach the Windows Azure Rights Management cloud via a proxy server.
  16. Finally you need to configure the Exchange or SharePoint servers on premises to use Windows Azure Active Directory via the newly installed connector.
    • To do this you can either download and run GenConnectorConfig.ps1 on the server you want to configure or use the same tool to generate Group Policy script or a registry key script that can be used to deploy across multiple servers.
    • Just run the tool and at the prompt enter the URL that you have configured in DNS for the connector followed by the parameter to make the local registry settings or the registry files or the GPO import file. Enter either http:// or https:// in front of the URL depending upon whether or not SSL is in use of the connectors IIS website.
    • For example .\GenConnectorConfig.ps1 –ConnectorUri http://rmsconnector.contoso.com -SetExchange2013 will configure a local Exchange 2013 server
  17. If you have lots of servers to configure then run the script with –CreateRegEditFiles or –CreateGPOScript along with –ConnectorUri. This will make five reg files (for Exchange 2010 or 2013, SharePoint 2010 or 2013 and the File Classification service). For the GPO option it will make one GPO import script.
  18. Note that the connector can only be used by Exchange Server 2010 SP3 RU2 or later or Exchange 2013 CU3 or later. The OS on the server also needs to be include a version of the RMS client that supports RMS Cryptographic Mode 2. This is Windows Server 2008 + KB2627272 or Windows Server 2008 R2 + KB2627273 or Windows Server 2012 or Windows Server 2012 R2.
  19. For Exchange Server you need to manually enable IRM as you would do if you had an on-premises RMS server. This is covered in http://technet.microsoft.com/en-us/library/dd351212.aspx but in brief you run Set-IRMConfiguration -InternalLicensingEnabled $true. The rest, such as transport rules and OWA and search configuration is covered in the mentioned TechNet article.
  20. Finally you can test if RMS is working with Test-IRMConfiguration –Sender billy@contoso.com. You should get a message at the end of the test saying Pass.
  21. If you have downloaded GenConnectorConfig.ps1 before May 1st 2014 then download it again, as the version before this date writes the registry keys incorrectly and you get errors such as “FAIL: Failed to verify RMS version. IRM features require AD RMS on Windows Server 2008 SP2 with the hotfixes specified in Knowledge Base article 973247” and “Microsoft.Exchange.Security.RightsManagement.RightsManagementException: Failed to get Server Info from http://rmsconnector.contoso.com/_wmcs/certification/server.asmx. —> System.Net.WebException: The request failed with HTTP status 401: Unauthorized.”. If you get these then turn of IRM, delete the “C:\ProgramData\Microsoft\DRM\Server” folder to remove old licences, delete the registry keys and run the latest version of GetConnectorConfig.ps1, refresh the RMS keys with Set-IRMConfiguration –RefreshServerCertificates and reset IIS with IISRESET.

Now you can encrypt messages on-premises using your AADRM licence and so not require RMS Server deployed locally.

An “Inexpensive” Exchange Lab In Azure

Posted on Leave a commentPosted in 2010, 2013, Azure, cloud, DNS, exchange, exchange online, hyper-v, IAmMEC, Office 365, vhd, vm, vpn

This blog post centres around two scripts that can be used to quickly provision an Exchange Server lab in Azure and then to remove it again. The reason why the blog post is titled “inexpensive” is that Azure charges compute hours even if the virtual machines are shut down. Therefore to make my Exchange lab cheaper to operate and to not charge me when the lab is not being used, I took my already provisioned VHD files and created a few scripts to create the virtual machines and cloud service and then to remove it again if needed.

Before you start using these scripts, you need to have already uploaded or created your own VHD’s in Azure and designed your lab as you need. These scripts will then take a CSV file with the relevant values in them and create a VM for each VHD in the correct subnet (that you have also created in Azure) and always in the correct order – thus ensuring they always get the same IP address from your virtual network (UPDATE: 14 March 2014 – Thanks to Bhargav, this script now reserves the IPs as well as this is a newish feature in Azure). Without reserving an IP, when you boot your domain controller first in each subnet it will always get the fourth available IP address. This IP is the DNS IP address in Azure and then each of the other machines are created and booted in the order of your choosing and so get the subsequent IP’s. Azure never used to guarantee the IP but updates in Feb 2014 now allow this with the latest Azure PowerShell cmdlets. This way we can ensure the private IP is always the same and machine dependancies such as domain controllers running first are adhered to.

These scripts are created in PowerShell and call the Windows Azure PowerShell cmdlets. You need to install the Azure cmdlets on your computer and these scripts rely on features found in version 0.7.3.1 or later. You can install the cmdlets from http://www.windowsazure.com/en-us/documentation/articles/install-configure-powershell/

Build-AzureExchangeLab.ps1

# Retrieve with Get-AzureSubscription 
$subscriptionName = "Visual Studio Premium with MSDN"

Import-AzurePublishSettingsFile 'downloaded.publishsettings.file.got.with.Get-AzurePublishSettingsFile'

# Select the subscription to work on if you have more than one subscription
Select-AzureSubscription -SubscriptionName $subscriptionName

# Name of Virtual Network to add VM's to
$VMNetName = "MCMHybrid"

# CSV File with following columns (BringOnline,VMName,StorageAccount,VMOSDiskName,VMInstanceSize,SubnetName,IPAddress,Location,AffinityGroup,WaitForBoot,PublicRDPPort)
)
$CSVFile = Import-CSV 'path\filename.csv'

# Loop to build lab here. Ultimately get values from CSV file
foreach ($VMItem in $CSVFile) {

    # Retrieve with Get-AzureStorageAccount  
    $StorageAccount = $VMItem.StorageAccount

    # Specify the storage account location containing the VHDs 
    Set-AzureSubscription -SubscriptionName $subscriptionName  -CurrentStorageAccount $StorageAccount
  
    # Not Used $location = $VMItem.Location     # Retrieve with Get-AzureLocation

    # Specify the subnet to use. Retreive with Get-AzureVNetSite | FL Subnets
    $subnetName = $VMItem.SubnetName

    $AffinityGroup = $VMItem.AffinityGroup      # From Get-AzureAffinityGroup (for association with a private network you have already created). 

    $VMName = $VMItem.VMName
    $VMOSDiskName = $VMItem.VMOSDiskName        # From Get-AzureDisk
    $VMInstanceSize = $VMItem.VMInstanceSize    # ExtraSmall, Small, Medium, Large, ExtraLarge 
    $CloudServiceName = $VMName
    $IPAddress = $VMItem.IPAddress              # Reserves a specific IP for the VM
    if ($VMItem.BringOnline -eq "Yes") {
        Write-Host "Creating VM: " $VMName
        $NewVM = New-AzureVMConfig -Name $VMName -DiskName $VMOSDiskName -InstanceSize $VMInstanceSize | Add-AzureEndpoint -Name 'Remote Desktop' -LocalPort 3389 -PublicPort $VMItem.PublicRDPPort -Protocol tcp | Add-AzureEndpoint -Protocol tcp -LocalPort 25 -PublicPort 25 -Name 'SMTP' | Add-AzureEndpoint -Protocol tcp -LocalPort 443 -PublicPort 443 -Name 'SSL' | Add-AzureEndpoint -Protocol tcp -LocalPort 80 -PublicPort 80 -Name 'HTTP' | Set-AzureSubnet –SubnetNames $subnetName | Set-AzureStaticVNetIP –IPAddress $IPAddress        
        # Creates new VM and waits for it to boot if required
        if ($VMItem.WaitForBoot -eq "Yes") {New-AzureVM -ServiceName $CloudServiceName -AffinityGroup $AffinityGroup -VMs $NewVM -VNetName $VMNetName -WaitForBoot}
            else {New-AzureVM -ServiceName $CloudServiceName -AffinityGroup $AffinityGroup -VMs $NewVM -VNetName $VMNetName }
    }
}

Remove-AzureExchangeLab.ps1

# Retrieve with Get-AzureSubscription 
$subscriptionName = "Visual Studio Premium with MSDN"

Import-AzurePublishSettingsFile 'downloaded.publishsettings.file.got.with.Get-AzurePublishSettingsFile'

# Select the subscription to work on if you have more than one subscription
Select-AzureSubscription -SubscriptionName $subscriptionName

# CSV File with following columns (BringOnline,VMName,StorageAccount,VMOSDiskName,VMInstanceSize,SubnetName,IPAddress,Location,AffinityGroup,WaitForBoot,PublicRDPPort)
$CSVFile = Import-CSV 'path\filename.csv'

# Loop to build lab here. Ultimately get values from CSV file
foreach ($VMItem in $CSVFile) {

    # Stop VM
    Stop-AzureVM -Name $VMItem.VMName -ServiceName $VMItem.VMName -Force

    # Remove VM but leave VHDs behind
    Remove-AzureVM -ServiceName $VMItem.VMName -Name $VMItem.VMName 

    # Remove Cloud Service
    Remove-AzureService $VMItem.VMName -Force
}

CSV File Format

The CSV file has a row per virtual machine, listed in order that the machine is booted:

BringOnline,VMName,StorageAccount,VMOSDiskName,VMInstanceSize,SubnetName,IPAddress,Location,AffinityGroup,WaitForBoot,PublicRDPPort
Yes,mh-oxf-dc1,portalvhdsjv47jtq9qdrmb,mh-oxf-mbx2-mh-oxf-mbx2-0-201312301745030496,Small,Oxford,10.0.0.4,West Europe,C7Solutions-AG,Yes,3389
etc,…

The columns are as follows:

  • BringOnline: Yes or No
  • VMName: This name is used for the VM and the Cloud Service. It must be unique within Azure. An example might be EX-LAB-01 (if that is unique that is)
  • StorageAccount: The name of the storage account that the VHD is stored in. This might be one you created yourself or one made by Azure with a name containing random letters. For example portalvhdshr4djwe9dwcb5 would be what this value might look like. Use Get-AzureStorageAccount to find this value.
  • VMOSDiskName: This is the disk name (not the file name) and is retrieved with Get-AzureDisk
  • VMInstanceSize: ExtraSmall, Small, Medium, Large or ExtraLarge
  • SubnetName: Get with Get-AzureVNetSite | FL Subnets
  • IPAddress: Sets a specific IP address for the VM. VM will always get this IP when it boots and other VM’s will not take it if the happen to boot before it
  • Location: Retrieve with Get-AzureLocation. This value is not used in the script as I use Affinity Groups and subnets instead.
  • AffinityGroup: From Get-AzureAffinityGroup (for association with a private network you have already created).
  • WaitForBoot: Yes or No. This will wait for the VM to come online (and thus get an IP correctly provisioned in order) or ensure things like the domain controller is running first.
  • PublicRDPPort: Set to 3389 unless you want to use a different port. For simplicity, the script sets ports 443, 80 and 25 as open on the IP addresses of the VM

Highly Available Office 365 to On-Premises Mail Routing

Posted on 20 CommentsPosted in 2010, 2013, cloud, DNS, EOP, exchange, exchange online, Exchange Online Protection, hybrid, IAmMEC, MX, Office 365, smarthost, smtp

This article looks at how to configure mail flow from Office 365 (via Exchange Online Protection – EOP) to your On Premises organization to ensure that it is highly available and work in disaster recovery scenarios with no impact. It is based on exactly the same principle to that which I blogged about in 2012: http://c7solutions.com/2012/05/highly-available-geo-redundancy-with-html on creating redundant outbound connections from Exchange on premises.

The best way to explain this feature is to describe it in the way of an example:

For example MCMEmail Ltd have Hybrid set up, and delivery to the cloud first. So the DNS zone for mcmemail.co.uk has MX pointing to EOP.

They then create a new DNS zone at either a subzone (as in this example) or a different domain if they have one available. In the example this could be hybrid.mcmemail.co.uk. Into this zone they add the following records:

10 MX oxford-a.hybrid.mcmemail.co.uk

10 MX oxford-b.hybrid.mcmemail.co.uk

20 MX nuneaton.hybrid.mcmemail.co.uk

The below picture shows an example of this configured in AWS Route 53 DNS (though there are other DNS providers available)

image

In Exchange Online Protection administration pages (Office 365 Portal > Exchange Admin > Mail Flow > Connectors and modify your on-premises connector to point to the new zone. Example shown in the below picture:

image

Then all email is always delivered to the Oxford datacentre and nothing to the Nuneaton one (where the DR servers reside) unless the two Oxford datacentres (A and B) are both offline and so the 10 preference does not answer at all. At that time and that time only does the 20 preference get connected to.