Duplicate Exchange Online and Exchange Server Mailboxes

Posted on 4 CommentsPosted in duplicate, EOP, exchange, exchange online, Exchange Online Protection, Exchange Server, mailbox, MX, Office 365

With a hybrid Exchange Online deployment, where you have Exchange Server on-premises and Exchange Online configured in the cloud, and utilising AADConnect to synchronize the directory, you should never find that a synced user object is configured as both a mailbox in the cloud and a mailbox on-premises.

When Active Directory is synced to Azure Active Directory, the ExchangeGUID attribute for the on-premises user is synced to the cloud (assuming that you have not do a limited attribute sync and not synced the Exchange attributes – as that is required for Exchange Online hybrid). The Exchange Online directory takes a sync of information relating to Exchange from Azure Active Directory (Azure AD), which is known as forward sync. This ensures that the ExchangeGUID attribute from the on-premises mailbox is synced into your Exchange Online directory.

When a user is given an Exchange Online licence, it becomes the job of Exchange Online to provision a mailbox for this user. When Exchange Online needs to provision a new mailbox, it will not do so where the ExchangeGUID attribute already exists. The existence of this attribute tells the provisioning process that the mailbox already exists on-premises and will be migrated here later and so not to create a conflicting mailbox. A cloud user who does not have an ExchangeGUID attribute synced from on-premises will get a mailbox created by the Exchange Online provisioning process upon a licence being assigned, and on-premises users that do not have a mailbox on-premises (who also have no ExchangeGUID attribute) will also find that granting them an Exchange Online licence will trigger the creation of a mailbox for them.

This is all well and good, and the above is what happens in most cases. But there is an edge case where an on-premises user with a mailbox (and therefore has the ExchangeGUID attribute populated) will also get a mailbox in Exchange Online. This happens where the organization manually created cloud mailboxes before enabling AADConnect to sync the directories, and these cloud users match the on-premises user by UserPrincipalName and they are given an Exchange Online licence.

In this above case, because they are cloud users with an Exchange Online licence they get a mailbox. Deleting the cloud user and then enabling sync will cause the original mailbox to be restored to the user account as the UserPrincipalName matches.

For example, the below shows a user being created in the cloud called “twomailboxes@domain.com”:

image

The user is granted a full Office 365 E3 licence, so this means the user has an Exchange Online mailbox. There is no AADConnect sync in place, but the UPN matches a user on-premises who has a mailbox.

In Exchange Online PowerShell, once the mailbox is provisioned we can see the following:

image

PS> get-mailbox twomailboxes | fl name,userprincipalname,exchangeguid



Name              : twomailboxes
UserPrincipalName : twomailboxes@domain.com
ExchangeGuid      : d893372b-bfe0-4833-9905-eb497bb81de4

Repeating the same on-premises will show a separate user (remember, no AD sync in place at this time) with the same UPN and a different ExchangeGUID.

image

[PS] >get-mailbox twomailboxes | fl name,userprincipalname,exchangeguid



Name              : Two Mailboxes
UserPrincipalName : twomailboxes@cwh.org.uk
ExchangeGuid      : 625d70aa-82ed-47a2-afa2-d3c091d149aa

Note that the on-premises object ExchangeGUID is not the same as the cloud ExchangeGUID. This is because there are two seperate mailboxes.

Get-User in the cloud will also report something useful. It will show the “PreviousRecipientTypeDetails” value, which is not modifiable by the administrator, in this case shows there was not a previous mailbox for the user but this can show that a previous mailbox did exist. For completion we also show the licence state:

image

PS > get-user twomailboxes | fl name,recipienttype,previousrecipienttypedetails,*sku*



Name                         : twomailboxes
RecipientType                : UserMailbox
PreviousRecipientTypeDetails : None
SKUAssigned                  : True

Now in preparation for the sync of the Active Directory to Azure Active Directory, the user accounts in the cloud are either left in place (and so sync will do a soft-match for those users) or they are deleted and the on-premises user account syncs to the cloud. In the first case, the clash on the sync will result in the cloud mailbox being merged into the settings from the on-premises mailbox. In the second case, there is no user account to merge into, but there is a mailbox to restore against this user. And even though the newly synced user has an ExchangeGUID attribute on-premises that is synced to the cloud, and they have a valid licence, Exchange Online reattaches the old mailbox associated with a different GUID.

The impact of this is minor to massive. In the scenario where MX points to on-premises and you have not yet moved any mailboxes to the cloud, this cloud mailbox will only get email from other cloud mailboxes in your tenant (there are none in this scenario) or internal alerts in Office 365 (and these are reducing over time, as they start to follow correct routing). It can be a major issue though if you use MX to Exchange Online Protection. As there is now a mailbox in the cloud for a user on-premises, inbound internet sourced email for your on-premises user will get delivered to the cloud mailbox and not appear on-premises. Where the invalid mailbox has no email, recovery is not required. Finally, where there is a duplicate mailbox, move requests for those users for onboarding to Exchange Online will fail:

image

This reads “a subscription for the migration user <email> couldn’t be loaded”. This occurs where the user was not licenced and so there was not a duplicate mailbox in the cloud, but the user was later licenced before the migration completed.

Where the invalid duplicate mailbox exists in the cloud and is getting valid emails delivered to it, the recovery work described below additionally will involve exporting email from this invalid mailbox and then removing the mailbox as part of the fix. Extraction of email from the duplicate mailbox needs to happen before the licence is removed.

To remove the cloud mailbox and to stop it being recreated, you need to ensure that the synced user does not have an Exchange Online licence. You can grant them other licences in Office 365, but not Exchange Online. I have noticed that if you do licencing via Azure AD group based licencing rules then this will also fail (these are still in preview at time of writing) and that you need to ensure that the user is assigned the licence directly in the Office 365 portal and that they do not get the Exchange Online licence. After licence reconciliation in the cloud occurs (a few minutes typically) the duplicate mailbox is removed (though I have seen this take a few hours). The Get-User cmdlet above will show the RecipientType being a MailUser and not Mailbox.

You are now in a position where your duplicate cloud mailbox is gone (which is why if that mailbox had been a target to valid emails before now, you would need to have extracted the data via discovery and search processes first).

Running the above Get-User and Get-Mailbox (and now Get-MailUser) cmdlets in the cloud will show you that the ExchangeGUID on the cloud object now matches the on-premises object and the duplication is gone. You can now migrate that mailbox to the cloud successfully.

We can take a look at what we see in remote PowerShell here:

Recall from above that there were two different ExchangeGUIDs, one in the cloud and one on-premises. These in my example where:

Cloud duplicate ExchangeGuid      : d893372b-bfe0-4833-9905-eb497bb81de4

On-premises mailbox ExchangeGuid  : 625d70aa-82ed-47a2-afa2-d3c091d149aa

Get-User before licences removed in the cloud, showing a mailbox and that it was previously a mailbox as well:

image

PS > get-user twomailboxes | fl name,recipienttype,previousrecipienttypedetails,*sku*



Name                         : Two Mailboxes
RecipientType                : UserMailbox
PreviousRecipientTypeDetails : UserMailbox
SKUAssigned                  : True

Get-Mailbox in the cloud showing the GUID was different from on-premises:

image

PS > get-mailbox twomailboxes | fl name,userprincipalname,exchangeguid



Name              : Two Mailboxes
UserPrincipalName :
twomailboxes@domain.com

ExchangeGuid      : d893372b-bfe0-4833-9905-eb497bb81de4

Once the licence is removed in Office 365 for Exchange Online and licence reconciliation completes (SKUAssigned is False) you will see that Get-User shows it is not a mailbox anymore:

image

PS > get-user twomailboxes | fl name,recipienttype,previousrecipienttypedetails,*sku*



Name                         : Two Mailboxes
RecipientType                : MailUser
PreviousRecipientTypeDetails : UserMailbox
SKUAssigned                  : False

And finally Get-MailUser (not Get-Mailbox now) shows the ExchangeGUID matches the on-premises, synced, ExchangeGUID value:

image

PS > get-mailuser twomailboxes | fl name,userprincipalname,exchangeguid



Name              : Two Mailboxes
UserPrincipalName : twomailboxes@domain.com
ExchangeGuid      : 625d70aa-82ed-47a2-afa2-d3c091d149aa

Note that giving these users back their Exchange Online licence will revert all of the above and restore their old mailbox. As these users cannot have an Exchange Online licence assigned in the cloud, for risk of their old mailbox returning you need to ensure that within 30 days of their on-premises mailbox being migrated to the cloud you do give then an Exchange Online mailbox. Giving them a licence after migration of their on-premises mailbox to the cloud will ensure their single, migrated, mailbox remains in Exchange Online. But giving their user a licence before migration will restore their old cloud mailbox.

For users that never had a matching UPN in the cloud and a cloud mailbox, you can licence them before you migrate their mailbox as they will work correctly within the provisioning system in Exchange Online.

Is Your SenderID/SPF or DKIM Record Correctly Configured

Posted on Leave a commentPosted in DNS, domain, exchange, exchange online, IAmMEC, MX, Office 365, spam

With Microsoft having just announced that DKIM is coming to Office 365 soon (release notes here) and SenderID is already available, I thought this is a good time to write a blog on the use of DMARC to show if your records are correct.

DMARC is a protocol that allows you to see the effect of your SenderID/SPF records from the viewpoint of the recipient – as long as the recipient is one of the larger email receivers in the world, but that should be enough to help you validate your SenderID/SPF records. DMARC is currently in use at Outlook.com/Hotmail, Gmail, Facebook, Yahoo and Twitter to name some of the larger users of it.

To receive the results of a DMARC report you need to add your DMARC policy as a new TXT record to DNS. It looks something like this:

_dmarc IN TXT “v=DMARC1;p=reject;pct=100;rua=mailto:postmaster@dmarcdomain.com”

This TXT record for “_dmarc” at the root of your domain tells the receiver to report back to you all (pct=100) SenderID/SPF failures. They also report DKIM failures as well. If you send lots of email you might want to report on a smaller number of overall failures, say pct=20! The record says who to send the report to (rua=email address).

Finally, and one of the most useful bits of DMARC is that you can tell the receiver what to do with failures in your SenderID/SFP or DKIM record. For example in SenderID/SPF you can set the modifier “-all” to the end. This says that anything not covered in the rest of the record is to fail SenderID. The recipient then decides what to do with your email (as it might be your email, as you might have got your SPF or DKIM record wrong). In the example above you tell the recipient email system you would like them to reject (p=) the email. The policy could be quarantine or none. The policy p=none means accept the message, but report it back to the rua email address in the DMARC record.

p=none allows you to introduce SPF/DKIM and ensure no rejection of your messages at the recipient end, but get reporting back on the IP addresses from which your emails (or spoofed emails) are coming from.

Once you are sure your SPF/DKIM records are correct change your policy to p=quarantine and then when you are finally sure, change it to p=reject.

An example of creating a DMARC policy on your domain is shown below. The below example is AWS Route 53 DNS, but any DNS management console that support TXT records should work:

image

Finally, as this is just a quick intro to DMARC, you can use the ruf= qualifier to set the email address of per message failure reports.

Highly Available Office 365 to On-Premises Mail Routing

Posted on 22 CommentsPosted in 2010, 2013, cloud, DNS, EOP, exchange, exchange online, Exchange Online Protection, hybrid, IAmMEC, MX, Office 365, smarthost, smtp

This article looks at how to configure mail flow from Office 365 (via Exchange Online Protection – EOP) to your On Premises organization to ensure that it is highly available and work in disaster recovery scenarios with no impact. It is based on exactly the same principle to that which I blogged about in 2012: http://c7solutions.com/2012/05/highly-available-geo-redundancy-with-html on creating redundant outbound connections from Exchange on premises.

The best way to explain this feature is to describe it in the way of an example:

For example MCMEmail Ltd have Hybrid set up, and delivery to the cloud first. So the DNS zone for mcmemail.co.uk has MX pointing to EOP.

They then create a new DNS zone at either a subzone (as in this example) or a different domain if they have one available. In the example this could be hybrid.mcmemail.co.uk. Into this zone they add the following records:

10 MX oxford-a.hybrid.mcmemail.co.uk

10 MX oxford-b.hybrid.mcmemail.co.uk

20 MX nuneaton.hybrid.mcmemail.co.uk

The below picture shows an example of this configured in AWS Route 53 DNS (though there are other DNS providers available)

image

In Exchange Online Protection administration pages (Office 365 Portal > Exchange Admin > Mail Flow > Connectors and modify your on-premises connector to point to the new zone. Example shown in the below picture:

image

Then all email is always delivered to the Oxford datacentre and nothing to the Nuneaton one (where the DR servers reside) unless the two Oxford datacentres (A and B) are both offline and so the 10 preference does not answer at all. At that time and that time only does the 20 preference get connected to.

Creating GeoDNS with Amazon Route 53 DNS

Posted on 3 CommentsPosted in 2013, cloud, exchange, GeoDNS, https, load balancer, mcm, microsoft, MX, networking, owa, smtp, transport

UPDATE: 13 Aug 2014 – Amazon Route 53 now does native GeoDNS within the product – see Amazon Route 53 GeoDNS Routing Policy

A new feature to Exchange 2013 is supported use of a single namespace for your global email infrastructure. For example mail.contoso.com rather than different ones for each region such as uk-mail.contoso.com; usa-mail.contoso.com and apac-mail.contoso.com.
GeoDNS means that you are given the IP address of a server that is in or close to the region that you are in. For example if you work in London and your mailbox is also in London then most of the time you will want to be connected to the London CAS servers as that gives you the best network response. So in Exchange 2010 you would use your local URL of uk-mail.contoso.com and if you used the others you would be told to use uk-mail.contoso.com. For GeoDNS support you use mail.contoso.com and as you are in the UK you get the IP address of the CAS array in London. When you travel to the US (occasionally) you would get the US CAS array IP address, but this CAS array is able to proxy your OWA, RPC/HTTP etc traffic to the UK mailbox servers.
The same is true for email delivery via SMTP. Email that comes from UK sourced IP addresses is on balance a statistical likelihood that it is going to the UK mailbox. So when you look up the MX record for contoso.com from a UK company you get the UK CAS array and the email gets delivered to the CAS array that is in the same site as the target mailbox. If the email is for a user in a different region and it hits the UK CAS array then it is proxied to the other region seamlessly.
GeoDNS is a feature provided by some high-scale DNS providers, but not something Amazon Web Services (AWS) Route 53 provides – so how do I configure GeoDNS with Amazon Web Services (AWS) Route 53 DNS Service?
Quite easily is the answer. Route 53 does not offer GeoDNS but does offer DNS that directs you towards the closest AWS datacentre. If your datacentres are in regions similar to AWS then the DNS redirection that AWS offers is probably accurate.
To set it up, open your Route 53 DNS console, or move your DNS to AWS (it costs $0.50/month for a zone at time of writing, AWS Route 53 pricing here) and then create your global Exchange 2013 namespace record in DNS:

  1. Click Create Record Set and enter the name. In the below example I’m using geo.c7solutions.com as I don’t actually have a globally distributed email infrastructure!
  2. Select A – IPv4 or if you are doing IPv6 select AAAA.
  3. Set Alias to No and enter the IP address of one of your datacentres
  4. Select the AWS region that is closest to this Exchange server(s) and enter a unique description for the Set ID value.
  5. The entry will look something like this:
    image
  6. Save the Record Set and create additional entries for other regions. For the purposes of this blog I have created geo.c7solutions.com in four regions with the following IP addresses:
    Region IP Address Region
    us-east-1 1.2.3.4 Northern Virginia
    us-west-1 6.7.8.9 Northern California
    eu-west-1 2.3.4.5 Ireland
    ap-northwest-1 3.4.5.6 Singapore
    sa-east-1 4.5.6.7 Sao Paulo
    ap-southeast-1 5.6.7.8 Sydney
  7. The configuration in AWS for the remaining entries looks like the following:
    imageimageimage
  8. And also, once created, it appears like this:
    image

In addition to this blog, I’ve left the record described above on my c7solutions.com DNS zone. So depending upon your location in the world, if you open a command prompt and ping geo.c7solutions.com you should get back the IP address for the AWS region closest to you, and so get back an IP that represents a Exchange resource in your global region. Of course the IP’s I have used are not mine to use and probably will not respond to ping requests – but all you need to do is see it DNS returns the IP above that best matches the region that you are in.
I wrote this blog when in a hotel in Orlando and as you can see from the image below, it returns 1.2.3.4 which is the IP address associated with us-east-1.
image
But when I connected to a server in the UK and did the same ping geo.c7solutions.com I got the following, which show GeoDNS working when equating GeoDNS to AWS Latency DNS.
image
What do you get for your regions? Add comments and let us where you are (approximately) and what region you got. If enough people respond from enough places we can see if AWS can go GeoDNS without massive cost.
[Updated 13 Nov 2012] Added Sydney (ap-southeast-1) and fake IP address of 5.6.7.8
[Updated 27 April 2013] Added Northern California (us-west-1) and fake IP of 6.7.8.9

Highly Available Geo Redundancy with Outbound Send Connectors in Exchange 2003 and Later

Posted on 6 CommentsPosted in 2003, 2007, 2010, cloud, DNS, domain, door, exchange, exchange online, load balancer, loadbalancer, mcm, microsoft, MX, Office 365, smarthost

This is something I’ve been meaning to write down for a while. I wrote an answer for this question to LinkedIn about a week ago and I’ve just emailed a MCM Exchange consultant with this – so here we go…

If you configure a Send Connector (Exchange 2007 and 2010) or Exchange 2003 SMTP Connector with multiple smarthosts for delivery to, then Exchange will round-robin across them all equally. This gives high availability, as if a smarthost is unavailable then Exchange will pick the next one and mail will get delivered, but it does not give redundancy across sites. If you add a smarthost in a remote site to the send connector Exchange will use it in turn equally.

So how can get get geographical redundancy with outbound smarthosts? Quite easily it appears, and it all uses a feature of Exchange that’s been around for a while. But first these important points:

  • This works for smarthost delivery and not MX (i.e. DNS) delivery.
  • This is only useful for companies with multiple sites, internet connections in these sites and smarthosts in those sites.
  • This is typically done on your internet send connectors, the ones using the * address space.

You do this by creating a fake domain in DNS. Lets say smarthost.local and then creating A records in this zone for each SMTP smarthost (i.e. mail.oxford.smarthost.local). Then create an MX record for your first site (oxford.smarthost.local MX 10 mail.oxford.smarthost.local). Repeat for each site, where oxford is the site name of the first site in this example.

Then you create second MX records, lower priority, in any site but use the A record of a smarthost in a different site (oxford.smarthost.local MX 20 mail.cambridge.smarthost.local).

Then add oxford.smarthost.local as the target smarthost in the send connector. Exchange will look up the address in DNS as MX first, A record second, IP address last), so it will find the MX record and resolve the A records for the highest priority for the domain and then round-robin across these A records.

If you have more than one smarthost in a site, add more than one MX 10 record, one per smarthost. Exchange will round-robin across the 10’s. When all the 10’s are offline then Exchange will automatically route to mail.cambridge.smarthost.local (MX priority 20 for the oxford site) without needing to disable the connector and retry the queues.

If you used servernames and not MX’s then it would round-robin amongst all entries, and so equally sent email to Cambridge for delivery. The MX option keeps mail in site for delivery until it cannot and then sends it automatically to the failover site.