Duplicate Exchange Online and Exchange Server Mailboxes


With a hybrid Exchange Online deployment, where you have Exchange Server on-premises and Exchange Online configured in the cloud, and utilising AADConnect to synchronize the directories, you should never find that a synced user object is configured as both a mailbox in Exchange Online and a mailbox on-premises.

When Active Directory is synced to Azure Active Directory, the ExchangeGUID attribute for the on-premises user is synced to the cloud (assuming that you have not done a limited attribute sync and excluded the Exchange attributes from syncing to AAD – as syncing the attributes is required for Exchange Online hybrid). Exchange Online though does not read attributes from Azure Active Directory. Exchange Online reads its attributes from the Exchange Online Directory Service (EXODS). The Exchange Online directory takes a sync of information relating to Exchange from Azure Active Directory (Azure AD), which is known as forward sync. This ensures that the ExchangeGUID attribute from the on-premises mailbox is synced into Exchange Online for your tenant.

When a user is given an Exchange Online licence, it becomes the job of Exchange Online to provision a mailbox for this user. When Exchange Online needs to provision a new mailbox, it will not do so where the ExchangeGUID attribute already exists. The existence of this attribute tells the provisioning process that the mailbox already exists on-premises and may be migrated here later and so not to create a conflicting mailbox. A cloud user who does not have an ExchangeGUID attribute synced from on-premises will get a mailbox created by the Exchange Online provisioning process upon a licence being assigned, and on-premises users that do not have a mailbox on-premises (who also have no ExchangeGUID attribute) will also find that granting them an Exchange Online licence will trigger the creation of a mailbox for them. Note that this last option will create a mailbox in the cloud – but all the attribute management of this mailbox must be done on-premises, as the object syncs from on-premises and so that is the source of the object. Therefore avoid licencing synced objects that do not have a mailbox or remote-mailbox on premises (see my session on this at Microsoft Ignite 2018 “THR2145 – Why do we need to keep an Exchange Server on-premises when we move to the cloud?“)

The above is what happens in most cases – the user on-premises has a ExchangeGUID value, that is synced to the cloud, and then the user is licenced and a second mailbox is not created. But there is an edge case where an on-premises user with a mailbox (and therefore has the ExchangeGUID attribute populated) will also get a mailbox in Exchange Online. This happens where the organization manually created cloud mailboxes before enabling AADConnect to sync the directories, and these cloud users match the on-premises user by UserPrincipalName or primary SMTP address.

In this above case, because they are cloud users with an Exchange Online licence they get a mailbox. Deleting the cloud user and then enabling sync will cause the original cloud mailbox to be restored to the user account as the UserPrincipalName matches.

For example, the below shows a user being created in the cloud called “twomailboxes@domain.com”:

image

The user is granted a full Office 365 E3 licence, so this means the user has an Exchange Online mailbox. There is no AADConnect sync in place, but the UPN matches a user on-premises who has a mailbox.

In Exchange Online PowerShell, once the mailbox is provisioned we can see the following:

image

PS> Get-Mailbox twomailboxes | FL name,userprincipalname,exchangeguid


Name              : twomailboxes
UserPrincipalName : twomailboxes@domain.com
ExchangeGuid      : d893372b-bfe0-4833-9905-eb497bb81de4

Repeating the same on-premises will show a separate user (remember, no AD sync in place at this time) with the same UPN and of course a different ExchangeGUID.

image

[PS] > Get-Mailbox twomailboxes | FL name,userprincipalname,exchangeguid


Name              : Two Mailboxes
UserPrincipalName : twomailboxes@domain.com
ExchangeGuid      : 625d70aa-82ed-47a2-afa2-d3c091d149aa

Note that the on-premises object ExchangeGUID is not the same as the cloud ExchangeGUID. This is because there are two seperate mailboxes.

Get-User in the cloud will also report something useful. It will show the “PreviousRecipientTypeDetails” value, which is not modifiable by the administrator, which in this case shows there was not a previous mailbox for the user but this can show that a previous mailbox did exist. For completion we also show the licence state:

image

PS > Get-User twomailboxes | FL name,recipienttype,previousrecipienttypedetails,*sku*


Name                         : twomailboxes
RecipientType                : UserMailbox
PreviousRecipientTypeDetails : None
SKUAssigned                  : True

Now in preparation for the sync of the Active Directory to Azure Active Directory, the user accounts in the cloud are either left in place (and so sync will do a soft-match for those users) or they are deleted and the on-premises user account syncs to the cloud. In the first case, the clash on the sync will result in the cloud mailbox being merged into the settings from the on-premises mailbox (on-premises values overwrite cloud values unless on-premises value is null or does not exist). In the second case, cloud object deleted before sync, there is no user account to merge into, but there is a mailbox to restore against this user. And even though the newly synced user has an ExchangeGUID attribute on-premises that is synced to the cloud, and they have a valid licence, Exchange Online reattaches the old mailbox associated with a different GUID but same UPN/email address.

The impact of this is minor to massive. In the scenario where MX points to on-premises and you have not yet moved any mailboxes to the cloud, this cloud mailbox will only get email from other cloud mailboxes in your tenant (there are none in this scenario) or internal alerts in Office 365 (and these are reducing over time, as they start to follow correct routing). It can be a major issue though if you use MX to Exchange Online Protection. As there is now a mailbox in the cloud for a user on-premises, inbound internet sourced email for your on-premises user will get delivered to the cloud mailbox and not appear on-premises!

The other problems are that where there is a duplicate mailbox, move requests for those users for onboarding to Exchange Online will fail:

image

This reads “a subscription for the migration user <email> couldn’t be loaded”. This occurs where the user was not licenced and so there was not a duplicate mailbox in the cloud, but the user was later licenced before the migration completed.

Where the invalid duplicate mailbox exists in the cloud and is getting valid emails delivered to it, the recovery work described below additionally will involve exporting email from this invalid mailbox and then removing the mailbox as part of the fix. Extraction of email from the duplicate mailbox needs to happen before the licence is removed.

To remove the cloud mailbox and to stop it being recreated, you need to ensure that the synced user does not have an Exchange Online licence. You can grant them other licences in Office 365, but not Exchange Online. I have noticed that if you do licencing via Azure AD group based licencing rules then this will also fail (these are still in preview at time of writing) and that you need to ensure that the user is assigned the licence directly in the Office 365 portal and that they do not get the Exchange Online licence. After licence reconciliation in the cloud occurs (a few minutes typically) the duplicate mailbox is removed (though I have seen this take a few hours). The Get-User cmdlet above will show the RecipientType being a MailUser and not Mailbox.

You are now in a position where your duplicate cloud mailbox is gone (which is why if that mailbox had been a target to valid emails before now, you would need to have extracted the data via discovery and search processes first).

Running the above Get-User and Get-Mailbox (and now Get-MailUser) cmdlets in the cloud will show you that the ExchangeGUID on the cloud object now matches the on-premises object and the duplication is gone. You can now migrate that mailbox to the cloud successfully.

We can take a look at what we see in remote PowerShell here:

Recall from above that there were two different ExchangeGUIDs, one in the cloud and one on-premises. These in my example where:

Cloud duplicate ExchangeGuid      : d893372b-bfe0-4833-9905-eb497bb81de4

On-premises mailbox ExchangeGuid  : 625d70aa-82ed-47a2-afa2-d3c091d149aa

Get-User before licences removed in the cloud, showing a mailbox and that it was previously a mailbox as well:

image

PS > Get-User twomailboxes | FL name,recipienttype,previousrecipienttypedetails,*sku*


Name                         : Two Mailboxes
RecipientType                : UserMailbox
PreviousRecipientTypeDetails : UserMailbox
SKUAssigned                  : True

Get-Mailbox in the cloud showing the GUID was different from on-premises:

image

PS > Get-Mailbox twomailboxes | FL name,userprincipalname,exchangeguid


Name              : Two Mailboxes
UserPrincipalName :
twomailboxes@domain.com

ExchangeGuid      : d893372b-bfe0-4833-9905-eb497bb81de4

Once the licence is removed in Office 365 for Exchange Online and licence reconciliation completes (SKUAssigned is False) you will see that Get-User shows it is not a mailbox anymore:

image

PS > Get-User twomailboxes | FL name,recipienttype,previousrecipienttypedetails,*sku*


Name                         : Two Mailboxes
RecipientType                : MailUser
PreviousRecipientTypeDetails : UserMailbox
SKUAssigned                  : False

And finally Get-MailUser (not Get-Mailbox now) run in Exchange Online shows the ExchangeGUID matches the on-premises, synced, ExchangeGUID value:

image

PS > Get-MailUser twomailboxes | FL name,userprincipalname,exchangeguid


Name              : Two Mailboxes
UserPrincipalName : twomailboxes@domain.com
ExchangeGuid      : 625d70aa-82ed-47a2-afa2-d3c091d149aa

Note that giving these users back their Exchange Online licence will revert all of the above and restore their old mailbox. As these users cannot have an Exchange Online licence assigned in the cloud before you migrate their on-premises mailbox, as that will restore the old cloud mailbox, you need to ensure that within 30 days of their on-premises mailbox being migrated to the cloud you do give then an Exchange Online licence. Giving them a licence after migration of their on-premises mailbox to the cloud will ensure their single, migrated, mailbox remains in Exchange Online. But giving their user a licence before migration will restore their old cloud mailbox.

For users that never had a matching UPN in the cloud and a cloud mailbox, you can licence them before you migrate their mailbox as they will work correctly within the provisioning system in Exchange Online.


Tags:

Comments

8 responses to “Duplicate Exchange Online and Exchange Server Mailboxes”

  1. Rkast avatar
    Rkast

    Very detailed post once again!
    Mr Benoît just posted a new feature in EXO to mitigate this problem if im correct.
    See his article http://blog.hametbenoit.info/_layouts/15/mobile/disppost.aspx?List=a66c98bb%2Dce6d%2D4608%2Db0ed%2Dac39d78c0998&ID=1009

    1. Brian Reid avatar

      Thanks. If I had come across that article two days ago (but it was not written) I would have saved about four hours work and a blog post!

  2. Brian Reid avatar

    Microsoft posted a notification of an update to https://blogs.technet.microsoft.com/exchange/2018/01/17/permanently-clear-previous-mailbox-info/ about 12 hours after I wrote the above. So whilst the same issue can still occur, within a few days from date of writing you will be able run Set-MailUser name -PermanentlyClearPreviousMailboxInfo to permanently delete the previous mailbox belonging to the user. Note though that if that mailbox contained content and you are in a regulated industry or have compliance requirements then don’t use this switch and use the full steps.above, as you will need to protect this mailbox. You will find that permanently deleting mailboxes where you have holds in place is not a good idea. So the above scenario and licence management is still required in some places.

    1. Rkast avatar
      Rkast

      Thanks Brian for the update!
      Sorry for ‘ruining’ your mood.

  3. Hristo avatar
    Hristo

    Thats nice explained, as my experience confirms the same. But what is the root cause of failing ExchangeGUID on-prem to be synced to MailUser on cloud ? Have you find anything in your environments ?

    1. Brian Reid avatar

      The root cause is as described in the post. Either 1) you are not syncing Exchange attributes, or 2) you created the user first and gave it a mailbox – Exchange will not accept changes to the ExchangeGUID now

  4. kostas k avatar
    kostas k

    Hi,
    I have read your article 2-3 times and my issue here is that i have different exchange guid’s on cloud and on premises. This results in a failed migration batch.

    I have enabled aad sync and all users created on o365. Then run the exchange hybrid wizard and everything completed fine.
    i am syncing my user’s OU and the exchange ones only. I had never created users on cloud before migration
    I have run remove-msoluser with the removefromrecyclebin command and user was removed. I also run the new cmdlet to clean the user so the deleted mailbox do not reconnect back when i assign the lisence. However when i reenable the license to the user then a new mailbox is probably provisioned and when checking for exchangeguid i see that its different.

    Why o365 cannot see that this user has a mailbox on premises? When i run command to check for immutableid on cloud it seemed empty.
    Thats my facts. I hope i was clear enough

    How can i get out of this? how to troubleshoot why this attribute is not syncing with azure so i can start migrating them?

    Thank you in advance

    1. Brian Reid avatar

      Is your AADConnect correctly syncing Exchange AD attributes? That is, did you install AADConnect before Exchange Server? AADConnect needs to be able to read the msExchMailboxGuid attribute to sync it, and it needs to know it is there so that it can sync it. Other option is that you are doing selective sync and not choosing to sync Exchange attributes. For the first, refresh the schema in AADConnect and for the second ensure you are syncing all attributes and not selective attributes.

Leave a Reply to Brian Reid Cancel reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.