Exchange Transport Rules Corrupt On Installing New Exchange Server Version

Posted on Leave a commentPosted in 2013, 2016, Exchange Server, ndr, rules, transport

When you install Exchange Server into an existing Exchange organization, your existing configuration typically remains intact and associated with the previous servers and some configuration, that is global in nature, also works across both versions.

I can across a scenario where this does not work the other day. The scenario was the installation of Exchange Server 2016 CU12 as a brand new Exchange installation into an existing Exchange Server 2013 deployment. This AD forest has previously seen Exchange 2003 and Exchange 2010, but these server versions are now long gone.

The issue was that the transport rules all appeared in Exchange Server 2016 as disabled, but where all enabled in Exchange Server 2013. The Exchange Admin Center could not open the rules and an error was displayed at the bottom which when expanded showed that the RejectEnhancedStatus was invalid, along with lots of the settings of the rule – they all are missing in the right-hand side of the EAC view.

image

RejectEnhancedStatus is the error code returned when you write a rule that rejects messages with notification. In Exchange Server 2016 only 5.7.1 and 5.7.900 through 5.7.999 are allowed for the RejectMessageEnhancedStatusCode parameter, but the Exchange Server 2013 deployment at CU21 does not block the creation of other status codes. Therefore, if you have transport rules with codes other than the ones allowed in Exchange Server 2016 you get corrupted transport rules:

image

So – how to fix. Well you cannot set the RejectMessageEnhancedStatusCode to a new value in Exchange Server 2016, because this server says you also need to set the RejectMessageReasonText value as that is also an empty string and also shows that a lot of the rule properties are also empty. So you need to fix it in the older version of Exchange.

In Exchange Server 2016 running Get-TransportRule “name of rule” results in:

WARNING: The object transport rule name has been corrupted or isn’t compatible with Microsoft support requirements, and it’s in an inconsistent state. The following validation errors happened:

WARNING: Rule ‘transport rule name’ is corrupt. The specified enhanced status code ‘5.7.x’ is invalid or isn’t compatible with Transport Rule policy requirement. Valid values are 5.7.1, or a value in the range between 5.7.900 and 5.7.999. The code must contain no spaces or other characters.

Parameter name: RejectEnhancedStatus

But running the same on Exchange Server 2013 is successful:

image

Run the following Exchange Management Shell cmdlets in Exchange 2013:

Get-TransportRule “name of rule” | FL Name,RejectMessage*

This will return the configuration of the current rule regarding the RejectMessageEnhancedStatusCode (which is wrong for 2016) and the RejectMessageReasonText.

Then run the cmdlet to change the code to a supported value as shown:

image

Set-TransportRule “name of rule” -RejectMessageEnhancedStatusCode 5.7.1 –RejectMessageReasonText “copy the reason from the output of the above cmdlet”

You need to set the code to 5.7.1 and provide the text again, or Exchange will replace the text with its own text or you need to create a New-SystemMessage for the status codes .900 and above and then use that code in the transport rule.

Once the change has been made on Exchange Server 2013, and this change is written to the configuration partition of Active Directory, that change will replicate around AD. Once the change reaches the DC used by Exchange Server 2016 the error will disappear and Exchange Control Panel can be refreshed to remove the error.

image or image

Review and Audit Offensive Language in Office 365 Communications

Posted on Leave a commentPosted in cyber bullying, exchange, exchange online, Exchange Server, offensive, Office 365, supervision

A new feature as of May 2018 in Office 365 is to filter communications based upon the offensive language machine learning filter. This is part of the Supervision settings that have been available for a number of years. The Offensive Language model uses a combination of machine learning, artificial intelligence, and keywords to identify inappropriate email messages as part of anti-harassment and cyber bullying monitoring requirements.

Here we will walk through the process of setting up the offensive language filter and testing it out (without offending anyone)!

Setting Up Offensive Language Supervision

Open the Compliance Center at https://compliance.microsoft.com and select Supervision on the left as shown:

image

At the time of writing, the Compliance Center is new and not everything is visible here. By the time you read this article it might be possible to create your supervision reviews from this portal, but for now we need to go to the Security and Compliance Center – so click the link at the top of the page. You will see this:

image

If you cannot see this then you do not have the right permissions. Add yourself to the Supervisory Review role group so you can set up policies. Anyone who has this role assigned can access the Supervision page in the Compliance Center.

Click Create to create a supervision review. Enter a name and a description. You cannot change the name later on.

image

In the next page, select the users to supervise. Start with a test group before editing this policy to add a group that contains everyone.

image

You can also select users who are in the group and specifically exclude them if needed. Communications via Exchange and Teams are included by default. Third party sources can be added as well.

Click Next and move to the Choose communications to review tab. Here select Internal communications (which is not selected by default) and choose Use match data model condition. There is only one model, and that is the Offensive Language model – so that gets selected by default.

image

If you want to scope the filter a bit more then you can select Add a condition and set up rules – for example you could exclude a specific domain inbound.

Click Next and get to the Specify percentage to review tab

image

Here you get to set the percentage of communications to review. The default is 10%. This means that only 10% of all communications are reviewed, and the results you see are based on what was found in that 10%. In large organizations, 10% could be a lot of communications, and therefore could be a fair amount of offensive content. Therefore ensure both your reviewers are able to manage the review process without undue impact and understand that whatever you find – there is 10 times more of it happening. Smaller organizations might want to increase the percentage to review, or at least consider increasing the percentage to review.

Click Next and enter the email addresses of the reviewers. They need to have an Exchange Online mailbox to be able to do this, but the content for review does not go into the reviewers mailbox.

image

Click Next and get to the Review your settings tab. Check everything is okay and click Finish.

image

Your policy will be listed so that you can update it, apart from the name, in the future.

The policy is also displayed in a pop-out as shown:

image

In this pop-out you can see the name of the mailbox that the content for review will go into – therefore those users who are reviewers will need to have access to this mailbox if they want to use Outlook to do their review process. If the reviewers have access to the Compliance Center then review can be done there instead of in Outlook/OWA. Permissions need to be granted to the mailbox using PowerShell. The two cmdlets are, using your supervisory review mailbox as listed in the policy results.

Add-MailboxPermission "SupervisoryReview{GUID}@domain.onmicrosoft.com" -User "alias or email address of the account that has reviewer permissions to the supervision mailbox" -AccessRights FullAccess
Set-Mailbox "SupervisoryReview{GUID}@domain.onmicrosoft.com" -HiddenFromAddressListsEnabled: $false

You can add “-AutoMapping $false” to the Add-MailboxPermission if you want the review mailbox not always to appear as an additional mailbox in Outlook.

To Review Your Supervision Policy

In the Supervision Review pop-out (which you can get back by clicking on the policy name), click Open at the top.

This takes you to:

image

Here I can see I have nothing to review or pending items to look at. If you want to test this, think of something offensive and send it to yourself! It might turn up in the review portal, or it might not – remember only 10% of communications are subject to review.

Note: Emails subject to defined policies are processed in near real-time and can be tested immediately after the policy is configured. Chats in Microsoft Teams can take up to 24 hours to fully process in a policy.

I’m not going to send anything, but I will take a look back here later and I might update this blog if I ever get any hits!

To review the content, the menu across the top for Review and Resolved Items will show you the items and those that have been resolved. The actual HR and discipline process is obviously not covered by anything in this review process. Once resolved in the company, mark it as resolved here.

In OWA, you can open an additional mailbox and enter “super” and the supervisoryreview{GUID} mailbox appears:

image

Inside the supervisory review mailbox, there is a folder for the policy you just created and inside that are subfolders that indicate review (Non-Compliant and Questionable) and Resolved:

image

Blocking Offensive Language

This is just a review process. If you want to block content, then create a DLP policy that uses a dictionary of words to block. For more on the dictionary creation see https://docs.microsoft.com/en-us/office365/securitycompliance/create-a-keyword-dictionary

Teams Calendar Fails To On-Premises Mailbox

Posted on 1 CommentPosted in 2016, 2019, autodiscover, autodiscover v2, calendar, exchange, exchange online, Exchange Server, Microsoft Teams, Teams

In Microsoft Teams, you have a calendar  (previously called meetings) icon in the main display that shows your diary and meetings etc. – except it does not work if your mailbox is not either in Exchange Online or, if if your mailbox is on-premises, you are not using Exchange Server 2016 CU3 or later.

The reason for this is that the Teams calendar uses AutoDiscover v2, which is only supported by Exchange Server 2016 CU3 and Exchange Online (note that CU3 is not the current version of Exchange Server 2016 and versions later than CU3 also support AutoDiscover v2).

This means that if you have an earlier version of Exchange Server on-premises then the calendar in Teams is not functional. This raises IT support calls as users expect it to be available, and this impacts your deployment of Teams as it appears broken.

So how can we fix this. Well clearly migrating to Exchange Online or installing the 2016 or later version of Exchange Server is the obvious option from the above, but there is another option to work around this issue. The “fix” is to remove the calendar icon from Teams. This does not stop you booking meetings, as you can still do that in Outlook with the Teams add-in or in the Outlook mobile client, where Teams meeting support is rolling out as I write this blog. If I remove the calendar icon, then the source of the errors disappears, but Teams is not really adversely impacted.

So this is what we start with:

image

And we remove the icon by creating a new App Setup Policy in the Teams Admin centre and then deploying that policy to all your users (with on-premises mailboxes on older versions of Exchange, or those not using Exchange for calendaring). You can easily roll this out as a test, though its about 24 hours for the effect to be seen, and then roll it out in bulk for all your impacted users. We will cover all this below:

 

1. Creating App Setup Policy

In the Teams Admin centre (https://admin.teams.microsoft.com) expand Teams Apps > Setup Policies and create a new policy. This policy is based on your current Global policy.

Select the Calendar app and remove it from this new policy. You should see something like this:

SNAGHTML31beae7d

Here I have created an app policy called “With OnPrem Mailboxes” and removed the Calendar app from it.

2. Applying App Setup Policy To A Test User

Once you have the policy ready, its time to test it. Policy changes will take 24 hours to apply (so say the docs) and I found on my testing it was 18 hours when I ran through these steps – so this is not quick!

To make sure your changes work, the plan here is to deploy this new policy to a few selected individuals in the Teams admin centre.

Find the first user and click on their name. In the details page you will see the policies applied to the lower left:

image

Click Edit at the top right of this section and change the App setup policy to your new policy:

image

And click Save:

image

You will see your new policy in the list.

Repeat for the rest of your test pool of users using the portal. We will not use the portal for deploying it to all users though, that will take too long!

Next day, these users should see something like this – no calendar:

image

3. Applying App Setup Policy To All Users

To apply this change to all users once your test users are happy we will use PowerShell, and we will use the Skype for Business Online PowerShell cmdlets (not the Teams PowerShell!).

The following one-line PowerShell, once you have connected to your tenant, is:

Get-CSOnlineUser | ForEach-Object { Grant-CsTeamsAppSetupPolicy -PolicyName "With OnPrem Mailboxes" -Identity $_.WindowsEmailAddress }

This gets all your users and applies a new Teams App Setup Policy to each of them. This works initially with this problem, as we assume all users are affected. If only a subset of your users are on-premises, then do not use this cmdlet to apply the initial change, but use the below to be more selective.

Within 24 hours the Calendar app will disappear from Teams for your users and they will not be phoning the help desk with issues that none of you can easily fix!

4. Applying App Setup Policy To Selected Users

The above cmdlet is a single run – it does not affect later and new users, nor is there a concept of a default policy that you can set as the one each new users gets. So every so often depending upon how often new users start employment you will want to run the below:

Get-CsOnlineUser -Filter { TeamsAppSetupPolicy -ne "With OnPrem Mailboxes" } | Grant-CsTeamsAppSetupPolicy -PolicyName "With OnPrem Mailboxes"

This gets all users where they do not have the selected App Policy already set and sets this just for these users. This is quicker than setting it for all users regardless.

You can use other filters to select users – for example, you could look for users without an on-premises mailbox and then run the ForEach against each of these users instead – this would work in a hybrid deployment.

When you are in a hybrid deployment and you move mailboxes to Exchange Online from on-premises, you will want to set those users just moved back to a policy that includes the calendar app. The same would go for organizations migrating to Exchange Server 2016 with inbound AutoDiscover from Office 365. Here you could use something like importing a CSV file of mailboxes being migrated (the same list you used to build the migration batches in the first place would do) and then run the ForEach for each item on the CSV file.

Save Time! Have All Your Meetings End Early

Posted on 4 CommentsPosted in calendar, exchange online, Exchange Server, monthly channel, Office 365, Office 365 ProPlus, Outlook, semi-annual channel

I am sure you have been in a meeting, where the meeting end time rolls around and there is a knock at the door from the people who want the meeting room now, as their meeting time has started and yours has finished.

What if you could recover five, eight, ten or more minutes per meeting so that the next meeting party can get into the room on time, and you have time to get out and get to your next meeting, and be on time.

Well since the beginning of 2019, Microsoft have come to your rescue.

image

The above are the new calendar “End appointments and meetings early” option. It is available in Outlook for Windows that is part of Office 365 ProPlus and you need to have a version of the software released new in 2019 for the feature to be available – more on the version and what to do in the technical section below.

The above option is found from File > Options > Calendar and then looking under Calendar Options as shown.

Check the option ”End appointments and meetings early” and then choose the time that a meeting under 1 hour will end early, and you can choose 5, 8 or 10 minutes, and then a second option for meetings over 1 hour – these can end 5, 10 or 15 minutes early. You can also enter your own preferred end early time.

Click OK and go create a new meeting. It should not matter how you create the meeting.

As you can see from my options above, my default meeting is 30 minutes – so on creating a new meeting I see the following:

image 

I’ve highlighted the new end time – its 25 minutes after the meeting starts! The adjustment applies to the default meeting length and shortens it for me.

If for this meeting I want it to be the full 30 minutes, I can just write in the new time – all Outlook is doing is setting a new adjustable default for me.

For meetings where you drag out a custom duration in your calendar – it works here as well:

image

As you can see I have dragged out 1pm to 4pm on Thursday. Look what happens when I enter some text for the meeting subject:

image

The meeting is created with an end time ten minutes early (my preferred time saving duration for meetings over one hour). As with the above, I can adjust the time of this meeting to the full hour if I want to very easily – just drag the meeting block to the full hour and it is kept. Its just the default time when I first create the meeting that is adjusted.

Note that existing meetings are not changed – but if you go into an existing meeting and look at the end time drop down, you will see suggestions for the duration that take the early end time into consideration:

image

So, that’s how you can save time on your meetings (or at least one way, being prepared for them is another and technology cannot help there – yet!)

Changing The Defaults For Everyone

But what if you are the HR department or the representative of the department for digital change – what if you want to try and improve company culture and change these defaults across the board – well this is a job for IT, but they can easily roll out a setting to all your computers that set a end early time for both short and longer meeting durations.

They need to deploy a group policy setting that changes the registry at HKEY_CURRENT_USER\Software\Microsoft\Office\16.0\Outlook\Options\Calendar and updates both EndEarlyShort and EndEarlyLong values as well as the EndEventsEarly key. EndEarlyShort is of course the value that affects meetings under one hour – and you do not need to accept the Microsoft suggested durations of 5, 8 and 10 minutes. For example if I edit this DWORD registry key and set the value to 3, upon restarting Outlook my new meetings under one hour end three minutes early:

image

The EndEventsEarly value is the setting that turns the feature on. So as well as setting the end early times, you need to set this value to 1 as well.

If you want to roll out this change centrally and ensure that the end user cannot set their own custom end early time then you can change the registry key policy settings via HKEY_CURRENT_USER\Software\Policies\Microsoft\Office\16.0\Outlook\Options\Calendar. Changes in this registry location mean the user cannot adjust the end early times.

image

You can disable this option centrally as well by setting EndEventsEarly DWORD value to 0 – this has the effect of disabling the check box and so users cannot turn the option on.

All these three settings are included in the latest update to the Office365 Administrative Templates, available on Microsoft Download Center: https://www.microsoft.com/en-us/download/details.aspx?id=49030 as well.

Checking Your Outlook Version

Version 1812 or later in use on the Monthly Channel is required before you can use this feature. In most businesses you are probably using the Semi-Annual channel, and this has features deferred by at least six months. So to check, click File > Office Account in any Office application (shown below). To the right hand side you will see the below. You need to check you are running the Subscription Product and that under About Outlook (or whatever Office app you are checking), it reads Version 1812 or later and Monthly Channel. The Semi-Annual Channel is released in January and July each year and is deferred by at least six months, so as this feature was released in Dec 2018, this feature will not appear in the Semi-Annual Channel until at least July 2019 – build 1812 of the Semi-Annual Channel (and possibly not until build 1907). More on this release cycle can be found at https://docs.microsoft.com/en-us/deployoffice/overview-of-update-channels-for-office-365-proplus

image

Too Many Folders To Successfully Migrate To Exchange Online

Posted on 1 CommentPosted in activesync, android, email, exchange, exchange online, Exchange Server, iPad, iPhone

Exchange Online has a limit of 10,000 folders within a mailbox. If you try and migrate a mailbox with more than this number of folders then it will fail – and that would be expected. But what happens if you have a mailbox with less than this number of folders and it still fails for this same reason? This is the problem, with resolution, I outline below.

I was moving some mailboxes to Exchange Online when I came across the following error in the migration batch results:

Data migrated: 18.18 MB ‎(19,060,890 bytes)‎
Migration rate: 0 B ‎(0 bytes)‎
Error: MigrationMRSPermanentException: Error: Could not create folder 2288. –> MapiExceptionFolderHierarchyChildrenCountQuotaExceeded: Unable to create folder. ‎(hr=0x80004005, ec=1253)‎ Diagnostic context: Lid: 55847 EMSMDBPOOL.EcPoolSessionDoRpc called [length=204] Lid: 43559 EMSMDBPOOL.EcPoolSessionDoRpc returned [ec=0x0][length=468][latency=1] Lid: 52176 ClientVersion: 15.20.1730.17 Lid: 50032 ServerVersion: 15.20.1730.6019 Lid: 35180 Lid: 23226 — ROP Parse Start — Lid: 27962 ROP: ropCreateFolder [28] Lid: 17082 ROP Error: 0x4E5 Lid: 25953 Lid: 21921 StoreEc: 0x4E5 Lid: 27962 ROP: ropExtendedError [250] Lid: 1494 —- Remote Context Beg —- Lid: 38698 Lid: 29818 dwParam: 0x0 Msg: f28f1e21-62aa-4999-977f-ce310efea309-61f0997f-74d5-4421-9050-64f8272e5ac2[9]-28A06 Lid: 29920 dwParam: 0xB Lid: 29828 qdwParam: 0x2711 Lid: 29832 qdwParam: 0x2710 Lid: 45884 StoreEc: 0x4E5 Lid: 29876 StoreEc: 0x4E5 Lid: 30344 StoreEc: 0x4E5 Lid: 54080 StoreEc: 0x4E5 Lid: 56384 StoreEc: 0x4E5 Lid: 38201 StoreEc: 0x4E5 Lid: 35904 Lid: 45434 Guid: f12f3e45-67aa-89012-345f-ce678efea901 Lid: 10786 dwParam: 0x0 Msg: 15.20.1730.017:VI1PR0502MB2975:145a3769-3902-4e6b-9fe4-6db564e4eb92 Lid: 1750 —- Remote Context End —- Lid: 31418 — ROP Parse Done — Lid: 22417 Lid: 30609 StoreEc: 0x4E5 Lid: 29073 Lid: 20369 StoreEc: 0x4E5 Lid: 64464 Lid: 64624 StoreEc: 0x4E5

In the above I have highlighted some of the errors I was seeing – with the “could not create folder” message, the first indicator is that I have too many folders to migrate or I have a corrupt mailbox. Running Get-MoveRequestStatistics and including a full report (with -IncludeReport) shows in part the below. This was run to get more info on the move request. This was run from Exchange Online:

​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​26/03/2019 17:10:09 [VI1PR0502MB3855] ‘MigrationService (on behalf of ‘Brian.Reid@domain.co.uk’)’ created move request.
26/03/2019 17:10:15 [DB8PR05MB6025] The Microsoft Exchange Mailbox Replication service ‘DB8PR05MB6025.eurprd05.prod.outlook.com’ (15.20.1730.17 ServerCaps:01FFFFFF, ProxyCaps:07FFFFC7FD6DFDBF5FFFFFCB07EFFF, MailboxCaps:, legacyCaps:01FFFFFF) is examining the request.
26/03/2019 17:10:15 [DB8PR05MB6025] Content from the Shard mailbox (Mailbox Guid: f12f3e45-67aa-89012-345f-ce678efea901, Database: cc980daf-4402-4645-b26c-2a83760b161c) will be merged into the target mailbox.
26/03/2019 17:10:15 [DB8PR05MB6025] Connected to target mailbox ‘tenant.onmicrosoft.com\2c065e32-3bd5-4524-9aac-03880fa8e961 (Primary)’, database ‘EURPR05DG090-db014’, Mailbox server ‘DB8PR05MB6025.eurprd05.prod.outlook.com’ Version 15.20 (Build 1730.0).
26/03/2019 17:10:20 [DB8PR05MB6025] Connected to source mailbox ‘tenant.onmicrosoft.com\2c065e32-3bd5-4524-9aac-03880fa8e961 (Primary)’, database ‘DB’, Mailbox server ‘onprem.server.domain.com’ Version 15.0 (Build 847.0), proxy server ‘onprem.server.domain.com’ 15.0.847.40 ServerCaps:, ProxyCaps:, MailboxCaps:, legacyCaps:1FFFCB07FFFF.
26/03/2019 17:10:21 [DB8PR05MB6025] Request processing started.
26/03/2019 17:10:21 [DB8PR05MB6025] Source mailbox information:
Regular Items: 8443, 905.4 MB (949,422,345 bytes)
Regular Deleted Items: 1149, 189.9 MB (199,115,692 bytes)
FAI Items: 4651, 11.72 MB (12,285,701 bytes)
FAI Deleted Items: 9, 19.26 KB (19,721 bytes)
26/03/2019 17:10:21 [DB8PR05MB6025] Cleared sync state for request 2c065e32-3bd5-4524-9aac-03880fa8e961 due to ‘CleanupOrphanedMailbox’.
26/03/2019 17:10:21 [DB8PR05MB6025] Mailbox signature will not be preserved for mailbox ‘tenant.onmicrosoft.com\f12f3e45-67aa-89012-345f-ce678efea901 (Primary)’. Outlook clients will need to restart to access the moved mailbox.
26/03/2019 17:11:20 [DB8PR05MB6025] Stage: CreatingFolderHierarchy. Percent complete: 10.
26/03/2019 17:12:38 [DB8PR05MB6025] Initializing folder hierarchy from mailbox ‘tenant.onmicrosoft.com\2c065e32-3bd5-4524-9aac-03880fa8e961 (Primary)’: 29048 folders total.
26/03/2019 17:21:21 [DB8PR05MB6025] Folder creation progress: 1102 folders created in mailbox ‘tenant.onmicrosoft.com\2c065e32-3bd5-4524-9aac-03880fa8e961 (Primary)’.
26/03/2019 17:31:22 [DB8PR05MB6025] Folder creation progress: 2730 folders created in mailbox ‘tenant.onmicrosoft.com\2c065e32-3bd5-4524-9aac-03880fa8e961 (Primary)’.
26/03/2019 17:41:22 [DB8PR05MB6025] Folder creation progress: 4535 folders created in mailbox ‘tenant.onmicrosoft.com\2c065e32-3bd5-4524-9aac-03880fa8e961 (Primary)’.
26/03/2019 17:51:23 [DB8PR05MB6025] Folder creation progress: 6257 folders created in mailbox ‘tenant.onmicrosoft.com\2c065e32-3bd5-4524-9aac-03880fa8e961 (Primary)’.
26/03/2019 18:01:23 [DB8PR05MB6025] Folder creation progress: 7919 folders created in mailbox ‘tenant.onmicrosoft.com\2c065e32-3bd5-4524-9aac-03880fa8e961 (Primary)’.
26/03/2019 18:11:23 [DB8PR05MB6025] Folder creation progress: 9570 folders created in mailbox ‘tenant.onmicrosoft.com\2c065e32-3bd5-4524-9aac-03880fa8e961 (Primary)’.
26/03/2019 18:14:15 [DB8PR05MB6025] Fatal error StoragePermanentException has occurred

The move request logs show an increasing folder count, and when this exceeds 10,000 a storage error occurs.

So the next thing to do is to check what I have on-premises. I have generally two options to try and fix a mailbox I am moving to Exchange Online. One is to move the mailbox elsewhere on-premises (on the basis that I discard errors on-premises and then move a cleaner mailbox to the cloud) or run repairs on the mailbox. Note that running repairs on-premises is part of the move to the cloud anyway as Exchange Server does this as part of the move.

But this revealed nothing! The move request logs on-premises showed the same – there was over 10,000 folders (indeed some of my mailboxes had over 20,000 folders) and this was enumerated in the move request logs. A New-MailboxRepairRequest did nothing either. But interestingly, Get-MailboxFolderStatistics | Measure showed only 200 folders! Each of my failing mailboxes had between 150 and 263 folders – nothing like the +10,000 that the move request was finding!

So I opened the mailbox in Outlook having granted myself permissions to it – again nothing.

So I opened MFCMapi and had a look at the folders. Now MFCMapi shows everything in the mailbox, and not just items under the “top of the information store” folder. I went about expanding each subfolder I could find and I came across a subfolder that everytime i expanded it, MFCMapi would hang. I would close and restart MFCMapi and the same thing!

image

I had found my suspect folder – its a iPhone device that had created the +10,000 folders. Now that I had a good candidate for my issue, the fix was easy. I listed the active-sync devices using Get-MobileDevice -Mailbox “Richard Redmond” | FL Identity and then removed the suspect device using Remove-ActiveSyncDevice “domain.co.uk/OU/Richard Redmond/ExchangeActiveSyncDevices/iPhone§A9BCDE7FG57HIJ81KL1M08NOPQ” -Confirm:$false where the device identity was returned in the Get-MobileDevice cmdlet run just before.

This Remove-ActiveSyncDevice (or Remove-MobileDevice) cleans up this mailbox and deletes the partnership with the device.

Once this was done, I moved the mailbox again and it was ~200 folders and moved to Exchange Online without further issue.

Where I tested the move to Exchange Server rather than Exchange Online, I found that looking in the move request report (I had prestaged the move and then removed the corrupt mobile device), the move report showed information like the following and all I had done was removed one mobile device from the mailbox!

26/03/2019 17:41:22 [servername] Folder hierarchy changes reported in source ‘Primary (a8c13a2f-535b-d996-908e-ff84b1484a7)’: 200 changed folders, 24080 deleted folders.

From the users perspective, if the phone is an active device and is syncing email, then removing the phone causes it to create a new partnership. If the server allows any device then this is seamless to the user. If the server requires authorization to add a new device, then the user will be told this and service desk/admin will need to approve the device again. So if Allow/Block/Quarantine (ABQ) is not enabled on the server, one wonders if deleting all active sync partnerships before migrating any mailbox is an idea worth considering – there could be mailboxes I have moved that are <10,000 folders but not far from that number and therefore storing up issues for the future!

Exchange Move Requests | Large Items | And Setting TCP KeepAliveTime To A Large Value

Posted on Leave a commentPosted in exchange online, Exchange Server, mailbox, move, networking

I have seen this situation a number of times. A large mailbox (or mailbox and archive) wont move to the target because the process of checking what the changes are in the mailbox take too long, the network or Exchange Server times out the users move and then reports the mailbox is locked.

The fix for this is counter though to everything else you read online about this. Often you will see to reduce the TCP KeepAliveTime and reboot the server. This is the opposite – increase the value and do not reboot the server. Here is why:

First make sure no bad items in your failed moves – this is not a fix for bad items, this is a fix where things timeout:

Get-MoveRequest -MoveStatus failed | Get-MoveRequestStatistics | fl badite*

View the Move Request Statistics log for one of your failed mailbox moves:

Get-MoveRequestStatistics "&lt;name&gt;" -IncludeReport | fl | Out-File movereport.txt

Search the report that you have saved in the above cmdlet and search for “Error” in the text file. If you get the following then the mailbox is probably too large for a successful move, which means the source server or network has not got the resources. What can happen is the move is progressing and a check happens for changes to the source mailbox – this takes a long time to complete and something times out. When target Exchange tries to connect again, the source has lost the TCP port and so a new move is started, but the mailbox is still locked for the old move. Therefore the move cannot continue.

I have found that by increasing TCP KeepAliveTime (contrary to all the advise online) that this solves the issue. Now I need to be clear here – all I am doing is changing the registry key for this setting and restarting the MRS service on the source Exchange Server. I am NOT restarting Windows, and so I am not changing the KeepAliveTime for the entire network stack. I think MRS checks the registry key to see the KeepAliveTime and sets this to the lock time on the mailbox during the move. If I can lock the mailbox for longer, moves don’t timeout and fail is the theory behind why this happens

The error I get in the MailboxStatistics report (see above for cmdlet) reads:

Message                                : Error: Couldn’t switch the mailbox into Sync Source mode.
                                          This could be because of one of the following reasons:
                                            Another administrator is currently moving the mailbox.
                                            The mailbox is locked.
                                            The Microsoft Exchange Mailbox Replication service (MRS) doesn’t have the correct permissions.
                                            Network errors are preventing MRS from cleanly closing its session with the Mailbox server. If this is the case, MRS may continue to encounter this error for up to 2 hours – this duration is controlled by the TCP KeepAlive settings on the Mailbox server.
                                          Wait for the mailbox to be released before attempting to move this mailbox again. –> An error occurred while saving the changes on the folder “FolderID/”. Error details: Failed, Property: [0x66180003]
                                          InTransitStatus, PropertyErrorCode: AccessDenied, PropertyErrorDescription: .
                                          –> Property: [0x66180003] InTransitStatus, PropertyErrorCode: AccessDenied,
                                          PropertyErrorDescription: .

Of interest in the error is the point that says “MRS may continue to encounter this error for up to 2 hours ”. This time value matches the default TCP KeepAliveTime value. Raising this in the registry and restarting the MRS service (not the server) changes the lock timout, which means that when the long job that is happening on the target finishes (and takes longer than two hours), the source server is still waiting for the connection and does not throw the above error.

Once you have your mailboxes moved, delete the registry value (to put it back to the default of two hours) and avoid rebooting the server when this key is set to a different value. If you started with a different value return to that one instead of deleting the registry value.

The KeepAliveTime setting is found at \HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters, and its a DWORD value called KeepAliveTime. The value is in milliseconds, so 7200000 is two hours and 86400000 is 24 hours (which is the value I tend to use to resolve this issue). This change is made on the mailbox server and the service restarted on that server (or servers if you have more than one).

bin/ExSMIME.dll Copy Error During Exchange Patching

Posted on Leave a commentPosted in 2013, 2016, exchange, Exchange Server, update, upgrade

I have seen a lot of this, and there are some documents online but none that described what I was seeing. I was getting the following on an upgrade of Exchange 2013 CU10 to CU22 (yes, a big difference in versions):

     The following error was generated when “$error.Clear();
           $dllFile = join-path $RoleInstallPath “bin\ExSMIME.dll”;
           $regsvr = join-path (join-path $env:SystemRoot system32) regsvr32.exe;
          start-SetupProcess -Name:”$regsvr” -Args:”/s `”$dllFile`”” -Timeout:120000;
         ” was run: “Microsoft.Exchange.Configuration.Tasks.TaskException: Process execution failed with exit code 3.
    at Microsoft.Exchange.Management.Tasks.RunProcessBase.InternalProcessRecord()
   at Microsoft.Exchange.Configuration.Tasks.Task.<ProcessRecord>b__b()
    at Microsoft.Exchange.Configuration.Tasks.Task.InvokeRetryableFunc(String funcName, Action func, Boolean terminatePipelineIfFailed)”.

The Exchange Server setup operation didn’t complete. More details can be found
in ExchangeSetup.log located in the <SystemDrive>:\ExchangeSetupLogs folder.

In this error the file ExSMIME.dll fails to copy. You can find the correct copy of this file in the CU source files at …\CU22\setup\serverroles\common. I copy the ExSMIME.dll file from here directly into the \Program Files\Microsoft\Exchange Server\v15\bin folder and then restart the upgrade.

I have found that the upgrade fails again here if it things there is a pending reboot due to other installations and I have also seen at this point the detection for the VC++ runtime fails. I have documented this elsewhere, and the workaround for the is found at https://c7solutions.com/2019/02/exchange-server-dependency-on-visual-c-failing-detection.

A reboot later and the installation is successful. The error somehow seems to think that the file is not where it is looking for it. In the ExchangeSetup.log file it records the issue as “Error 3”, which generally means “not found”!

CRM Router and Dynamics CRM V9 Online–No Emails Being Processed

Posted on Leave a commentPosted in crm, Dynamics, exchange, exchange online, Exchange Server, router

This one is an interesting one – and it was only resolved by a call to Microsoft Support, who do not document that this setting is required.

The scenario is that you upgrade your CRM Router to v9 (as this is required before you upgrade Dynamics to V9) and you enable TLS 1.2 on the router server as well (also documented as required as part of the upgrade).

Dynamics is updated and all your email that is processed using the Router stops. Everything was working before and now it is not!

The fix is simple though – and complex as well. The simple thing is that it is a a single check box you need to set. The complex thing is that as this is a GDPR setting, each user needs to do it themselves and it cannot be enabled in bulk!

The option each user needs to allow is “Allow other Microsoft Dynamics 365 users to send email on your behalf” and that this was checked. This option is located in CRM > Options > Email > Select whether other users can send email for you

image

Once each user does this, the router will start to process emails for this user again.

Exchange Server Dependency on Visual C++ Failing Detection

Posted on Leave a commentPosted in exchange, Exchange Server, install, vc++

Exchange Server for rollup updates and cumulative updates at the time of writing (Feb 2019) has a dependency on Visual C++ 2012. The link in the error message you get points you to the VC++ 2013 Redistributable though, and there is are later versions of this as well.

I found that by installing all versions VC++ 2011, 2012 and 2014 I was able to get past the following error – which I had on only one out of many servers.

Performing Microsoft Exchange Server Prerequisite Check

    Configuring Prerequisites                                 COMPLETED
     Prerequisite Analysis                                     FAILED
      Visual C++ 2012 Redistributable Package is a required component. Please ins
tall the required binaries and re-run the setup. Use URI https://www.microsoft.c
om/download/details.aspx?id=30679 to download the binaries.
      For more information, visit: http://technet.microsoft.com/library(EXCHG.150
)/ms.exch.setupreadiness.VC2012RedistDependencyRequirement.aspx

So regardless of what you see in the error and the download site you go to, you need another version.

I found this article lists all versions: https://stackoverflow.com/questions/12206314/detect-if-visual-c-redistributable-for-visual-studio-2012-is-installed/34209692

And I specifically installed the following versions which then put some DLL’s onto the server to get past the error:

image image image

451 4.7.0 Temporary server error. Please try again later. PRX2

Posted on 1 CommentPosted in DNS, error, Exchange Server

There are a few articles online about this error, but none were correct for the scenario i found a clients network in.

Not that I think the specifics matter, but this was Exchange Server 2016, Windows Domain Controllers running 2012 R2 and Exchange Hybrid. All the mailboxes had already moved to the cloud and the Exchange Server is used for attribute management and SMTP relay.

Sometimes, randomly it would seem, the applications fail to send email and get back the above error. So what does it mean! Lets dive into the Exchange logs to find out more.

In my example, TCP 25 is listening on a number of separate IPs on two different network cards on a server hosted in Azure (maybe all that matters for this case?)

Protocol Logs (Frontend)

In the Exchange Transport logs I turned on Protocol Logging for all connectors and sent some emails and had them rejected with the PRX2 error in the title. After 5 or so minutes the protocol logs contained the erroring session as shown below:

2019-01-31T13:45:09.477Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,0,10.10.10.16:25,10.150.14.108:59877,+,,
2019-01-31T13:45:09.478Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,1,10.10.10.16:25,10.150.14.108:59877,&amp;gt;,220 COMPANY Relay Connector SERVER,
2019-01-31T13:45:09.479Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,2,10.10.10.16:25,10.150.14.108:59877,&amp;lt;,HELO,
2019-01-31T13:45:09.479Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,3,10.10.10.16:25,10.150.14.108:59877,&amp;gt;,250 SERVER.internal.co.uk Hello [10.150.14.108],
2019-01-31T13:45:09.480Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,4,10.10.10.16:25,10.150.14.108:59877,&amp;lt;,MAIL FROM: &amp;lt;appserver@international.com&amp;gt;,
2019-01-31T13:45:09.480Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,5,10.10.10.16:25,10.150.14.108:59877,*,08D68772EDC476C6;2019-01-31T13:45:09.477Z;1,receiving message
2019-01-31T13:45:09.480Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,6,10.10.10.16:25,10.150.14.108:59877,&amp;gt;,250 2.1.0 Sender OK,
2019-01-31T13:45:09.482Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,7,10.10.10.16:25,10.150.14.108:59877,&amp;lt;,RCPT TO: &amp;lt;internal.user@international.com&amp;gt;,
2019-01-31T13:45:09.482Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,8,10.10.10.16:25,10.150.14.108:59877,&amp;gt;,250 2.1.5 Recipient OK,
2019-01-31T13:45:09.483Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,9,10.10.10.16:25,10.150.14.108:59877,&amp;lt;,RCPT TO: &amp;lt;brian@nbconsult.co&amp;gt;,
2019-01-31T13:45:09.483Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,10,10.10.10.16:25,10.150.14.108:59877,&amp;gt;,250 2.1.5 Recipient OK,
2019-01-31T13:45:09.484Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,11,10.10.10.16:25,10.150.14.108:59877,&amp;lt;,DATA,
2019-01-31T13:45:09.484Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,12,10.10.10.16:25,10.150.14.108:59877,&amp;gt;,354 Start mail input; end with &amp;lt;CRLF&amp;gt;.&amp;lt;CRLF&amp;gt;,
2019-01-31T13:45:09.498Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,13,10.10.10.16:25,10.150.14.108:59877,*,,Proxy destination(s) obtained from OnProxyInboundMessage event. Correlation Id:80e0d560-be23-4910-bcb0-43139bee131f
2019-01-31T13:45:09.501Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,14,10.10.10.16:25,10.150.14.108:59877,*,,Message or connection acked with status Retry and response 451 4.4.0 DNS query failed. The error was: DNS query failed with error InfoNoRecords -&amp;gt; DnsQueryFailed: InfoNoRecords
2019-01-31T13:45:09.501Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,15,10.10.10.16:25,10.150.14.108:59877,&amp;gt;,451 4.7.0 Temporary server error. Please try again later. PRX2 ,
2019-01-31T13:45:09.503Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,16,10.10.10.16:25,10.150.14.108:59877,&amp;lt;,QUIT,
2019-01-31T13:45:09.503Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,17,10.10.10.16:25,10.150.14.108:59877,&amp;gt;,221 2.0.0 Service closing transmission channel,
2019-01-31T13:45:09.503Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,18,10.10.10.16:25,10.150.14.108:59877,-,,Local

The protocol logs contain a number of columns to the left. The interesting ones for this are the connector name (“SERVER\From Internal Servers (Relay)”), the session ID (08D68772EDC476C6) and the sequence number (each item on the protocol has a incrementing sequence number, in the above it goes from 0 where the session connects (which is the + at the end) to 18, where it disconnects (the – at the end of the last line).

This log looks no different from a session that works (as it was random as I said above), but we see more about the error. Specifically we see the following:

Proxy destination(s) obtained from OnProxyInboundMessage event. Correlation Id:80e0d560-be23-4910-bcb0-43139bee131f
Message or connection acked with status Retry and response 451 4.4.0 DNS query failed. The error was: DNS query failed with error InfoNoRecords -&amp;gt; DnsQueryFailed: InfoNoRecords
451 4.7.0 Temporary server error. Please try again later. PRX2 ,

So we see that it is DNS. Online there are articles about this being to do with IPv6, AAAA records and invalid responses to those queries and fixes include using external DNS settings or smarthost values. None of this worked in this example.

So lets follow down the logs some more

Connectivity Logs

In the connectivity logs we search the same date/time/hour log for the session number, which in this case is 08D68772EDC476C6 from the above logs. In the connectivity logs we see a session that matches for this ID and its for “internalproxy”

2019-01-31T13:45:09.499Z,08D68772EDC476C7,SMTP,internalproxy,+,Undefined 00000000-0000-0000-0000-000000000000;QueueLength=&amp;lt;no priority counts&amp;gt;. Starting outbound connection for inbound session 08D68772EDC476C6
2019-01-31T13:45:09.501Z,08D68772EDC476C7,SMTP,internalproxy,&amp;gt;,DNS server returned InfoNoRecords reported by 10.10.10.21. [Domain:Result] = SERVER.internal.co.uk:InfoNoRecords;
2019-01-31T13:45:09.501Z,08D68772EDC476C7,SMTP,internalproxy,-,Messages: 0 Bytes: 0 (The DNS query for&amp;nbsp; 'Undefined':'internalproxy':'00000000-0000-0000-0000-000000000000' failed with error : InfoNoRecords)

Internalproxy is what Exchange users to send email from the frontend transport service to the hub transport service. But which hub transport service are we going to use? If does not matter if you have 1 or x number of Exchange Servers in your site, it will use DNS to look up the IP of one of these servers. So even if you have a single Exchange box, DNS is vital.

In the above log we see that DNS 10.10.10.21 returns InfoNoRecords when queried for the Exchange Servers own name.

So I resort to nslookup to check DNS from this Exchange server. I have two DNS server, .20 and .21. The error appears to be related to .21 in this case.

To I enter “nslookup server.internal.co.uk 10.10.10.21” which means look up the name of the server using the DNS server 10.10.10.21. I got back a message saying cannot find server.internal.co.uk: Query refused.

When I tried the other DNS server I got back a successful response and the IP address of the server.

So for immediate fix, I removed 10.10.10.21 as an option for DNS for this server. Exchange immediately went back to work and PRX2 errors where not displayed and email got to its destination.

Now to go and see who has broken DNS!

Public Folder Migrations and the Changing Cmdlets

Posted on 1 CommentPosted in exchange, exchange online, Exchange Server, migration, Public Folders

To complete a public folder migration from Exchange 2013/2016 to Exchange Online you need to run

Set-OrganizationConfig -PublicFolderMailboxesLockedForNewConnections $true

But if you look at lots of the documentation that is out there with their tips and tricks etc. you will see that lots of them say:

Set-OrganizationConfig –PublicFoldersLockedForMigration $true

So very near – but its the wrong cmdlet now and it does nothing. It does not lock out the public folders and in the cloud all you get is:

PS C:\Users\BrianReid> Complete-MigrationBatch PublicFolderMigration
The public folders in the source environment are not ready for finalizing the migration. Make sure that public folder
access is locked on the source Exchange server, and there are no active public folder mailbox moves or public folder
moves in the source.
     + FullyQualifiedErrorId : [Server=VI1PR09MB2909,RequestId=ca0ffb4a-cc9f-4195-94fd-e3dd060587e6,TimeStamp=13/12/2018 18:03:00] [FailureCategory=Cmdlet-MigrationBatchCannotBeCompletedException] 2FB8651C,Microsoft.Exchange.Management.Migration.MigrationService.Batch.CompleteMigrationBatch
     + PSComputerName        : outlook.office365.com

And there is nothing useful on the web for this error, so I wrote this to help you get out of this hole!

Run the correct cmdlet and migrations will start!

Test Connectivity Website and TLS 1.2

Posted on Leave a commentPosted in certificates, exchange online, Exchange Server, Kemp, SSL

An excellent resource for Microsoft Exchange Server and Exchange Online administrators and consultants is the Remote Test Connectivity website at http://exrca.com or https://testconnectivity.microsoft.com/.

Here I am going to document an error that indicated that the Exchange Server (in this case) was not working, but we could see that the phone was connecting fine to the server. The error we say was:

“The certificate couldn’t be validated because SSL negotiation wasn’t successful. This could have occurred as a result of a network error or because of a problem with the certificate installation.”

and also

“The Microsoft Connectivity Analyzer wasn’t able to obtain the remote SSL certificate”

The error looked like the following:

exrca tls 10 support[96033]

This error occurs when TLS 1.0 is disabled either on the end server or on a load balancer in front of the server. In my case this as the case with the Kemp load balancer we were using – TLS 1.0 was disabled under SSL Properties. Once we restored TLS 1.0 the Remote Connectivity Test tool, the tool worked instantly:

TLS Kemp setting[96034]

Public Folder Sync–Duplicate Name Error

Posted on Leave a commentPosted in AADConnect, exchange, exchange online, Exchange Server, migration, Office 365, Public Folders

I came across this error with a client today and did not find it documented anywhere – so here it is!

When running the Public Folder sync script Sync-ModernMailPublicFolders.ps1 which is part of the process of preparing your Exchange Online environment for a public folder migration, you see the following error message:

UpdateMailEnabledPublicFolder : Active Directory operation failed on O365SERVERNAME.)365DATACENTER.PROD.OUTLOOK.COM. The
object ‘CN=PublicFolderName,OU=tenant.onmicrosoft.com,OU=Microsoft Exchange Hosted
Organizations,DC=)365DATACENTER,DC=PROD,DC=OUTLOOK,DC=COM’ already exists.
At C:\ExchangeScripts\pfToO365\Sync-ModernMailPublicFolders.ps1:746 char:9
+         UpdateMailEnabledPublicFolder $folderPair.Local $folderPair.Remote;
+         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
     + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,UpdateMailEnabledPublicFolder

This is caused because you have a user or other object in Active Directory that has the same name as the mail enabled public folder object.

In Exchange Online PowerShell if you run Get-User PublicFolderName you should not get anything back, as its a Public Folder and not a user, but where you see the above error you do get a response to Get-User (or maybe Get-Contact or any other object that is not a Public Folder. This class of object name (common name or cn) means the script can create the public folder in the cloud, but not update it on subsequent runs of the script.

The easiest fix is to rename the common name of the public folder object in Active Directory for all clashing public folders, unless you know you do not need the other object that clashes – as renaming that and letting AADConnect sync process the change is another way to resolve this.

To rename the mail public folder, in Exchange Server management shell run Set-MailPublicFolder PublicFolderName –Name NewPublicFolderName

I have changed my names to start with pf, so PublicFolderName becomes pfPublicFolderName and then the script runs without issue.

CannotEnterFinalizationTransientException On Exchange Move Request

Posted on Leave a commentPosted in error, exchange, exchange online, Exchange Server, migration, move

Did not find a lot on the internet on this particular error, so I guess it does not happen very often, but in my case it delayed to move of the mailbox in question by a week or so until I could resolve it.

When a mailbox is moving to a different Exchange organization (cross-forest or to/from Exchange Online) the move process copies the mailbox data to the target database and then right at the end of the move updates Active Directory in both the source and target forest. In the source it changes the object type from mailbox to mailuser (or remotemailbox if Exchange Online is in play, though this is really a special form of mailuser) and in the target, updates the mailuser to become a mailbox.

This particular error occurs at this stage. The Get-MoveRequest cmdlet reports Failed as the status, and Get-MoveRequestStatistics reports FailedOther as the status. If you get the move logs (Get-MoveRequestStatistics <name> -IncludeReport | FL | Out-File <filename.txt>) then in the logs you will see CannotEnterFinalizationTransientException as the error repeated many times until the move fails.

The fix for this issue is as follows:

1. Check that the Exchange System account has permission to the Active Directory object in question. In Active Directory Users and Computers choose View > Advanced to enable the Security tab and then view the security tab on the object in question. Edit > Advanced and then check or click “Enable Inheritance” option or button (depending upon version of AD tools). If inheritance is already set to enabled there is probably no harm in disabling inheritance, copying permissions and then enabling inheritance again.

2. Move the mailbox to a different database in the source Exchange Organization (New-MoveRequest <name>) and waiting for that to complete.

3. Removing and restarting the move in the target forest. If you do not remove and restart the move in the target you will see both MailboxIsNotInExpectedMDBPermanentException and SourceMailboxAlreadyBeingMovedTransientException. The first of these is because the mailbox is not where the target move expects it to be, and the second of these is becuase the source is currently being moved and so cannot be moved to the correct target forest at the same time.

This should resolve your ultimate move request – it did for me! 

Exchange Server Object ID Error With Windows Server 2016 Domain Controllers

Posted on 1 CommentPosted in 2010, 2013, 2016, active directory, ADDS, error, Exchange Server

Saw this error the other day:

image

When you open Exchange Control Panel and view the Mailbox Delegation tab of any user account you get the following:

The object <name> has been corrupted, and it’s in an inconsistent state. The following validation errors happened: The access control entry defines the ObjectType ‘9b026da6-0d3c-465c-8bee-5199d7165cba’ that can’t be resolved..

You do not see this error on any mailboxes that you have moved to Office 365 in hybrid mode, that is you do not see it on any RemoteMailbox objects.

The issue is because ObjectType ‘9b026da6-0d3c-465c-8bee-5199d7165cba’ is the GUID of the DS-Validated-Write-Computer Control Access Right introduced in WS2016 AD DS which is new to your Active Directory upon installing your first 2016 domain controller. Exchange Server reads this access control list when you open the Mailbox Delegation tab in Exchange Control Panel or when you run Get-ADPermission on the mailbox. This error is cosmetic, but to remove it you just need to reboot all your Exchange Servers in turn (relying on your database availability groups and load balancers to maintain service). Once you have rebooted each server, the error goes away when you are connected to that server for administrative functions. There is no impact on user connectivity whilst this error is in place, though it may impact you ability to assign permissions without error.

Therefore recommend that you reboot one server as soon as you can and then use that server as your target for administration until you can reboot the remaining servers.

Copy Links and Backlinks Between Users and Shared Mailboxes (automapping)

Posted on 1 CommentPosted in cross-forest, Exchange Server, mailbox, migration, msExchDelegateListBL, msExchDelegateListLink, shared mailbox

Automap for shared mailboxes does not work across forests when moving mailboxes.

When the user is granted permission to a shared mailbox, the default behaviour of automapping means that the shared mailbox has msExchDelegateListLink set to the DN of the user, and the backlink (hidden in AD by default) on the user is populated with the DN of the shared mailbox. Whenever the link attribute is updated, the backlink is automatically updated as well. For more on back links see https://neroblanco.co.uk/2015/07/links-and-backlinks-in-active-directory-for-exchange/

That is, is UserMailbox is granted full access to SharedMailbox you will see the following in Active Directory (Advanced View) > Attribute Editor > msExchDelegateListLink = “CN=UserMailbox,OU=etc” (on the SharedMailbox). And for the UserMailbox in Active Directory (Advanced View) > Attribute Editor > msExchDelegateListBL = “CN=SharedMailbox,OU=etc”.

When you migrate mailboxes across forests you make use of Prepare-MoveRequest.ps1 to copy all the attributes. The msExchDelegateListLink is not part of this attribute set and the msExchDelegateListBL is auto populated so we can ignore it for now – if msExchDelegateListLink was copied and updated to the new forest name, then msExchDelegateListBL would be filled in automatically.

So how do we copy the msExchDelegateListLink value for each user and then write it to the mail user object in the target forest before the mailbox is migrated (or if you have already done your migration and found this property missing and so automapping of shared mailboxes having failed (though the permissions have been copied fine), how can you grab the data from the old source forest and apply it to the mailboxes in the target?

Using PowerShell and the ActiveDirectory module is how.

First you need to export a list of all the automapped shared mailboxes each user has (this is the msExchDelegateListBL values for the user mailboxes you have migrated). There are two cmdlets to run here, the first does the entire source directory and the second filters the output to an OU and its child OU’s (so you can export a subset of data) using SearchBase. Only one of these two cmdlets is needed.

This code is PowerShell and needs to be run from any domain joined computer.

Import-Module ActiveDirectory
Get-ADUser -Properties msExchDelegateListBL,msExchDelegateListLink -LDAPFilter "(msExchDelegateListBL=*)" | Select name,DistinguishedName,@{Name='SharedMailbox';Expression={$_.msExchDelegateListBL -Join ";"}} | Export-csv automap-userlist.csv -NoTypeInformation -NoClobber -Encoding UTF8
Get-ADUser -Properties msExchDelegateListBL,msExchDelegateListLink -LDAPFilter "(msExchDelegateListBL=*)" -SearchBase 'OU=Sales,DC=domain,DC=local' | Select name,DistinguishedName,@{Name='SharedMailbox';Expression={$_.msExchDelegateListBL -Join ";"}} | Export-CSV automap-userlist.csv -NoTypeInformation -NoClobber -Encoding UTF8

These cmdlets return a CSV file listing each mailbox that has an automapping to a shared mailbox and what that shared mailbox is.

The CSV file then needs copying to the target AD forest, and as the target forest is very unlikely to contain the same OU structure and domain names, the DN of each object in the CSV file needs updating. This can be done with Find/Replace in Excel or Notepad quite easily.

For example, in a CSV I might see:

“name”,”Distinguishedname”,”SharedMailbox”

“First User”,”CN=First User,OU=Sales,DC=domain,DC=local”,”CN=SharedMailbox,CN=Users,DC=domain,DC=local CN=AnotherSharedMailbox,OU=Shared Mailboxes,OU=Exchange,DC=domain,DC=local”

“Second User”,”CN=Second User,CN=Users,DC=domain,DC=local”,”CN=Sales,OU=Shared Mailboxes,OU=Exchange,DC=domain,DC=local”

In this I have the DN of the mailbox and the DN of the shared mailbox in the source forest. Use Find and Replace to change all the source DN’s (or the OU/DC bits) to suit the location of the matching object in the target forest. For example, my above “second user” was as shown, but after updating the DN might be “CN=Second User,OU=Migrated,DC=target,DC=forest”. So in that case I find/replace “CN=Users,DC=domain,DC=local” for “OU=Migrated,DC=target,DC=forest”.

For my examples that follow on from here, I have saved the edited CSV file as automap-userlist-target-dn-updated.csv

Once I have the CSV file updated for the values in the target forest, I need to split each row where a user has more than one shared mailbox listed into multiple rows. This is simple with PowerShell:

Import-Csv -Path automap-userlist-target-dn-updated.csv |
% {$row = $_; $_.SharedMailbox.split(";")} |
% {$row.SharedMailbox=$_; $row} |
Export-Csv automap-userlist-target-dn-updated-split.csv -NoClobber -NoTypeInformation -Encoding UTF8

Now that I have a row in the CSV for each Shared Mailbox to User Mailbox mapping, I can set the msExchDelegateListLink value on each shared mailbox for the DN of the user that has access to it. This will update the msExchDelegateListBL on the user object automatically.

Import-Module ActiveDirectory
Import-CSV "automap-userlist-target-dn-updated-split.csv" | % {
Write-Host Add $_.DistinguishedName to $_.SharedMailbox
Get-ADUser -identity $_.SharedMailbox | Set-ADUser -Add @{msExchDelegateListlink=$_.DistinguishedName} 
}

In terms of errors in the above, if you get “get-aduser : Directory object not found” then the DN value for the Shared Mailbox is wrong, and if you see “set-aduser : The name reference is invalid” then the DN value for the user who has access to the shared mailbox is wrong (the DistinguishedName column in the CSV). The script can be run multiple times, so you are safe to fix the CSV file and import the entire list again. It will only add a given DN once in total per shared mailbox.

If your target (or source) forest has more than one domain, run the script from a server in the correct domain or use “-Server DC-name” in both the Get-ADUser and the Set-ADUser cmdlets.