Categories
DNS EOP exchange exchange online Exchange Online Protection Exchange Server smtp

Enabling Better Mail Flow Security for Exchange Online

At Microsoft Ignite 2020, Microsoft announced support for MTA-STS, or Mail Transfer Agent Strict Transport Security. This is covered in RFC 8461 and it includes making TLS for mail flow to your domains mandatory whereas it is currently down to the decision of the sender.

You can publish your SMTP endpoint and offer the STARTTLS verb but there is no requirement for the sender to use it unless you have configured the sender as well to ensure that they only email you over TLS (for example RequireTLS and TLSDomain settings in Exchange Server/Exchange Online connectors). MTA-STS allows you, the domain owner, to publish your TLS requirements.

You publish your requirements by placing a policy file in your websites “.well-known” directory. The policy will have version: STSv1 and mode: [testing|enforce|none] and mx record. “Testing” for mode says send the delivery of the email will work regardless of success or failure, but also send a report if it failed. “Enforce” means security must pass or the message delivery fails and “none” clears the policy, acting as if you don’t have a policy but giving you a route to remove the policy cleanly rather than what might happen if the policy was to disappear (mail flow should stop). The policy will also have a max_age value in seconds on how long the sender should cache the policy. For example:

version: STSv1
mode: testing
mx: mail.domain.com
mx: c7solutions-com.mail.protection.outlook.com
max_age: 600

In the above example, my policy is for testing and so I have set a short max_age value, though a value of weeks or more would typically be expected with 31557600 being the largest value you can set (a year and 1/4 of a day in seconds).

The text file must be called mta-sts.txt in the .well-known folder of the mts-sts domain, for example https://mta-sts.c7solutions.com/.well-known/mta-sts.txt

Finally, the policy is published via DNS with the _mta-sts subdomain record:

_mta-sts.c7solutions.com  TXT  "v=STSv1; id=202009241541"

This DNS record must be v=STSv1 and the id needs to be a value that changes when the policy file changes, so I have just used a date string, but it could be anything that you change as the policy changes. The DNS record can also be a CNAME record instead of a TXT record when someone else hosts your email infrastructure and in this case the value points to the MTA-STS domain of the provider instead.

Testing mode was mentioned above, and that is covered in my second blog post today on this topic – Reporting on MTA-STS Failures

Categories
EOP exchange exchange online Exchange Online Protection Exchange Server

Reporting on MTA-STS Failures

This article is a follow up to the Enabling Better Mail Flow Security for Exchange Online which discusses setting up MTA-STS and in this article we cover the reporting for MTA-STS.

To get daily reports from each sending infrastructure to receive reports on MTA-STS you just create a DNS record in the following format:

_smtp._tls.c7solutions.com IN TXT "v=TLSRPTv1;rua=mailto:hostmaster@c7solutions.com"

Once I get some reports I will update this blog post with examples of what I received.

Categories
enhanced filtering EOP exchange exchange online Exchange Online Protection Exchange Server mimecast Office 365 spam

Enable EOP Enhanced Filtering for Mimecast Users

Enhanced Filtering is a feature of Exchange Online Protection (EOP) that allows EOP to skip back through the hops the messages has been sent through to work out the original sender.

Take for example a message from SenderA.com to RecipientB.com where RecipientB.com uses Mimecast (or another cloud security provider). The MX record for RecipientB.com is Mimecast in this example. When EOP gets the message it will have gone from SenderA.com > Mimecast > RecipientB.com > EOP, or it will have gone SenderA.com > Mimecast > EOP if you are not sending via any other system such as an on-premises network.

EOP though, without Enhanced Filtering, will see the source email as the previous hop – in the above example the email will appear to come from Mimecast or the on-premises IP address – and neither of these are the true sender for SenderA.com and so the message fails SPF if it is set to -all (hard fail) and possibly DMARC if set to p=reject. EOP won’t, because of this complexity in routing, reject hard fails or DMARC rejects immediately.

So how can you tell EOP about your complex routing – this is Enhanced Filtering. You add the IPs of your on-premises network and your cloud filter to the inbound connector that you create in EOP to receive your emails. For any source you need the list of IPs and here are the IPs at the time of writing for Mimecast EU datacenters in an easy to use PowerShell cmdlet to add them to your Inbound Connector in EOP.

Set-InboundConnector "Inbound from Mimecast EU" -EFSkipIPs 207.82.80.0/24,146.101.78.0/24,185.58.84.0/22,91.220.42.0/24,195.130.217.0/24,193.7.205.0/24,193.7.204.0/24

In the above, get the name of the connector correct and it adds the IPs for you. It takes about an hour to take effect, but after this time inbound emails via Mimecast EU are skipped for spf/DMARC checking in EOP. For organisations with complex routing this is something you need to implement.

Categories
attribution domain enhanced filtering EOP exchange exchange online Exchange Online Protection Exchange Server mimecast Office 365 smtp transport

Mail Flow To The Correct Exchange Online Connector

In a multi-forest Exchange Server/Exchange Online (single tenant) configuration, you are likely to have multiple inbound connectors to receive email from the different on-premises environments. There are scenarios where it is important to ensure that the correct connector is used for the inbound message rather than any of your connectors. Here is one such example.

With multiple inbound connectors you might be happy and successfully complete your testing if the email from on-premises appears in the correct cloud mailbox. But what about when you use Enhanced Filtering. Here you need to add the intermediate IP addresses of all the hops the message can go through to the specific connector so that Exchange Online Protection can determine the real source IP address and then do spam/spf etc. on the true sender IP and not the hop before Exchange Online Protection (likely your on-premises server and not the actual source).

For example, lets send an email from SenderDomain.com to RecipientDomain.com, where RecipientDomain.com uses Mimecast, has Exchange Servers and has moved mailboxes to Exchange Online. The mail flow for this scenario is:

SenderDomainServer Public IP > MX (Mimecast) > Mimecast IPs > On-Premises IPs (internal) > Public IP for on-premises servers > EOP

From the EOP view point, the email is received from the public IP for the on-premises servers and not from the actual sending IP address. This means that the message will fail SPF as you have complex routing in-front of the receipt by EOP. This, out of interest, is the reason why EOP will not reject SPF failures even if DMARC reject is in place.

When the message arrives at EOP, the message needs to be attributed to the correct connector. If you have multiple Exchange Server orgs in separate on-premises environments you need to make sure that the message is associated (attributed) to the correct Inbound Connector.

This message attribution is done by looking for all Inbound Connectors of type On-Premises in your tenant. If you have more than one connector of type On-Premises, looking up the TlsSenderCertificateName value on the Inbound Connectors to find the connector that best matches the certificate used to encrypt the inbound message. So lets take a look at the example above again. In the “Public IP for on-premises servers > EOP” hop this message will be encrypted with a certificate called (lets say) “mail.recipientdomain.com” and the Exchange Hybrid Wizard will have created the Inbound Connector for this mail flow with TlsSenderCertificateName set to *.recipientdomain.com. Other Inbound Connectors from other on-premises orgs are possibly going to have similar certificates (they should not have the same one) with similar subject names and the Hybrid Wizard could have made more than one Inbound Connector with *.recipientdomain.com as the TlsSenderCertificateName value. If you have multiple Inbound Connectors of type On-Premises and more than one connector with TlsSenderCertificateName set to *.recipientdomain.com then the message could be attributed to the wrong connector.

If you have set Enhanced Filtering IPs to the other connector though, the Enhanced Filtering will not work because the message is not received by the connector you think it should be received by.

So how do you fix this. You modify the Hybrid Wizard created Inbound Connector TlsSenderCertificateName value to be the subject name of the certificate, so not *.recipientdomain.com but mail.recipientdomain.com and you register mail.recipientdomain.com as a domain in Office 365. You need to do both. The reason the Hybrid Wizard sets TlsSenderCertificateName to *.recipientdomain.com is to avoid you needing to add domains to Office 365 that match your certificate precisely, but if you have multiple connectors this is the only way to guarantee message attribution to the correct connector.

Now you can add the IPs you want to skip with Enhanced Filtering to the specific connector, mail flow will use the specific connector and the IPs will be skipped. EOP will resolve the correct sender IP (SenderDomain Public IP in the above example) even though the message has gone through Mimecast and on-premises servers as well. The message headers will now show:

X-MS-Exchange-SkipListedInternetSender ip=[Sender Server IP Address];domain=FQDN of sender

And not list Mimecast (or whomever you are using as a second cloud filter) or your on-premises IP addresses as the true sender.

Categories
cyber bullying exchange exchange online Exchange Server offensive Office 365 supervision

Review and Audit Offensive Language in Office 365 Communications

A new feature as of May 2018 in Office 365 is to filter communications based upon the offensive language machine learning filter. This is part of the Supervision settings that have been available for a number of years. The Offensive Language model uses a combination of machine learning, artificial intelligence, and keywords to identify inappropriate email messages as part of anti-harassment and cyber bullying monitoring requirements.

Here we will walk through the process of setting up the offensive language filter and testing it out (without offending anyone)!

Setting Up Offensive Language Supervision

Open the Compliance Center at https://compliance.microsoft.com and select Supervision on the left as shown:

image

At the time of writing, the Compliance Center is new and not everything is visible here. By the time you read this article it might be possible to create your supervision reviews from this portal, but for now we need to go to the Security and Compliance Center – so click the link at the top of the page. You will see this:

image

If you cannot see this then you do not have the right permissions. Add yourself to the Supervisory Review role group so you can set up policies. Anyone who has this role assigned can access the Supervision page in the Compliance Center.

Click Create to create a supervision review. Enter a name and a description. You cannot change the name later on.

image

In the next page, select the users to supervise. Start with a test group before editing this policy to add a group that contains everyone.

image

You can also select users who are in the group and specifically exclude them if needed. Communications via Exchange and Teams are included by default. Third party sources can be added as well.

Click Next and move to the Choose communications to review tab. Here select Internal communications (which is not selected by default) and choose Use match data model condition. There is only one model, and that is the Offensive Language model – so that gets selected by default.

image

If you want to scope the filter a bit more then you can select Add a condition and set up rules – for example you could exclude a specific domain inbound.

Click Next and get to the Specify percentage to review tab

image

Here you get to set the percentage of communications to review. The default is 10%. This means that only 10% of all communications are reviewed, and the results you see are based on what was found in that 10%. In large organizations, 10% could be a lot of communications, and therefore could be a fair amount of offensive content. Therefore ensure both your reviewers are able to manage the review process without undue impact and understand that whatever you find – there is 10 times more of it happening. Smaller organizations might want to increase the percentage to review, or at least consider increasing the percentage to review.

Click Next and enter the email addresses of the reviewers. They need to have an Exchange Online mailbox to be able to do this, but the content for review does not go into the reviewers mailbox.

image

Click Next and get to the Review your settings tab. Check everything is okay and click Finish.

image

Your policy will be listed so that you can update it, apart from the name, in the future.

The policy is also displayed in a pop-out as shown:

image

In this pop-out you can see the name of the mailbox that the content for review will go into – therefore those users who are reviewers will need to have access to this mailbox if they want to use Outlook to do their review process. If the reviewers have access to the Compliance Center then review can be done there instead of in Outlook/OWA. Permissions need to be granted to the mailbox using PowerShell. The two cmdlets are, using your supervisory review mailbox as listed in the policy results.

Add-MailboxPermission "SupervisoryReview{GUID}@domain.onmicrosoft.com" -User "alias or email address of the account that has reviewer permissions to the supervision mailbox" -AccessRights FullAccess
Set-Mailbox "SupervisoryReview{GUID}@domain.onmicrosoft.com" -HiddenFromAddressListsEnabled: $false

You can add “-AutoMapping $false” to the Add-MailboxPermission if you want the review mailbox not always to appear as an additional mailbox in Outlook.

To Review Your Supervision Policy

In the Supervision Review pop-out (which you can get back by clicking on the policy name), click Open at the top.

This takes you to:

image

Here I can see I have nothing to review or pending items to look at. If you want to test this, think of something offensive and send it to yourself! It might turn up in the review portal, or it might not – remember only 10% of communications are subject to review.

Note: Emails subject to defined policies are processed in near real-time and can be tested immediately after the policy is configured. Chats in Microsoft Teams can take up to 24 hours to fully process in a policy.

I’m not going to send anything, but I will take a look back here later and I might update this blog if I ever get any hits!

To review the content, the menu across the top for Review and Resolved Items will show you the items and those that have been resolved. The actual HR and discipline process is obviously not covered by anything in this review process. Once resolved in the company, mark it as resolved here.

In OWA, you can open an additional mailbox and enter “super” and the supervisoryreview{GUID} mailbox appears:

image

Inside the supervisory review mailbox, there is a folder for the policy you just created and inside that are subfolders that indicate review (Non-Compliant and Questionable) and Resolved:

image

Blocking Offensive Language

This is just a review process. If you want to block content, then create a DLP policy that uses a dictionary of words to block. For more on the dictionary creation see https://docs.microsoft.com/en-us/office365/securitycompliance/create-a-keyword-dictionary

Categories
2016 2019 autodiscover autodiscover v2 calendar exchange exchange online Exchange Server Microsoft Teams Teams

Teams Calendar Fails To On-Premises Mailbox

Article Depreciated: Microsoft now auto-hides the Calendar icon in Teams if your on-premises Exchange Server is not reachable via AutoDiscover V2 and at least Exchange Server 2016 CU3 or later. Once you move your mailbox to Exchange Online (or a supported on-premises version), assuming you did not do any of the below, your Calendar icon appears in the left rail shortly after mailbox migration, though I have seen it take six hours and needs a Teams client restart as well

In Microsoft Teams, you have a calendar  (previously called meetings) icon in the main display that shows your diary and meetings etc. – except it does not work if your mailbox is not either in Exchange Online or, if if your mailbox is on-premises, you are not using Exchange Server 2016 CU3 or later.

The reason for this is that the Teams calendar uses AutoDiscover v2, which is only supported by Exchange Server 2016 CU3 and Exchange Online (note that CU3 is not the current version of Exchange Server 2016 and versions later than CU3 also support AutoDiscover v2).

This means that if you have an earlier version of Exchange Server on-premises then the calendar in Teams is not functional. This raises IT support calls as users expect it to be available, and this impacts your deployment of Teams as it appears broken.

So how can we fix this. Well clearly migrating to Exchange Online or installing the 2016 or later version of Exchange Server is the obvious option from the above, but there is another option to work around this issue. The “fix” is to remove the calendar icon from Teams. This does not stop you booking meetings, as you can still do that in Outlook with the Teams add-in or in the Outlook mobile client, where Teams meeting support is rolling out as I write this blog. If I remove the calendar icon, then the source of the errors disappears, but Teams is not really adversely impacted.

So this is what we start with:

image

And we remove the icon by creating a new App Setup Policy in the Teams Admin centre and then deploying that policy to all your users (with on-premises mailboxes on older versions of Exchange, or those not using Exchange for calendaring). You can easily roll this out as a test, though its about 24 hours for the effect to be seen, and then roll it out in bulk for all your impacted users. We will cover all this below:

 

1. Creating App Setup Policy

In the Teams Admin centre (https://admin.teams.microsoft.com) expand Teams Apps > Setup Policies and create a new policy. This policy is based on your current Global policy.

Select the Calendar app and remove it from this new policy. You should see something like this:

SNAGHTML31beae7d

Here I have created an app policy called “With OnPrem Mailboxes” and removed the Calendar app from it.

2. Applying App Setup Policy To A Test User

Once you have the policy ready, its time to test it. Policy changes will take 24 hours to apply (so say the docs) and I found on my testing it was 18 hours when I ran through these steps – so this is not quick!

To make sure your changes work, the plan here is to deploy this new policy to a few selected individuals in the Teams admin centre.

Find the first user and click on their name. In the details page you will see the policies applied to the lower left:

image

Click Edit at the top right of this section and change the App setup policy to your new policy:

image

And click Save:

image

You will see your new policy in the list.

Repeat for the rest of your test pool of users using the portal. We will not use the portal for deploying it to all users though, that will take too long!

Next day, these users should see something like this – no calendar:

image

3. Applying App Setup Policy To All Users

To apply this change to all users once your test users are happy we will use PowerShell, and we will use the Skype for Business Online PowerShell cmdlets (not the Teams PowerShell!).

The following one-line PowerShell, once you have connected to your tenant, is:

Get-CSOnlineUser | ForEach-Object { Grant-CsTeamsAppSetupPolicy -PolicyName "With OnPrem Mailboxes" -Identity $_.WindowsEmailAddress }

This gets all your users and applies a new Teams App Setup Policy to each of them. This works initially with this problem, as we assume all users are affected. If only a subset of your users are on-premises, then do not use this cmdlet to apply the initial change, but use the below to be more selective.

Within 24 hours the Calendar app will disappear from Teams for your users and they will not be phoning the help desk with issues that none of you can easily fix!

4. Applying App Setup Policy To Selected Users

The above cmdlet is a single run – it does not affect later and new users, nor is there a concept of a default policy that you can set as the one each new users gets. So every so often depending upon how often new users start employment you will want to run the below:

Get-CsOnlineUser -Filter { TeamsAppSetupPolicy -ne "With OnPrem Mailboxes" } | Grant-CsTeamsAppSetupPolicy -PolicyName "With OnPrem Mailboxes"

This gets all users where they do not have the selected App Policy already set and sets this just for these users. This is quicker than setting it for all users regardless.

You can use other filters to select users – for example, you could look for users without an on-premises mailbox and then run the ForEach against each of these users instead – this would work in a hybrid deployment.

When you are in a hybrid deployment and you move mailboxes to Exchange Online from on-premises, you will want to set those users just moved back to a policy that includes the calendar app. The same would go for organizations migrating to Exchange Server 2016 with inbound AutoDiscover from Office 365. Here you could use something like importing a CSV file of mailboxes being migrated (the same list you used to build the migration batches in the first place would do) and then run the ForEach for each item on the CSV file.

Categories
activesync android email exchange exchange online Exchange Server iPad iPhone

Too Many Folders To Successfully Migrate To Exchange Online

Exchange Online has a limit of 10,000 folders within a mailbox. If you try and migrate a mailbox with more than this number of folders then it will fail – and that would be expected. But what happens if you have a mailbox with less than this number of folders and it still fails for this same reason? This is the problem, with resolution, I outline below.

I was moving some mailboxes to Exchange Online when I came across the following error in the migration batch results:

Data migrated: 18.18 MB ‎(19,060,890 bytes)‎
Migration rate: 0 B ‎(0 bytes)‎
Error: MigrationMRSPermanentException: Error: Could not create folder 2288. –> MapiExceptionFolderHierarchyChildrenCountQuotaExceeded: Unable to create folder. ‎(hr=0x80004005, ec=1253)‎ Diagnostic context: Lid: 55847 EMSMDBPOOL.EcPoolSessionDoRpc called [length=204] Lid: 43559 EMSMDBPOOL.EcPoolSessionDoRpc returned [ec=0x0][length=468][latency=1] Lid: 52176 ClientVersion: 15.20.1730.17 Lid: 50032 ServerVersion: 15.20.1730.6019 Lid: 35180 Lid: 23226 — ROP Parse Start — Lid: 27962 ROP: ropCreateFolder [28] Lid: 17082 ROP Error: 0x4E5 Lid: 25953 Lid: 21921 StoreEc: 0x4E5 Lid: 27962 ROP: ropExtendedError [250] Lid: 1494 —- Remote Context Beg —- Lid: 38698 Lid: 29818 dwParam: 0x0 Msg: f28f1e21-62aa-4999-977f-ce310efea309-61f0997f-74d5-4421-9050-64f8272e5ac2[9]-28A06 Lid: 29920 dwParam: 0xB Lid: 29828 qdwParam: 0x2711 Lid: 29832 qdwParam: 0x2710 Lid: 45884 StoreEc: 0x4E5 Lid: 29876 StoreEc: 0x4E5 Lid: 30344 StoreEc: 0x4E5 Lid: 54080 StoreEc: 0x4E5 Lid: 56384 StoreEc: 0x4E5 Lid: 38201 StoreEc: 0x4E5 Lid: 35904 Lid: 45434 Guid: f12f3e45-67aa-89012-345f-ce678efea901 Lid: 10786 dwParam: 0x0 Msg: 15.20.1730.017:VI1PR0502MB2975:145a3769-3902-4e6b-9fe4-6db564e4eb92 Lid: 1750 —- Remote Context End —- Lid: 31418 — ROP Parse Done — Lid: 22417 Lid: 30609 StoreEc: 0x4E5 Lid: 29073 Lid: 20369 StoreEc: 0x4E5 Lid: 64464 Lid: 64624 StoreEc: 0x4E5

In the above I have highlighted some of the errors I was seeing – with the “could not create folder” message, the first indicator is that I have too many folders to migrate or I have a corrupt mailbox. Running Get-MoveRequestStatistics and including a full report (with -IncludeReport) shows in part the below. This was run to get more info on the move request. This was run from Exchange Online:

​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​26/03/2019 17:10:09 [VI1PR0502MB3855] ‘MigrationService (on behalf of ‘Brian.Reid@domain.co.uk’)’ created move request.
26/03/2019 17:10:15 [DB8PR05MB6025] The Microsoft Exchange Mailbox Replication service ‘DB8PR05MB6025.eurprd05.prod.outlook.com’ (15.20.1730.17 ServerCaps:01FFFFFF, ProxyCaps:07FFFFC7FD6DFDBF5FFFFFCB07EFFF, MailboxCaps:, legacyCaps:01FFFFFF) is examining the request.
26/03/2019 17:10:15 [DB8PR05MB6025] Content from the Shard mailbox (Mailbox Guid: f12f3e45-67aa-89012-345f-ce678efea901, Database: cc980daf-4402-4645-b26c-2a83760b161c) will be merged into the target mailbox.
26/03/2019 17:10:15 [DB8PR05MB6025] Connected to target mailbox ‘tenant.onmicrosoft.com\2c065e32-3bd5-4524-9aac-03880fa8e961 (Primary)’, database ‘EURPR05DG090-db014’, Mailbox server ‘DB8PR05MB6025.eurprd05.prod.outlook.com’ Version 15.20 (Build 1730.0).
26/03/2019 17:10:20 [DB8PR05MB6025] Connected to source mailbox ‘tenant.onmicrosoft.com\2c065e32-3bd5-4524-9aac-03880fa8e961 (Primary)’, database ‘DB’, Mailbox server ‘onprem.server.domain.com’ Version 15.0 (Build 847.0), proxy server ‘onprem.server.domain.com’ 15.0.847.40 ServerCaps:, ProxyCaps:, MailboxCaps:, legacyCaps:1FFFCB07FFFF.
26/03/2019 17:10:21 [DB8PR05MB6025] Request processing started.
26/03/2019 17:10:21 [DB8PR05MB6025] Source mailbox information:
Regular Items: 8443, 905.4 MB (949,422,345 bytes)
Regular Deleted Items: 1149, 189.9 MB (199,115,692 bytes)
FAI Items: 4651, 11.72 MB (12,285,701 bytes)
FAI Deleted Items: 9, 19.26 KB (19,721 bytes)
26/03/2019 17:10:21 [DB8PR05MB6025] Cleared sync state for request 2c065e32-3bd5-4524-9aac-03880fa8e961 due to ‘CleanupOrphanedMailbox’.
26/03/2019 17:10:21 [DB8PR05MB6025] Mailbox signature will not be preserved for mailbox ‘tenant.onmicrosoft.com\f12f3e45-67aa-89012-345f-ce678efea901 (Primary)’. Outlook clients will need to restart to access the moved mailbox.
26/03/2019 17:11:20 [DB8PR05MB6025] Stage: CreatingFolderHierarchy. Percent complete: 10.
26/03/2019 17:12:38 [DB8PR05MB6025] Initializing folder hierarchy from mailbox ‘tenant.onmicrosoft.com\2c065e32-3bd5-4524-9aac-03880fa8e961 (Primary)’: 29048 folders total.
26/03/2019 17:21:21 [DB8PR05MB6025] Folder creation progress: 1102 folders created in mailbox ‘tenant.onmicrosoft.com\2c065e32-3bd5-4524-9aac-03880fa8e961 (Primary)’.
26/03/2019 17:31:22 [DB8PR05MB6025] Folder creation progress: 2730 folders created in mailbox ‘tenant.onmicrosoft.com\2c065e32-3bd5-4524-9aac-03880fa8e961 (Primary)’.
26/03/2019 17:41:22 [DB8PR05MB6025] Folder creation progress: 4535 folders created in mailbox ‘tenant.onmicrosoft.com\2c065e32-3bd5-4524-9aac-03880fa8e961 (Primary)’.
26/03/2019 17:51:23 [DB8PR05MB6025] Folder creation progress: 6257 folders created in mailbox ‘tenant.onmicrosoft.com\2c065e32-3bd5-4524-9aac-03880fa8e961 (Primary)’.
26/03/2019 18:01:23 [DB8PR05MB6025] Folder creation progress: 7919 folders created in mailbox ‘tenant.onmicrosoft.com\2c065e32-3bd5-4524-9aac-03880fa8e961 (Primary)’.
26/03/2019 18:11:23 [DB8PR05MB6025] Folder creation progress: 9570 folders created in mailbox ‘tenant.onmicrosoft.com\2c065e32-3bd5-4524-9aac-03880fa8e961 (Primary)’.
26/03/2019 18:14:15 [DB8PR05MB6025] Fatal error StoragePermanentException has occurred

The move request logs show an increasing folder count, and when this exceeds 10,000 a storage error occurs.

So the next thing to do is to check what I have on-premises. I have generally two options to try and fix a mailbox I am moving to Exchange Online. One is to move the mailbox elsewhere on-premises (on the basis that I discard errors on-premises and then move a cleaner mailbox to the cloud) or run repairs on the mailbox. Note that running repairs on-premises is part of the move to the cloud anyway as Exchange Server does this as part of the move.

But this revealed nothing! The move request logs on-premises showed the same – there was over 10,000 folders (indeed some of my mailboxes had over 20,000 folders) and this was enumerated in the move request logs. A New-MailboxRepairRequest did nothing either. But interestingly, Get-MailboxFolderStatistics | Measure showed only 200 folders! Each of my failing mailboxes had between 150 and 263 folders – nothing like the +10,000 that the move request was finding!

So I opened the mailbox in Outlook having granted myself permissions to it – again nothing.

So I opened MFCMapi and had a look at the folders. Now MFCMapi shows everything in the mailbox, and not just items under the “top of the information store” folder. I went about expanding each subfolder I could find and I came across a subfolder that everytime i expanded it, MFCMapi would hang. I would close and restart MFCMapi and the same thing!

image

I had found my suspect folder – its a iPhone device that had created the +10,000 folders. Now that I had a good candidate for my issue, the fix was easy. I listed the active-sync devices using Get-MobileDevice -Mailbox “Richard Redmond” | FL Identity and then removed the suspect device using Remove-ActiveSyncDevice “domain.co.uk/OU/Richard Redmond/ExchangeActiveSyncDevices/iPhone§A9BCDE7FG57HIJ81KL1M08NOPQ” -Confirm:$false where the device identity was returned in the Get-MobileDevice cmdlet run just before.

This Remove-ActiveSyncDevice (or Remove-MobileDevice) cleans up this mailbox and deletes the partnership with the device.

Once this was done, I moved the mailbox again and it was ~200 folders and moved to Exchange Online without further issue.

Where I tested the move to Exchange Server rather than Exchange Online, I found that looking in the move request report (I had prestaged the move and then removed the corrupt mobile device), the move report showed information like the following and all I had done was removed one mobile device from the mailbox!

26/03/2019 17:41:22 [servername] Folder hierarchy changes reported in source ‘Primary (a8c13a2f-535b-d996-908e-ff84b1484a7)’: 200 changed folders, 24080 deleted folders.

From the users perspective, if the phone is an active device and is syncing email, then removing the phone causes it to create a new partnership. If the server allows any device then this is seamless to the user. If the server requires authorization to add a new device, then the user will be told this and service desk/admin will need to approve the device again. So if Allow/Block/Quarantine (ABQ) is not enabled on the server, one wonders if deleting all active sync partnerships before migrating any mailbox is an idea worth considering – there could be mailboxes I have moved that are <10,000 folders but not far from that number and therefore storing up issues for the future!

Categories
2013 2016 exchange Exchange Server update upgrade

bin/ExSMIME.dll Copy Error During Exchange Patching

I have seen a lot of this, and there are some documents online but none that described what I was seeing. I was getting the following on an upgrade of Exchange 2013 CU10 to CU22 (yes, a big difference in versions):

     The following error was generated when “$error.Clear();
           $dllFile = join-path $RoleInstallPath “bin\ExSMIME.dll”;
           $regsvr = join-path (join-path $env:SystemRoot system32) regsvr32.exe;
          start-SetupProcess -Name:”$regsvr” -Args:”/s `”$dllFile`”” -Timeout:120000;
         ” was run: “Microsoft.Exchange.Configuration.Tasks.TaskException: Process execution failed with exit code 3.
    at Microsoft.Exchange.Management.Tasks.RunProcessBase.InternalProcessRecord()
   at Microsoft.Exchange.Configuration.Tasks.Task.<ProcessRecord>b__b()
    at Microsoft.Exchange.Configuration.Tasks.Task.InvokeRetryableFunc(String funcName, Action func, Boolean terminatePipelineIfFailed)”.

The Exchange Server setup operation didn’t complete. More details can be found
in ExchangeSetup.log located in the <SystemDrive>:\ExchangeSetupLogs folder.

In this error the file ExSMIME.dll fails to copy. You can find the correct copy of this file in the CU source files at …\CU22\setup\serverroles\common. I copy the ExSMIME.dll file from here directly into the \Program Files\Microsoft\Exchange Server\v15\bin folder and then restart the upgrade.

I have found that the upgrade fails again here if it things there is a pending reboot due to other installations and I have also seen at this point the detection for the VC++ runtime fails. I have documented this elsewhere, and the workaround for the is found at https://c7solutions.com/2019/02/exchange-server-dependency-on-visual-c-failing-detection.

A reboot later and the installation is successful. The error somehow seems to think that the file is not where it is looking for it. In the ExchangeSetup.log file it records the issue as “Error 3”, which generally means “not found”!

Categories
crm Dynamics exchange exchange online Exchange Server router

CRM Router and Dynamics CRM V9 Online–No Emails Being Processed

This one is an interesting one – and it was only resolved by a call to Microsoft Support, who do not document that this setting is required.

The scenario is that you upgrade your CRM Router to v9 (as this is required before you upgrade Dynamics to V9) and you enable TLS 1.2 on the router server as well (also documented as required as part of the upgrade).

Dynamics is updated and all your email that is processed using the Router stops. Everything was working before and now it is not!

The fix is simple though – and complex as well. The simple thing is that it is a a single check box you need to set. The complex thing is that as this is a GDPR setting, each user needs to do it themselves and it cannot be enabled in bulk!

The option each user needs to allow is “Allow other Microsoft Dynamics 365 users to send email on your behalf” and that this was checked. This option is located in CRM > Options > Email > Select whether other users can send email for you

image

Once each user does this, the router will start to process emails for this user again.

Categories
exchange Exchange Server install vc++

Exchange Server Dependency on Visual C++ Failing Detection

Exchange Server for rollup updates and cumulative updates at the time of writing (Feb 2019) has a dependency on Visual C++ 2012. The link in the error message you get points you to the VC++ 2013 Redistributable though, and there is are later versions of this as well.

I found that by installing all versions VC++ 2011, 2012 and 2014 I was able to get past the following error – which I had on only one out of many servers.

Performing Microsoft Exchange Server Prerequisite Check

    Configuring Prerequisites                                 COMPLETED
     Prerequisite Analysis                                     FAILED
      Visual C++ 2012 Redistributable Package is a required component. Please ins
tall the required binaries and re-run the setup. Use URI https://www.microsoft.c
om/download/details.aspx?id=30679 to download the binaries.
      For more information, visit: http://technet.microsoft.com/library(EXCHG.150
)/ms.exch.setupreadiness.VC2012RedistDependencyRequirement.aspx

So regardless of what you see in the error and the download site you go to, you need another version.

I found this article lists all versions: https://stackoverflow.com/questions/12206314/detect-if-visual-c-redistributable-for-visual-studio-2012-is-installed/34209692

And I specifically installed the following versions which then put some DLL’s onto the server to get past the error:

image image image

Categories
Azure Active Directory Azure AD download exchange exchange online

Read Only And Attachment Download Restrictions in Exchange Online

Microsoft have released a tiny update to Exchange Online that has massive implications. I say tiny in that it take like 30 seconds to implement this (ok, may 60 seconds then).

When this is enabled, and below I will describe a simple configuration for this, your users when using Outlook Web Access on a computer that is not compliant with a conditional access rule in Azure AD, will result in OWA that is read only – attachments can be viewed in the browser only and not downloaded. There is even a mode to have attachments completely blocked.

So how to do this.

Step 1: Enable the OwaMailboxPolicy New Setting

Only users whose OWAMailboxPolicy have the ConditionalAccessPolicy set to ReadOnly or ReadOnlyPlusAttachmentsBlocked are impacted by this feature and only when the Conditional Access policy so restricts their session. For example if you wanted a subset of users to always have this restriction regardless, but not other users then you would create a new OwaMailboxPolicy and set the ConditionalAccessPolicy setting. Once that is done you would apply the policy to the selected users, but if you wanted this restriction to apply to all users, but only when they are on a personal (not-compliant or trusted), then you would apply the OWA policy to all users and the conditional access policy to All Users as well.

In my example I am just going to update the default policy, becuase I want read only view for all users who fall out of the conditions of the policy. So in Exchange Online PowerShell I run the following:

Set-OwaMailboxPolicy -Identity OwaMailboxPolicy-Default -ConditionalAccessPolicy ReadOnly

This, once the conditional access policy takes effect will restrict downloads in OWA. The second option is to use ReadOnlyPlusAttachmentsBlocked instead of ReadOnly. This blocks attachment viewing as well. The value “Off” turns off the restrictions again. “Off” is the default value.

Step 2: Create a Conditional Access Policy in Azure AD

You need an Azure AD Premium P1 licence for this feature.

Here I created a policy that applied to one user and no other policy settings. This would mean this user is always in ReadOnly mode. In real world scenarios you would more likely create a policy that applied to a group or All Users and excluding network ranges or compliant devices and not individual users and forced ReadOnly only when other conditions such as non-compliant device (i.e. home computer) where in use.

The steps for this are:

imageimageimageimage

The pictures, as you cannot create the policies in the cmdline, are as follows:

a) New policy with a name. Here it is “Limited View for ZacharyP”

b) Under “Users and Groups” I selected my one test user. Here you are more likely to pick the users for whom data leakage is an issue

c) Under “Cloud apps” select Office 365 Exchange Online. I have also selected SharePoint, as the same idea exists in that service as well

d) Under Session, and this is the important one, select “Use app enforced restrictions”. For Exchange Online, app enforced restrictions is the value of ConditionalAccessPolicy for the given user.

Step 3: View the results

Ensure the user is licenced to have a mailbox and Azure AD Premium P1 and ensure they have an email with an attachment in it for testing.

In the screenshot you can see circled where the Download link is normally found:

image

And where the attachment is clicked, there is now a greyed out Download button and a banner is seen in both views telling the user of their limited access.

image

The new user interface to OWA looks as follows:

image

With ReadOnlyPlusAttachmentsBlocked set as the ConditionalAccessPolicy value, the attachment cannot be viewed. This is what this looks like (new OWA UI):

image

And SharePoint and OneDrive, just because it is very similar!

This is outlined in https://c7solutions.com/2019/04/read-only-and-document-download-restrictions-in-sharepoint-online

Categories
exchange exchange online Exchange Server migration Public Folders

Public Folder Migrations and the Changing Cmdlets

To complete a public folder migration from Exchange 2013/2016 to Exchange Online you need to run

Set-OrganizationConfig -PublicFolderMailboxesLockedForNewConnections $true

But if you look at lots of the documentation that is out there with their tips and tricks etc. you will see that lots of them say:

Set-OrganizationConfig –PublicFoldersLockedForMigration $true

So very near – but its the wrong cmdlet now and it does nothing. It does not lock out the public folders and in the cloud all you get is:

PS C:\Users\BrianReid> Complete-MigrationBatch PublicFolderMigration
The public folders in the source environment are not ready for finalizing the migration. Make sure that public folder
access is locked on the source Exchange server, and there are no active public folder mailbox moves or public folder
moves in the source.
     + FullyQualifiedErrorId : [Server=VI1PR09MB2909,RequestId=ca0ffb4a-cc9f-4195-94fd-e3dd060587e6,TimeStamp=13/12/2018 18:03:00] [FailureCategory=Cmdlet-MigrationBatchCannotBeCompletedException] 2FB8651C,Microsoft.Exchange.Management.Migration.MigrationService.Batch.CompleteMigrationBatch
     + PSComputerName        : outlook.office365.com

And there is nothing useful on the web for this error, so I wrote this to help you get out of this hole!

Run the correct cmdlet and migrations will start!

Categories
AADConnect exchange exchange online Exchange Server migration Office 365 Public Folders

Public Folder Sync–Duplicate Name Error

I came across this error with a client today and did not find it documented anywhere – so here it is!

When running the Public Folder sync script Sync-ModernMailPublicFolders.ps1 which is part of the process of preparing your Exchange Online environment for a public folder migration, you see the following error message:

UpdateMailEnabledPublicFolder : Active Directory operation failed on O365SERVERNAME.)365DATACENTER.PROD.OUTLOOK.COM. The
object ‘CN=PublicFolderName,OU=tenant.onmicrosoft.com,OU=Microsoft Exchange Hosted
Organizations,DC=)365DATACENTER,DC=PROD,DC=OUTLOOK,DC=COM’ already exists.
At C:\ExchangeScripts\pfToO365\Sync-ModernMailPublicFolders.ps1:746 char:9
+         UpdateMailEnabledPublicFolder $folderPair.Local $folderPair.Remote;
+         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
     + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,UpdateMailEnabledPublicFolder

This is caused because you have a user or other object in Active Directory that has the same name as the mail enabled public folder object.

In Exchange Online PowerShell if you run Get-User PublicFolderName you should not get anything back, as its a Public Folder and not a user, but where you see the above error you do get a response to Get-User (or maybe Get-Contact or any other object that is not a Public Folder. This class of object name (common name or cn) means the script can create the public folder in the cloud, but not update it on subsequent runs of the script.

The easiest fix is to rename the common name of the public folder object in Active Directory for all clashing public folders, unless you know you do not need the other object that clashes – as renaming that and letting AADConnect sync process the change is another way to resolve this.

To rename the mail public folder, in Exchange Server management shell run Set-MailPublicFolder PublicFolderName –Name NewPublicFolderName

I have changed my names to start with pf, so PublicFolderName becomes pfPublicFolderName and then the script runs without issue.

Categories
aadrm Azure Information Protection certificates exchange exchange online IRM Office Office 365 rms SSL

Azure Information Protection and SSL Inspection

I came across this issue the other day, so thought I would add it to my blog. We were trying to get Azure Information Protection operating in a client, and all we could see when checking the download of the templates in File > Info inside an Office application was the following:

02-Setup RMS Menu

03-Setup RMS Menu

04-Setup RMS Menu Error

The sequence of events was File > Info, click Set Permissions. You get the “Connect to Rights Management Servers and get templates” menu item. Clicking this shows a box saying “Retrieving templates from server” (which you might not see as this step takes no real time at all) and then an error that reads “Your machine isn’t set up for Information Rights Management (IRM). To set up IRM, sign into Office, open and existing IRM protected message or document, or contact your helodesk”.

For each of these recommendations, we tried them and still got the same message.

So what was the issue?

In https://docs.microsoft.com/en-us/azure/information-protection/get-started/requirements#firewalls-and-network-infrastructure Microsoft state the the IRM client in Windows uses Certificate Pinning. This is where the client application knows what certificate it expects to see at the service it is connecting to. If it gets a different certificate it will fail to connect. Within enterprise organizations, firewalls and proxy devices that do SSL Inspection change the certificate in use so that they can see the content being sent to the service in the clear. For the IRM client in Windows, this means that IRM does not trust the certificate and so will not work.

You can test for SSL Inspection on a URL by browsing the target URL in Chrome. For example, for IRM go to https://admin.na.aadrm.com/admin/admin.svc and click the Secure banner in the address bar:

image

You will get a popup – hover over the “Certificate (Valid)” message. If the certificate is not valid then either your PC is missing some important updates or SSL inspection is happening, but not implemented correctly!

You can use this same test to check for SSL Inspection on any network.

The certificate listed when you hover over the “Certificate (Valid)” message should read (for AIP) a Microsoft CA issued certificate. It should not list your company or proxy service as the issuer. Do not terminate the TLS client-to-service connections (for example, to do packet-level inspection) to the Azure Rights Management service. Doing so breaks the certificate pinning that RMS clients use with Microsoft-managed CAs to help secure their communication with the Azure Rights Management service.

For network performance, Microsoft also have a list of URLs that they recommend you do not inspect for Office 365 services. This list of endpoints that should not be inspected are those categorised as Optimize or Allowed when you browse

https://endpoints.office.com/endpoints/O365Worldwide?ClientRequestId=GUID. Interestingly at the time of writing this lists aadrm.com as Default, which means it can be inspected – I have reported this to the team that manages the endpoint service so that this URL can be moved up in its classification.

Once you bypass SSL Inspection for *.aadrm.com you will find that the Office and RMS clients work fine (assuming everything else is enabled correctly of course).

Categories
error exchange exchange online Exchange Server migration move

CannotEnterFinalizationTransientException On Exchange Move Request

Did not find a lot on the internet on this particular error, so I guess it does not happen very often, but in my case it delayed to move of the mailbox in question by a week or so until I could resolve it.

When a mailbox is moving to a different Exchange organization (cross-forest or to/from Exchange Online) the move process copies the mailbox data to the target database and then right at the end of the move updates Active Directory in both the source and target forest. In the source it changes the object type from mailbox to mailuser (or remotemailbox if Exchange Online is in play, though this is really a special form of mailuser) and in the target, updates the mailuser to become a mailbox.

This particular error occurs at this stage. The Get-MoveRequest cmdlet reports Failed as the status, and Get-MoveRequestStatistics reports FailedOther as the status. If you get the move logs (Get-MoveRequestStatistics <name> -IncludeReport | FL | Out-File <filename.txt>) then in the logs you will see CannotEnterFinalizationTransientException as the error repeated many times until the move fails.

The fix for this issue is as follows:

1. Check that the Exchange System account has permission to the Active Directory object in question. In Active Directory Users and Computers choose View > Advanced to enable the Security tab and then view the security tab on the object in question. Edit > Advanced and then check or click “Enable Inheritance” option or button (depending upon version of AD tools). If inheritance is already set to enabled there is probably no harm in disabling inheritance, copying permissions and then enabling inheritance again.

2. Move the mailbox to a different database in the source Exchange Organization (New-MoveRequest <name>) and waiting for that to complete.

3. Removing and restarting the move in the target forest. If you do not remove and restart the move in the target you will see both MailboxIsNotInExpectedMDBPermanentException and SourceMailboxAlreadyBeingMovedTransientException. The first of these is because the mailbox is not where the target move expects it to be, and the second of these is becuase the source is currently being moved and so cannot be moved to the correct target forest at the same time.

This should resolve your ultimate move request – it did for me! 

Categories
Authentication EOP exchange exchange online Exchange Online Protection Exchange Server hybrid smtp spam

Anonymous Emails Between On-Premises and Exchange Online

When you set up Exchange Hybrid, it should configure your Exchange organizations (both on-premises and cloud) to support the fact that an email from a person in one of the organizations should appear as internal to a recipient in the other organization. In Outlook that means you will see “Display Name” at the top of the message and not “Display Name” <email address>. An email from the internet is rightly treated as anonymous and so should appear as “Display Name” <email address> but when it comes from your on-premises environment or your cloud tenant it should be authenticated.

In the email headers you should see a header called AuthAs that reads internal. The SCL (Spam Confidence Level) should always be –1 and you should not have a header called X-CrossPremisesHeadersFilteredBySendConnector visible on internal emails.

Your hybrid setup can be incorrectly configured and cause this, and depending upon what Exchange Server version you are running and when you last ran the hybrid wizard you can end up with different results.

Lets take a quick view to some of the settings you should see:

  1. Exchange Server 2010 (with or without Edge Server 2010)
    1. Hybrid wizard will use Remote Domains to control internal vs external and authentication state. You should have a Remote Domain for tenant.mail.onmicrosoft.com that shows TNEFEnabled, TrustedMailOutboundEnabled, TargetDeliverDomain, and IsInternal all set to True on-premises
    2. TrustedMailnboundEnabled attribute is set to True on Get-RemoteDomain domain.com in the cloud
    3. The AllowedOOFType, which controls Out Of Office is set to InternalLegacy
  2. Exchange Server 2013 and later
    1. Your “Outbound to Office 365” send connector on-premises should have CloudServicesMailEnabled set to True
    2. The Remote Domains matter for Out of Office and moderated emails/voting buttons, but not for authentication as mentioned in #1 above
    3. The Inbound Connector for “Inbound from GUID” should have CloudServicesMailEnabled set to True
  3. Exchange Server 2010 with Exchange Server 2013 or later Edge (no 2013 on-premises, only Edge)
    1. The setting CloudServicesMailEnabled needs to be True, but 2010 does not support this setting, so you need to edit the directory using ADSIEdit and change the msExchSmtpSendFlags on the send connector from 64 to 131136. All this does is tell the 2013 or later Edge to enable CloudServicesMailEnabled
    2. See https://support.microsoft.com/en-us/help/3212872/email-sent-from-an-on-premises-exchange-2013-edge-transport-server-to for the steps to do this
  4. As #3 but with 2010 and 2013 on-premises – run the cmdlets and hybrid wizard from the 2013 server and not connected to the 2010 server!