Allowing “Phish” Training Emails in Exchange Online

With the introduction of “Secure by Default” in Exchange Online, where you used to add IPs and other settings such as allow-lists to allow your phish training emails to get through to your users this will no longer work, and shortly it will no longer work to bypass the Exchange Online filters using mail flow (transport) rules.

So to ensure that any third-party phish simulation companies “messages” reach your users, you need to turn on the Advanced Delivery policy. The steps for this are covered below. The IPs and other values you need are obtained from your phish training provider, but this blog post uses (some of) the values from KnowBe4.

The Microsoft “Attack Simulation Training” feature does not require this setup, as this training provider places the messages directly in the mailbox and does not deliver them via SMTP (where they could be filtered out).

The Advanced Delivery policy is found in the new Security Center (https://security.microsoft.com) > Policies & Rules > Threat Policies > Advanced Delivery. Then click the Phishing Simulation tab.

Click Add to add a new policy (as shown) or Edit if you are updating an existing policy. There is only a single policy, so if you use two or more phish training providers you would add all the values to a single policy.

To create the policy you need to add one or more Sending Domains. For KnowBe4 this is psm.knowbe4.com as shown:

The Sending Domains value can be the P1 domain of the phish simulation emails and from October 2021 it can be the DKIM signing domain as well. You are allowed to add 10 domains to this list, so in cases where the phish simulation provider uses lots of domains it may be that they are all signed with one or two DKIM domains.

The Sending IPs that you need are also provided by your simulation provider. In the case of KnowBe4 they list a US and EU range, though the EU range completely encompasses the US range. You would add all the IPs that your providers send from here as shown:

Once all the IPs are added you can click Add or Save (if editing) the policy as the Simulation URLs are option (see below).

This will result in a view in the admin portal that looks like this:

Once you have created your policy you can see that you have an edit button. Changes now will result in the flyout prompt showing you want you have changed as shown rather than the whole policy (as shown):

And the change above that I have made is to add the optional Simulation URLs. These (for KnowBe4) are provided from your Phish Simulation Admin Portal at KnowBe4, so these values here are just for demo purposes. For example, lets say that your phish simulation provider sends links from email.pretender-domain.net and intranet.pretender-domain.net. You could either add both of these domains, or *.pretender-domain.net or ~pretender-domain.net (the last of these is the root domain and all subdomains). The last of these is preferred as you can only add 10 domains to the Simulation URLs dialog.

Finally once you have added your policy you can go and clean up years of arbitrary settings to allow this in the past. These settings will be found in the following:

• AntiPhishing (Defender for Office 365) Policy
• AntiSpam Policy (your allow lists are in the anti-spam inbound policies) and the Connection Filter Policy (allowed IPs are likely here)
• Safe Attachments (Defender for Office 365) Policy
• Safe Links (Defender for Office 365) Policy
• Transport (or mail-flow) Rules

Of course, all the above are settings in Security and Compliance portals, and so can be set and modified by PowerShell. Here are the cmdlets to do all the above in one set of cmdlets (you need to connect to Exchange Online for New-TenantAllowBlockListItems and the Security and Compliance PowerShell (IPPSSession) for the PhishSimOverride cmdlets.

Connect-ExchangeOnline -UserPrincipalName admin@domain.com
Connect-IPPSSession -UserPrincipalName admin@domain.com
New-PhishSimOverridePolicy -Name PhishSimOverridePolicy
New-PhishSimOverrideRule -Name PhishSimOverrideRule -Policy PhishSimOverridePolicy -SenderDomainIs psm.knowbe4.com -SenderIpRanges 147.160.167.0/26,52.49.201.246,52.49.235.189,23.21.109.197,23.21.109.212
New-TenantAllowBlockListItems -ListType URL -ListSubType AdvancedDelivery -Entries "~pretender-domain.net" -Allow

And if you are testing, the following will clean up the policy and rule that PowerShell or the portal creates:

Connect-ExchangeOnline -UserPrincipalName admin@domain.com
Connect-IPPSSession -UserPrincipalName admin@domain.com
Remove-PhishSimOverridePolicy -Identity PhishSimOverridePolicy
Remove-TenantAllowBlockListItems -ListType URL -ListSubType AdvancedDelivery -Entries "~pretender-domain.net"