Exchange Transport Rules Corrupt On Installing New Exchange Server Version

Posted on Leave a commentPosted in 2013, 2016, Exchange Server, ndr, rules, transport

When you install Exchange Server into an existing Exchange organization, your existing configuration typically remains intact and associated with the previous servers and some configuration, that is global in nature, also works across both versions.

I can across a scenario where this does not work the other day. The scenario was the installation of Exchange Server 2016 CU12 as a brand new Exchange installation into an existing Exchange Server 2013 deployment. This AD forest has previously seen Exchange 2003 and Exchange 2010, but these server versions are now long gone.

The issue was that the transport rules all appeared in Exchange Server 2016 as disabled, but where all enabled in Exchange Server 2013. The Exchange Admin Center could not open the rules and an error was displayed at the bottom which when expanded showed that the RejectEnhancedStatus was invalid, along with lots of the settings of the rule – they all are missing in the right-hand side of the EAC view.

image

RejectEnhancedStatus is the error code returned when you write a rule that rejects messages with notification. In Exchange Server 2016 only 5.7.1 and 5.7.900 through 5.7.999 are allowed for the RejectMessageEnhancedStatusCode parameter, but the Exchange Server 2013 deployment at CU21 does not block the creation of other status codes. Therefore, if you have transport rules with codes other than the ones allowed in Exchange Server 2016 you get corrupted transport rules:

image

So – how to fix. Well you cannot set the RejectMessageEnhancedStatusCode to a new value in Exchange Server 2016, because this server says you also need to set the RejectMessageReasonText value as that is also an empty string and also shows that a lot of the rule properties are also empty. So you need to fix it in the older version of Exchange.

In Exchange Server 2016 running Get-TransportRule “name of rule” results in:

WARNING: The object transport rule name has been corrupted or isn’t compatible with Microsoft support requirements, and it’s in an inconsistent state. The following validation errors happened:

WARNING: Rule ‘transport rule name’ is corrupt. The specified enhanced status code ‘5.7.x’ is invalid or isn’t compatible with Transport Rule policy requirement. Valid values are 5.7.1, or a value in the range between 5.7.900 and 5.7.999. The code must contain no spaces or other characters.

Parameter name: RejectEnhancedStatus

But running the same on Exchange Server 2013 is successful:

image

Run the following Exchange Management Shell cmdlets in Exchange 2013:

Get-TransportRule “name of rule” | FL Name,RejectMessage*

This will return the configuration of the current rule regarding the RejectMessageEnhancedStatusCode (which is wrong for 2016) and the RejectMessageReasonText.

Then run the cmdlet to change the code to a supported value as shown:

image

Set-TransportRule “name of rule” -RejectMessageEnhancedStatusCode 5.7.1 –RejectMessageReasonText “copy the reason from the output of the above cmdlet”

You need to set the code to 5.7.1 and provide the text again, or Exchange will replace the text with its own text or you need to create a New-SystemMessage for the status codes .900 and above and then use that code in the transport rule.

Once the change has been made on Exchange Server 2013, and this change is written to the configuration partition of Active Directory, that change will replicate around AD. Once the change reaches the DC used by Exchange Server 2016 the error will disappear and Exchange Control Panel can be refreshed to remove the error.

image or image

Teams Calendar Fails To On-Premises Mailbox

Posted on Leave a commentPosted in 2016, 2019, autodiscover, autodiscover v2, calendar, exchange, exchange online, Exchange Server, Microsoft Teams, Teams

In Microsoft Teams, you have a calendar  (previously called meetings) icon in the main display that shows your diary and meetings etc. – except it does not work if your mailbox is not either in Exchange Online or, if if your mailbox is on-premises, you are not using Exchange Server 2016 CU3 or later.

The reason for this is that the Teams calendar uses AutoDiscover v2, which is only supported by Exchange Server 2016 CU3 and Exchange Online (note that CU3 is not the current version of Exchange Server 2016 and versions later than CU3 also support AutoDiscover v2).

This means that if you have an earlier version of Exchange Server on-premises then the calendar in Teams is not functional. This raises IT support calls as users expect it to be available, and this impacts your deployment of Teams as it appears broken.

So how can we fix this. Well clearly migrating to Exchange Online or installing the 2016 or later version of Exchange Server is the obvious option from the above, but there is another option to work around this issue. The “fix” is to remove the calendar icon from Teams. This does not stop you booking meetings, as you can still do that in Outlook with the Teams add-in or in the Outlook mobile client, where Teams meeting support is rolling out as I write this blog. If I remove the calendar icon, then the source of the errors disappears, but Teams is not really adversely impacted.

So this is what we start with:

image

And we remove the icon by creating a new App Setup Policy in the Teams Admin centre and then deploying that policy to all your users (with on-premises mailboxes on older versions of Exchange, or those not using Exchange for calendaring). You can easily roll this out as a test, though its about 24 hours for the effect to be seen, and then roll it out in bulk for all your impacted users. We will cover all this below:

 

1. Creating App Setup Policy

In the Teams Admin centre (https://admin.teams.microsoft.com) expand Teams Apps > Setup Policies and create a new policy. This policy is based on your current Global policy.

Select the Calendar app and remove it from this new policy. You should see something like this:

SNAGHTML31beae7d

Here I have created an app policy called “With OnPrem Mailboxes” and removed the Calendar app from it.

2. Applying App Setup Policy To A Test User

Once you have the policy ready, its time to test it. Policy changes will take 24 hours to apply (so say the docs) and I found on my testing it was 18 hours when I ran through these steps – so this is not quick!

To make sure your changes work, the plan here is to deploy this new policy to a few selected individuals in the Teams admin centre.

Find the first user and click on their name. In the details page you will see the policies applied to the lower left:

image

Click Edit at the top right of this section and change the App setup policy to your new policy:

image

And click Save:

image

You will see your new policy in the list.

Repeat for the rest of your test pool of users using the portal. We will not use the portal for deploying it to all users though, that will take too long!

Next day, these users should see something like this – no calendar:

image

3. Applying App Setup Policy To All Users

To apply this change to all users once your test users are happy we will use PowerShell, and we will use the Skype for Business Online PowerShell cmdlets (not the Teams PowerShell!).

The following one-line PowerShell, once you have connected to your tenant, is:

Get-CSOnlineUser | ForEach-Object { Grant-CsTeamsAppSetupPolicy -PolicyName "With OnPrem Mailboxes" -Identity $_.WindowsEmailAddress }

This gets all your users and applies a new Teams App Setup Policy to each of them. This works initially with this problem, as we assume all users are affected. If only a subset of your users are on-premises, then do not use this cmdlet to apply the initial change, but use the below to be more selective.

Within 24 hours the Calendar app will disappear from Teams for your users and they will not be phoning the help desk with issues that none of you can easily fix!

4. Applying App Setup Policy To Selected Users

The above cmdlet is a single run – it does not affect later and new users, nor is there a concept of a default policy that you can set as the one each new users gets. So every so often depending upon how often new users start employment you will want to run the below:

Get-CsOnlineUser -Filter { TeamsAppSetupPolicy -ne "With OnPrem Mailboxes" } | Grant-CsTeamsAppSetupPolicy -PolicyName "With OnPrem Mailboxes"

This gets all users where they do not have the selected App Policy already set and sets this just for these users. This is quicker than setting it for all users regardless.

You can use other filters to select users – for example, you could look for users without an on-premises mailbox and then run the ForEach against each of these users instead – this would work in a hybrid deployment.

When you are in a hybrid deployment and you move mailboxes to Exchange Online from on-premises, you will want to set those users just moved back to a policy that includes the calendar app. The same would go for organizations migrating to Exchange Server 2016 with inbound AutoDiscover from Office 365. Here you could use something like importing a CSV file of mailboxes being migrated (the same list you used to build the migration batches in the first place would do) and then run the ForEach for each item on the CSV file.

bin/ExSMIME.dll Copy Error During Exchange Patching

Posted on Leave a commentPosted in 2013, 2016, exchange, Exchange Server, update, upgrade

I have seen a lot of this, and there are some documents online but none that described what I was seeing. I was getting the following on an upgrade of Exchange 2013 CU10 to CU22 (yes, a big difference in versions):

     The following error was generated when “$error.Clear();
           $dllFile = join-path $RoleInstallPath “bin\ExSMIME.dll”;
           $regsvr = join-path (join-path $env:SystemRoot system32) regsvr32.exe;
          start-SetupProcess -Name:”$regsvr” -Args:”/s `”$dllFile`”” -Timeout:120000;
         ” was run: “Microsoft.Exchange.Configuration.Tasks.TaskException: Process execution failed with exit code 3.
    at Microsoft.Exchange.Management.Tasks.RunProcessBase.InternalProcessRecord()
   at Microsoft.Exchange.Configuration.Tasks.Task.<ProcessRecord>b__b()
    at Microsoft.Exchange.Configuration.Tasks.Task.InvokeRetryableFunc(String funcName, Action func, Boolean terminatePipelineIfFailed)”.

The Exchange Server setup operation didn’t complete. More details can be found
in ExchangeSetup.log located in the <SystemDrive>:\ExchangeSetupLogs folder.

In this error the file ExSMIME.dll fails to copy. You can find the correct copy of this file in the CU source files at …\CU22\setup\serverroles\common. I copy the ExSMIME.dll file from here directly into the \Program Files\Microsoft\Exchange Server\v15\bin folder and then restart the upgrade.

I have found that the upgrade fails again here if it things there is a pending reboot due to other installations and I have also seen at this point the detection for the VC++ runtime fails. I have documented this elsewhere, and the workaround for the is found at https://c7solutions.com/2019/02/exchange-server-dependency-on-visual-c-failing-detection.

A reboot later and the installation is successful. The error somehow seems to think that the file is not where it is looking for it. In the ExchangeSetup.log file it records the issue as “Error 3”, which generally means “not found”!

Exchange Server Object ID Error With Windows Server 2016 Domain Controllers

Posted on 1 CommentPosted in 2010, 2013, 2016, active directory, ADDS, error, Exchange Server

Saw this error the other day:

image

When you open Exchange Control Panel and view the Mailbox Delegation tab of any user account you get the following:

The object <name> has been corrupted, and it’s in an inconsistent state. The following validation errors happened: The access control entry defines the ObjectType ‘9b026da6-0d3c-465c-8bee-5199d7165cba’ that can’t be resolved..

You do not see this error on any mailboxes that you have moved to Office 365 in hybrid mode, that is you do not see it on any RemoteMailbox objects.

The issue is because ObjectType ‘9b026da6-0d3c-465c-8bee-5199d7165cba’ is the GUID of the DS-Validated-Write-Computer Control Access Right introduced in WS2016 AD DS which is new to your Active Directory upon installing your first 2016 domain controller. Exchange Server reads this access control list when you open the Mailbox Delegation tab in Exchange Control Panel or when you run Get-ADPermission on the mailbox. This error is cosmetic, but to remove it you just need to reboot all your Exchange Servers in turn (relying on your database availability groups and load balancers to maintain service). Once you have rebooted each server, the error goes away when you are connected to that server for administrative functions. There is no impact on user connectivity whilst this error is in place, though it may impact you ability to assign permissions without error.

Therefore recommend that you reboot one server as soon as you can and then use that server as your target for administration until you can reboot the remaining servers.

Exchange Edge Server and Common Attachment Blocking In Exchange Online Protection

Posted on Leave a commentPosted in 2007, 2010, 2013, 2016, Edge, EOP, exchange, exchange online, Exchange Online Protection, FOPE, IAmMEC, Office 365

Both Exchange Server Edge role and Exchange Online Protection have an attachment filtering policy. The default in Edge Server is quite long, and the default in EOP is quite short. There is also a few values that are common to both.

So, how do you merge the lists so that your Edge Server attachment filtering policy is copied to Exchange Online in advance of changing your MX record to EOP?

You run

Set-MalwareFilterPolicy Default -FileTypes ade,adp,cpl,app,bas,asx,bat,chm,cmd,com,crt,csh,exe,fxp,hlp,hta,inf,ins,isp,js,jse,ksh,lnk,mda,mdb,mde,mdt,mdw,mdz,msc,msi,msp,mst,ops,pcd,pif,prf,prg,ps1,ps11,ps11xml,ps1xml,ps2,ps2xml,psc1,psc2,reg,scf,scr,sct,shb,shs,url,vb,vbe,vbs,wsc,wsf,wsh,xnk,ace,ani,docm,jar

This takes both the Edge Server default list and the EOP default list, minus the duplicate values and adds them to EOP. If you have a different custom list then use the following PowerShell to get your two lists and then use the above (with “Default” being the name of the policy) PowerShell to update the list in the cloud

Edge Server: Get-AttachmentFilterEntry

EOP: $variable = Get-MalwareFilterPolicy Default
$variable.FileTypes

Photos, Exchange, And The File System

Posted on 1 CommentPosted in 2013, 2016, Exchange Server, Office 365, owa

On an Exchange 2013 and later server this is a folder called photos that gets created after installation and can contain a couple of user photos for some of your users. How does it get there and what does it contain?

The photos folder appears (on 2016 anyway) when the user uploads a photo (via OWA). Two images are created one 96square and the other 648square. They are made in a folder unique to the user and on the mailbox server that contains their active mailbox at the time of upload.

To reproduce this, login to OWA. Determine which server is currently the active server for that mailbox and then access the file system of that server. You are looking for “C:\Program Files\Microsoft\Exchange Server\V15\ClientAccess” though it will be wherever Exchange Server was installed if not the C: drive. If anyone has uploaded photos already via this server then you will see a folder named photo. You can delete this folder without impact (unless someone is actively uploading a photo at that exact time).

In OWA, click the photo icon top right and then click Change:

image

image

Click Upload photo and select a photo. I’ve used the sample pictures that are installed on Windows 7 in this example:

image

At this point a copy of the photo is uploaded to a web service on Exchange Server. Click Save above your chosen photo. At this point the photo folder in the ClientAccess folder on the server that is active for your mailbox is created. Inside this folder you will see a subfolder called _domain.com-UNIQUEID. Inside this folder will be two subfolders called HR96x96 and HR648x648. Inside each of these will be the JPEG file that was created at the time of saving the upload. The size of each will match the folder name and the name of the file will be _Alias-UNIQUEID. If the user deletes their photo then a 0 byte JPEG file will be created in the folder.

Note that these two photos are not a cache of the photo for the Exchange Server to download to other users. They are just used during uploading the photos. Once uploaded they are resized using this file system location and then stored in their respective locations. The 96×96 photo (at less than 100Kb) is stored in the Active Directory and the 648×648 image is stored in the Exchange 2013 or later mailbox for use by Exchange, Skype for Business and SharePoint.

If there are policies and privacy laws that state the caching of images on the file system must be avoided, then you should be able to delete the photo upload cache at your convenience.

The photo folder does not appear on another server when viewing that user with a photo in their mailbox. Requesting the photo is done via owa/service.svc and not AFAIK from a file on the file system.

Deleting the folder after the fact did not impact my test users photo (as its now in the mailbox and not read from the file system). If this mailbox is later migrated to Office 365, then the photo will migrate with the mailbox as it is part of the mailbox. If the photo stored in AD is less than 100kb then it will be synced to Azure AD.

Installing Office 365 ProPlus Click To Run Via Group Policy

Posted on 114 CommentsPosted in 2016, Click To Run, Group Policy, Office, Office 365, Office 365 ProPlus

Note: Article updated October 2018 to remove references to “Office 2016” and replace it with “Office 365 ProPlus” as the rollover to the 2019 release is seamless and does not change this products name.

Note: Article updated April 2018 to support the new Channel names and XML updates

Office 365 ProPlus Click To Run (which comes with Office 365 subscriptions) can be deployed via Group Policy, but there are a few things that you need to know and do first. These are:

  1. You cannot use the “Software Installation” features of GPO’s to deploy the Office 2016 click to run software as this is an exe file, and “Software Installation” runs MSI files.
  2. You cannot run software with elevated installation rights, as the setup.exe shells out to other processes to run the installation (the officeclick2run.exe service).
  3. You cannot just drop the latest versions of the files in an existing 2013 deployment folder and expect the clients to update automatically – you must install 2016 to upgrade it and install it for the first time. This is not the case with the “2019” release. An existing installation installed before the “2019” release in October 2018 will seamlessly move to this version at the next applicable update.

Therefore you need to deploy the software via a computer startup script. But this is not simple either as startup scripts run each time the computer starts up (obviously!) but will run regardless of whether the software is already installed. Therefore you need to run the installation by way of a startup script that first checks if Office 365 ProPlus click to run has already been installed or not.

To do this you need to following:

  1. A read only file share containing the Office 2016 click to run files. Not this folder should not be the folder that already contains the Office 2013 files if you have them on your network.
  2. A read/write file share to store log files on (the deployment script logs the start and completion of the installation in a central location)
  3. An XML file to install Office 365 ProPlus click to run customised to your environment and the fact that you are using GPO deployment
  4. A batch file to detect an existing Office 365 ProPlus click to run deployment and if not present to install Office 365 ProPlus click to run from your file share.
  5. And finally the Office Deployment Tool setup program. This software has been updated a few times over the years, so ensure you download the current version before starting.

Steps 1 and 4 are part of a standard Office 365 ProPlus click to run deployment process and so not covered in this blog post. But once you have downloaded the Office Deployment Tool and created the XML using the creator tool at https://config.office.com you have your configuration.xml file. In step 3 you can run the deployment tool with setup.exe /download configuration.xml to download the Office binaries to the file share mentioned in step 1. If you have Office 2013 already deployed via this method (see http://c7solutions.com/2014/09/installing-office-365-proplus-click-to-run-via-gpo-deployment for these steps) then make sure that this folder for the binaries is not the same folder as contains 2013 files. The Office installer for Office 2013 Click To Run creates a subfolder called Office then another subfolder called Data. Into this it places v32.cab (or v64.cab) and other files. This cab file contains info relating to the version number of the software in this folder and if you download the current version to the same folder it will replace this file, but 2013 installed machines will still try and upgrade from this folder and fail. Therefore create another folder. This is shown in the example scripts below.

So here are the steps and details for doing all this for GPO deployment:

Creating Deployment File Shares

Create a software deployment file share that you have read/write access to and everyone else read only and create a folder called Office365ProPlus inside this to store the binaries.

Create a second file share that everyone has read/write access to (or CREATOR OWNER has write so that only the creator of the file can write it to the share and others can read or not see it at all). Create a sub folder in InstallLogs called Office365ProPlus.

In my demo these two shares and subfolders are called \\server\Software\Office365ProPlus and \\server\InstallLogs\Office365ProPlus.

Create an XML File for Office 2016 Click to Run Deployment

This XML file is as follows and is saved to \\server\Software\Office365ProPlus root folder. Call this file config.xml. You can create this XML file using the wizard at https://config.office.com

<Configuration>
<Add SourcePath="\\server\Software\Office365ProPlus\" OfficeClientEdition="32" Channel="Broad" AllowCdnFallback="True" ForceUpgrade="TRUE">
  <Product ID="O365ProPlusRetail">
    <Language ID="en-us" />
    <Language ID="MatchOS" Fallback="en-us" />
  </Product>
</Add>
<Updates Enabled="TRUE" UpdatePath="\\server\Software\Office365ProPlus\" Channel="Broad"/>
<Display Level="None" AcceptEULA="TRUE" />
<Logging Level="Standard" Path="%temp%" />
<Property Name="PinIconsToTaskbar" Value="TRUE" />
</Configuration>

The important entries of “no display” and the “Extended User Licence Agreement” having been accepted are important, as GPO deployment works as a system service and so cannot display anything to the screen. Also see http://technet.microsoft.com/en-us/library/jj219426(v=office.15).aspx for the XML reference file for other settings you can contain here such as updates from the Internet (UpdatePath=””) or no updates (Updates Enabled=”FALSE”), the Channel value names and multiple languages (add more <Language ID=”xx-xx” /> nodes to the file) or to match the language to the OS (MatchOS, with Fallback language and allowing internet download if the language is not available with AllowCdnFallback), etc.

Download the Office ProPlus Click to Run Binaries

Download the Office Deployment Tool from http://www.microsoft.com/en-us/download/details.aspx?id=49117 and if you downloaded this a few months ago, download it again as it changes frequently and improves the setup process.

Install this software to get setup.exe and some example XML files. Copy setup.exe to \\server\Software\OfficeProPlus.

Run \\server\Software\Office365ProPlus\setup.exe /download \\server\Software\Office365ProPlus\config.xml to download the latest version (or the specified version if you have added Version=”15.1.2.3″ to config.xml where 15.1.2.3 is the build number you want to install). This will create the Office\Data folder in the \\server\Office365ProPlus share and download the binaries and any languages specified in the XML to that location – do not modify the folder structure as the Office Deployment Tool will expect this structure to find the files under during installation.

Create A CMD File To Script The Install

In Notepad create a cmd file and save it to <strong\\server\Office365ProPlus as well. It will eventually go in the GPO folder location, but this will be your master copy. The cmd file will look like the following and for this demo is called _InstallOffice365ProPlusGPO.cmd

setlocal
REM *********************************************************************
REM Environment customization begins here. Modify variables below.
REM *********************************************************************
REM Set DeployServer to a network-accessible location containing the Office source files.
set DeployServer=\\server\Software\Office365ProPlus
REM Set ConfigFile to the configuration file to be used for deployment (required)
set ConfigFile=\\server\Software\Office365ProPlus\config.xml
REM Set LogLocation to a central directory to collect script log files (install log files are set in XML file).
set LogLocation=\\server\InstallLogs\Office365ProPlus
REM *********************************************************************
REM Deployment code begins here. Do not modify anything below this line (check quotes are quotes though).
REM *********************************************************************
IF NOT "%ProgramFiles(x86)%"=="" (goto ARP64) else (goto ARP86)
REM Operating system is X64. Check for 32 bit Office in emulated Wow6432 registry key
:ARP64
reg query HKLM\SOFTWARE\WOW6432NODE\Microsoft\Office\16.0\ClickToRunStore\Packages\{9AC08E99-230B-47e8-9721-4577B7F124EA}
if NOT %errorlevel%==1 (goto End)
REM Check for 32 and 64 bit versions of Office 2013 in regular registry key.(Office 64bit would also appear here on a 64bit OS)
:ARP86
reg query HKLM\SOFTWARE\Microsoft\Office\16.0\ClickToRunStore\Packages\{9AC08E99-230B-47e8-9721-4577B7F124EA}
if %errorlevel%==1 (goto DeployOffice) else (goto End)
REM If 1 returned, the product was not found. Run setup here.
:DeployOffice
echo %date% %time% Setup started. >> %LogLocation%\%computername%.txt
pushd "%DeployServer%"
start /wait setup.exe /configure "%ConfigFile%"
echo %date% %time% Setup ended with error code %errorlevel%. >> %LogLocation%\%computername%.txt
REM If 0 or other was returned, the product was found or another error occurred. Do nothing.
:End
Endlocal

This will be run by GPO and at computer startup look for the Click To Run registry key that indicates Office has been installed. If not found for 64 or 32 bit OS’s and 64 or 32 bit installations of Office then it will deploy office.

Create A Group Policy Object

Create in your domain a GPO object over an OU that contains the computers you want to install Office 2016 click to run on. This will run on all computers in this OU, so start with a test OU containing one or a few computers or use permissions to lock the GPO object down to specific computer accounts.

In this GPO set the following:

  1. A startup script that runs _InstallOffice365ProPlusGPO.cmd. A startup script will have a folder the script is located in (click Show Files button in the GPO editor) and copy the above cmd file from the Office deployment share to this folder.
  2. Then click Add and select the file – there are no script parameters.
  3. Your GPO object will look like this.
    image
  4. In Adminstrative Templates/System/Scripts set the Maximum wait time for Group Policy scripts to 1800 seconds. This is 30 minutes. The default is 10 minutes (600 seconds) but I have found Office installs take just over ten minutes on a LAN and longer if the fileshare is remote to the client computer. The script will be cancelled if it takes over 30 minutes, so you may need a higher value for your network.

Deploy Office 365 ProPlus Click to Run Click To Run

Run gpupdate /force on a test computer that is under the scope of your GPO object and then reboot the computer. The installation will start automatically and Office will be ready to use a few minutes after reboot. Office takes about 10 minutes to fully install on a LAN but can be used about 2 or 3 minutes after installation starts. Though in my lab with a low resourced file server it took 30 minutes to install. Do not reboot the PC in that time.

Check \\server\InstallLogs\Office365ProPlus for a file named after the computer. This will have two lines, one for the start of the deployment and one at the end (with “Setup ended with error code 0” if successful).