Placing Exchange 2013 Into Maintenance Mode

Posted on 7 CommentsPosted in 2008 R2, 2012, 2013, backup, exchange, hotfix, lab, load balancer, loadbalancer, update, upgrade

Updated 5 Feb 2013 to include Redirect-Message cmdlet
Exchange 2013 has a feature called Managed Availability. This feature detects issues with a server and in the event of an issue attempts to fix the component at issue. Fixes range from simple restarts of the component (for example restarting the service) to doing what is called a bugcheck. A bugcheck is forcing the server to “blue screen” and therefore reboot. Bugchecks occur when earlier simple fixes do not work. For example if the service cannot be restarted then service is moved to another node in the DAG or the Exchange 2013 aware load-balancer takes the CAS server out of service. If Managed Availability still cannot fix the server it is bugchecked.
There is one or two obvious issues with this though – the first is when you are upgrading or patching the server and the second is in a lab environment. In both these scenarios you could have servers that are considered not responsive to Managed Availability when its only because a patch or Exchange cumulative update (CU, previously known as Rollup Updates), is being installed.
This blog will discuss how to tell Managed Availability not to cause things such as reboots to happen during updates or in low spec’ed lab environments.

Patching Exchange 2013 Servers

When the patch process starts on Windows or a Cumulative Update for Exchange is installed services are stopped and possibly disabled. Disk I/O might be higher and your underlying disk subsystem might not cope well (though this is more likely to be an issue in a lab environment). The last thing you want is services being restarted, services therefore failing, and therefore Managed Availability considering that the server is dead and needs a reboot – and so in the middle of an update it blue screens.
To place a server into Maintenance Mode before you upgrade it you need to run the following Exchange Management Shell cmdlets

Maintenance Mode on Mailbox or Multi-Role Servers

Set-ServerComponentState $env:COMPUTERNAME -Component HubTransport -State Draining -Requester Maintenance

Redirect-Message -Server $env:COMPUTERNAME -Target

Suspend-ClusterNode $env:COMPUTERNAME

Set-MailboxServer $env:COMPUTERNAME -DatabaseCopyActivationDisabledAndMoveNow $True

Set-MailboxServer $env:COMPUTERNAME -DatabaseCopyAutoActivationPolicy Blocked

Set-ServerComponentState $env:COMPUTERNAME -Component ServerWideOffline -State Inactive -Requester Maintenance

Get-ServerComponentState $env:COMPUTERNAME | Format-Table Component,State -Autosize

Get-MailboxServer $env:COMPUTERNAME | Format-Table DatabaseCopy* -Autosize

Get-ClusterNode $env:COMPUTERNAME | Format-List

Maintenance Mode on CAS Servers

Set-ServerComponentState $env:COMPUTERNAME -Component ServerWideOffline -State Inactive -Requester Maintenance

For mailbox or multi-role servers step 1 should be done independent of other steps. Step 1 places the transport queues into “draining” mode, which means the server processes existing queues but does not accept new connections. Once the queue has drained, which can be checked with Get-Queue, then do steps 3 to 9. (Added 5th Feb 2013**): To speed up draining of the queues it is possible in Exchange 2013 to move the messages to another server using Redirect-Message. The Target in RedirectMessage must be an FQDN and if the Server (i.e. where the queue is sourced) is missing then the local server is used. Only active queues are moved with this command, poison and shadow queues are not moved (End of Update**). Steps 3 to 6 place the DAG node offline and move mailbox databases onto other nodes in the DAG. Steps 7 to 9 confirm these changes with a report to the screen.

Note these cmdlets all use $env:COMPUTERNAME so they run on the local machine that you want to place into Maintenance Mode. You can replace $env:COMPUTERNAME with the actual server you want to effect if you want to run the cmdlets remotely.

CAS only servers only have one step, and that is the same as step 6 in the mailbox/multi-role server process.

Ending Maintenance Mode

On a CAS server, to return to functional mode, run the following:

Set-ServerComponentState $env:COMPUTERNAME -Component ServerWideOffline -State Active -Requester Maintenance

On a mailbox or multi-role server run the following:

Set-ServerComponentState $env:COMPUTERNAME -Component ServerWideOffline -State Active -Requester Maintenance

Resume-ClusterNode $env:COMPUTERNAME

Set-MailboxServer $env:COMPUTERNAME -DatabaseCopyActivationDisabledAndMoveNow $False

Set-MailboxServer $env:COMPUTERNAME -DatabaseCopyAutoActivationPolicy Unrestricted

Set-ServerComponentState $env:COMPUTERNAME -Component HubTransport -State Active -Requester Maintenance

Once mailbox or multi-role server steps are complete you need to move databases that you want back to this server, or start maintenance on another server (as that might move databases to this server for you).

Finally note that going into maintenance mode is not an immediate step. It takes somewhere between 5 and 10 minutes (in my tests) for the Health Service to pick up these changes and implement them. Also note that where you only have one server or one DAG node available, the Health Service will not action maintenance mode as it will reduce availability to a point where service fails – for example if you only have one CAS server then the above command will not stop connections to OWA of Frontend Transport through that one CAS server.

Building Exchange 2013 Lab Environments

All of the information for managing maintenance mode above is valid for lab environments, but its also worth considering the following cmdlet:

Set-ServerComponentState $env:COMPUTERNAME -Component RecoveryActionsEnabled -State Inactive -Requester Sidelined

The above will tell Managed Availability not to do any recovery actions in the event of an issue. Therefore if your lab is (for example) slow because you are overworking the disks, then your Exchange Servers don’t blue screen and add to the load on the disk.

If you see your lab environment is regularly reporting that the server recovered from an unexpected failure then see if the following bugcheck codes are in the Event Viewer. I’ve seen these as being caused due to attempts to force a bugcheck and reboot some of my lab machines whilst I was installing Exchange on other servers on the same disk.

  • 0x000000ef (i.e. CRITICAL_PROCESS_DIED)
  • 0x00000F4 (i.e. CRITICAL_OBJECT_TERMINATION)

HTTPS Load Balancer Issues with Exchange 2010 SP2

Posted on 4 CommentsPosted in 2010, citrix, exchange, https, load balancer, loadbalancer, Netscaler, networking, owa, update, upgrade

When you install Service Pack 2 (and maybe SP1 too) on Exchange 2010 it resets the SSL flag on the root directory of the IIS website. You might have removed this setting for a number of reasons, mainly to do with having a HTTP to HTTPS redirect, but it can also be removed if you are doing SSL Offloading to a load balancer and that load balancer checks the state of the client access server by doing HTTP requests for the root home page. The Citrix Netscaler is one such load balancer that has this as a default setting.

The configuration documentation for the Citrix Netscaler (found here) does not discuss changing the load balancer to use a different directory on IIS to monitor the availability of the site, so when you install SP2 for Exchange 2010 and that update resets the root directory to require SSL, your load balancer thinks the site is offline and does not pass through any traffic!

image

image

To fix this issue in the short term, just uncheck the Require SSL option on the root of the Default Web Site on each of your Client Access Servers. Your load balancer should notice within a few seconds and service will resume, for example the Citrix Netscaler checks the root directory via the monitor properties every five seconds for a HTTP success code (and not a HTTPS success code!).

To fix this issue in the long term you should make a new virtual directory on each server covered by the load balancer and get the load balancer to look at this directory to determine if the service is up or down rather than looking at the root directory. Your virtual directory will not be reconfigured by future Exchange service packs (or indeed any other application that you are load balancing that might reset the SSL option on the root directory).

To complete these steps do the following:

1. Create a folder in the inetpub directory called “monitor” or similar (in the examples below the folder is called “netscaler_monitor”).

2. Place an index.htm file in this folder that is a very simple webpage that when browsed returns the page. If you want to make the page more complex to include code (so that issues with the code are picked up by the load balancer then this is fine). A simple page would look like the following:

<html><head>
<title>Netscaler Monitor for Exchange 2010title>
head><body>
<p>This page returns a success code to the netscalers if IIS is running. This page must always work over HTTP and never require an SSL connection.p>
body>html>

3. In IIS require SSL and then uncheck require SSL – this forces a setting into the IIS config file (applicationHost.config) that says that this folder must always be over HTTP and not require SSL. If you do not do this then this folder will take the setting from the parent folder, and as we have already seen, this will cause the monitor folder to require SSL when you apply the service pack.

This SSL change will result in the following configuration at the bottom of applicationHost.config, which can be added directly to the config file rather than in IIS Manager.

    <location path="Default Web Site/netscaler_monitor">
<system.webServer>
<security>
<access sslFlags="None" />
security>
system.webServer>
location>
configuration>

4. Update your load balancer so that it has a new monitor for checking the service state on the managed machine. This monitor would be something like the following for a Citrix Netscaler, each load balancer being different. This monitor checks HEAD /netscaler_monitor/ and expects to get back a 200 status code. You need to change the folder name to match, but ensure the / is before and after the folder name.

image

5. Change the configuration for each client access server in the load balancer so that it uses the new monitor rather than the default HTTP monitor.

image

6. Save your changes to the load balancer. The next time you service pack Exchange 2010 the resetting of the SSL flag on the root directory will not cause you any issues.

Shadow Redundancy Promotion Disabled After Exchange Server 2010 Service Pack 2

Posted on 3 CommentsPosted in 2010, exchange, update, upgrade

One of the new features added in Exchange Server 2010 Service Pack 1 was called Shadow Redundancy Promotion.

Service Pack 1 creates a setting in EdgeTransport.exe.config to control the behaviour of this setting, and it defaults to a value of “False”, so that the feature is turned off.
Enabling Shadow Redundancy Promotion is done by setting the ShadowRedundancyPromotionEnabled setting to “True” in EdgeTransport.exe.config (found in c:\program files\microsoft\exchange server\v14\bin) and restarting the MSExchangeTransport service.

When you install Service Pack 2 for Exchange Server 2010 it resets this value to false and therefore disables this setting. This is not documented (at the time of writing) in the read me for the service pack.

This does mean of course that if you have chosen to enable Shadow Redundancy within your SP1 Exchange 2010 organization, this will get turned off again when you apply the service pack. Note that other settings in EdgeTransport.exe.config are not reset on applying the service pack.
When the service pack resets the value, it does make a comment in the config file to sort of indicate what it has done. It notes the following text above the line that disables Shadow Redundancy.

<!–Exchange task Set-AppConfigValue updated value for dictionary key ShadowRedundancyPromotionEnabled from True to False at 12/5/2011 5:09:54 AM—>

It does not say it has turned your configuration off again, but that is the effect of the change that the service pack installation makes.

Exchange Management Shell

Posted on Leave a commentPosted in 2007, exchange, powershell, upgrade

I need a place to store useful Powershell commands for the administration of Exchange Server 2007, so I thought I would add them here:

Upgrading Exchange Organisation
Place Replicas of Public Folders on New Exchange Server

get-publicfolder -recurse Set-PublicFolder -Replicas:”server\public folder
database”,”server\public folder store (server)”

Enable ActiveSync Policy for Windows Mobile 2003 Smartphones and Pocket PC’s

New-ActiveSyncMailboxPolicy “Windows Mobile 2003 Users” -AttachmentsEnabled:$false -DevicePasswordEnabled:$false -AlphanumericDevicePasswordRequired:$false -PasswordRecoveryEnabled:$false -DeviceEncryptionEnabled:$false -AllowNonProvisionableDevices:$true -AllowSimpleDevicePassword:$false -DevicePasswordExpiration:unlimited -WSSAccessEnabled:$false -UNCAccessEnabled:$false

Enable ActiveSync Policy for Windows Mobile 5 Smartphones and Pocket PC’s

New-ActiveSyncMailboxPolicy “Windows Mobile 5 Users” -AttachmentsEnabled:$true -DevicePasswordEnabled:$true -AlphanumericDevicePasswordRequired:$false -PasswordRecoveryEnabled:$true -DeviceEncryptionEnabled:$false -MinDevicePasswordLength:4 -MaxInactivityTimeDeviceLock:00:15:00 -MaxDevicePasswordFailedAttempts:8 -AllowNonProvisionableDevices:$false -AllowSimpleDevicePassword:$false -DevicePasswordExpiration:unlimited -WSSAccessEnabled:$true -UNCAccessEnabled:$true

Set ActiveSync Policy Against All Users for a Given Policy

get-mailbox Set-CASMailbox -ActiveSyncMailboxPolicy:”Name Of Policy