Slow Virtualization Networking

Posted on Leave a commentPosted in error, ip, ipv4, networking, virtualisation

I had a complaint from a customer today that there new virtual server with lots of resources was too slow on copying files and could I take a look.

So I did! The virtual server was connected to a 1GB switch via two different network cables. One was configured in the Hyper-V virtualization software as available for the guest and one for the host. In the host I could copy a test file across the network from a file server at close to the expected speed for the network:

image

Copying the same file from inside a virtual machine on that same host was considerably slower:

image

Notice the speeds. 85.3 MB/s compared to 355 KB/s!!!

I tried a tracert to the same file server. From the host I got  1    <1 ms    <1 ms    <1 ms  fileserver.domain.local[192.168.1.2] but from the guest I got   1    <200 ms    <201 ms    <38 ms  fileserver.domain.local[192.168.1.2]. Every host to file server tracert had a latency in excess of what I expect of some of the internet!

So how to fix…

On the host I checked which physical network card was being used by the Hyper-V guest (Virtual Switch Manager in the Hyper-V tools) and found it was Broadcom NetXtreme Gigabit Network #4 in this case. So in the host I brought up the properties of that same network card (Network and Sharing Center > Change adapter settings > and hover over each NIC to see the hardware description).

image

Bring up the properties of this network card and click Configure to bring up the configuration of the network card hardware for this specific NIC.

Change to the Advanced tab and select Virtual Machine Queues. Change this to Disabled.

image

Click OK. The network card will reset and come back online in about 3 to 5 seconds. The guest machine will be offline for this time as well. If you are changing the network card that the host uses as well, you will be disconnected from the host for this time as well (though you will be warned about this in the Hyper-V management tools).

Results when I connected to the guest again:

image

Now this is half the speed of the host – and I did not find on this machine other network settings to improve this, so I guess this is the impact of using virtualization over physical machines.

Secret NSA Listening Ports in Exchange Server 2013? Of Course Not…

Posted on Leave a commentPosted in 2013, Edge, exchange, firewall, IAmMEC, iis, networking, powershell, transport

But what do those extra ports in Exchange Server 2013 that are listening actually do.

If you bring up a command prompt on an Exchange Server 2013 machine and run netstat –ano | find “:25”. You will get back a list of IP addresses that are listening on any port starting 25. The last number on the line is the process ID for that listening port. So for Mailbox only role servers you are interested in the row that shows 0.0.0.0:25 and for a multirole (CAS and Mailbox) server, the row that shows 0.0.0.0:2525 as shown:

image

Above you can see that process 21476 is listening on 2525 and as this is a multirole server this process ID will be EdgeTransport.exe – you can verify this in Task Manager if you want.

Repeat the netstat cmd, this time for the process ID you have selected: netstat –ano | find “xxxxx” where xxxxx is the process ID for EdgeTransport.exe, as shown:

image

You will now see that EdgeTransport.exe is listening in on a range of ports for both IPv4 and the same ports for IPv6. These ports are 25 or 2525, which is used by 2007 or 2010 Edge role servers, or by other 2007/2010 Hub Transport role servers or by other 2013 Mailbox role servers or by 2013 CAS role servers to send emails to this server via SMTP. Port 465 is the port that 2013 CAS servers proxy authenticated SMTP connections to that they receive on port 587 from mail clients. But what about port 29952 in my example (and on your servers a different port) which changes each time the service is restarted?

If you do netstat again just for this port you will see something like the following:

image

This shows that nothing is connected currently to these ports, and so they seem to be doing nothing. But if on a different Exchange 2013 server you do some viewing of the transport queues then these ports will start to show some activity.

On a different server in my environment I ran Get-Queue –Server remoteservername. If I do this on the local server, then nothing special happens as Exchange does not need to connect to these ports, but if it is run from a different server and I ask it to show the queue on the first server that we have been looking at above, then these ports become used:

image

image

Above we can see Exchange Management Shell on mail5 connecting to mail4 (the original server in this blog post). The second picture from mail4 shows that port 29952 have received a connection from the IPv6  address of mail5 and specifically from port 65172 on that remote server.

If I look finally at the second server in this exercise and see what process is connecting from port 65172 (and again, your ports will be different) I see that process 15156 is doing this (the process ID is the last column in the output)

image

Taking this process ID to Task Manager, I see process 15156 is the IIS Worker Process, which is the process that PowerShell connects to to do its work.

image

Therefore, the random and changing port that EdgeTransport.exe listens on is nothing to do with PRISM, but all to do with your management of remote queues.

Installing and Configuring AD RMS and Exchange Server

Posted on 2 CommentsPosted in 2007, 2008 R2, 2010, active directory, certificates, exchange, exchange online, microsoft, networking, Office 365, organization relationships, owa, rms, server administrator

Earlier this week at the Microsoft Exchange Conference (MEC 2012) I led a session titled Configuring Rights Management Server for Office 365 and Exchange On-Premises [E14.314]. This blog shows three videos covering installation, configuration and integration of RMS with Exchange 2010 and Office 365. For Exchange 2013, the steps are mostly identical.

Installing AD RMS

This video looks at the steps to install AD RMS. For the purposes of the demonstration, this is a single server lab deployment running Windows Server 2008 R2, Exchange Server 2010 (Mailbox, CAS and Hub roles) and is the domain controller for the domain. As it is a domain controller, a few of the install steps are slightly different (those that are to do with user accounts) and these changes are pointed out in the video, as the recommendation is to install AD RMS on its own server or set of servers behind a IP load balancer.

Configuring AD RMS for Exchange 2010

The second video looks at the configuration of AD RMS for use in Exchange. For the purposes of the demonstration, this is a single server lab deployment running Windows Server 2008 R2, Exchange Server 2010 (Mailbox, CAS and Hub roles) and is the domain controller for the domain. This video looks at the default ‘Do Not Forward’ restriction as well as creating new templates for use in Exchange Server (OWA and Transport Rules) and then publishing these templates so they can be used in Outlook and other Microsoft Office products.

 

Integrating AD RMS with Office 365

The third video looks at the steps needed to ensure that your Office 365 mailboxes can use the RMS server on premises. The steps include exporting and importing the Trusted Publishing Domain (the TPD) and then marking the templates as distributed (i.e. available for use). The video finishes with a demo of the templates in action.

Creating GeoDNS with Amazon Route 53 DNS

Posted on 3 CommentsPosted in 2013, cloud, exchange, GeoDNS, https, load balancer, mcm, microsoft, MX, networking, owa, smtp, transport

UPDATE: 13 Aug 2014 – Amazon Route 53 now does native GeoDNS within the product – see Amazon Route 53 GeoDNS Routing Policy

A new feature to Exchange 2013 is supported use of a single namespace for your global email infrastructure. For example mail.contoso.com rather than different ones for each region such as uk-mail.contoso.com; usa-mail.contoso.com and apac-mail.contoso.com.
GeoDNS means that you are given the IP address of a server that is in or close to the region that you are in. For example if you work in London and your mailbox is also in London then most of the time you will want to be connected to the London CAS servers as that gives you the best network response. So in Exchange 2010 you would use your local URL of uk-mail.contoso.com and if you used the others you would be told to use uk-mail.contoso.com. For GeoDNS support you use mail.contoso.com and as you are in the UK you get the IP address of the CAS array in London. When you travel to the US (occasionally) you would get the US CAS array IP address, but this CAS array is able to proxy your OWA, RPC/HTTP etc traffic to the UK mailbox servers.
The same is true for email delivery via SMTP. Email that comes from UK sourced IP addresses is on balance a statistical likelihood that it is going to the UK mailbox. So when you look up the MX record for contoso.com from a UK company you get the UK CAS array and the email gets delivered to the CAS array that is in the same site as the target mailbox. If the email is for a user in a different region and it hits the UK CAS array then it is proxied to the other region seamlessly.
GeoDNS is a feature provided by some high-scale DNS providers, but not something Amazon Web Services (AWS) Route 53 provides – so how do I configure GeoDNS with Amazon Web Services (AWS) Route 53 DNS Service?
Quite easily is the answer. Route 53 does not offer GeoDNS but does offer DNS that directs you towards the closest AWS datacentre. If your datacentres are in regions similar to AWS then the DNS redirection that AWS offers is probably accurate.
To set it up, open your Route 53 DNS console, or move your DNS to AWS (it costs $0.50/month for a zone at time of writing, AWS Route 53 pricing here) and then create your global Exchange 2013 namespace record in DNS:

  1. Click Create Record Set and enter the name. In the below example I’m using geo.c7solutions.com as I don’t actually have a globally distributed email infrastructure!
  2. Select A – IPv4 or if you are doing IPv6 select AAAA.
  3. Set Alias to No and enter the IP address of one of your datacentres
  4. Select the AWS region that is closest to this Exchange server(s) and enter a unique description for the Set ID value.
  5. The entry will look something like this:
    image
  6. Save the Record Set and create additional entries for other regions. For the purposes of this blog I have created geo.c7solutions.com in four regions with the following IP addresses:
    Region IP Address Region
    us-east-1 1.2.3.4 Northern Virginia
    us-west-1 6.7.8.9 Northern California
    eu-west-1 2.3.4.5 Ireland
    ap-northwest-1 3.4.5.6 Singapore
    sa-east-1 4.5.6.7 Sao Paulo
    ap-southeast-1 5.6.7.8 Sydney
  7. The configuration in AWS for the remaining entries looks like the following:
    imageimageimage
  8. And also, once created, it appears like this:
    image

In addition to this blog, I’ve left the record described above on my c7solutions.com DNS zone. So depending upon your location in the world, if you open a command prompt and ping geo.c7solutions.com you should get back the IP address for the AWS region closest to you, and so get back an IP that represents a Exchange resource in your global region. Of course the IP’s I have used are not mine to use and probably will not respond to ping requests – but all you need to do is see it DNS returns the IP above that best matches the region that you are in.
I wrote this blog when in a hotel in Orlando and as you can see from the image below, it returns 1.2.3.4 which is the IP address associated with us-east-1.
image
But when I connected to a server in the UK and did the same ping geo.c7solutions.com I got the following, which show GeoDNS working when equating GeoDNS to AWS Latency DNS.
image
What do you get for your regions? Add comments and let us where you are (approximately) and what region you got. If enough people respond from enough places we can see if AWS can go GeoDNS without massive cost.
[Updated 13 Nov 2012] Added Sydney (ap-southeast-1) and fake IP address of 5.6.7.8
[Updated 27 April 2013] Added Northern California (us-west-1) and fake IP of 6.7.8.9

HTTPS Load Balancer Issues with Exchange 2010 SP2

Posted on 4 CommentsPosted in 2010, citrix, exchange, https, load balancer, loadbalancer, Netscaler, networking, owa, update, upgrade

When you install Service Pack 2 (and maybe SP1 too) on Exchange 2010 it resets the SSL flag on the root directory of the IIS website. You might have removed this setting for a number of reasons, mainly to do with having a HTTP to HTTPS redirect, but it can also be removed if you are doing SSL Offloading to a load balancer and that load balancer checks the state of the client access server by doing HTTP requests for the root home page. The Citrix Netscaler is one such load balancer that has this as a default setting.

The configuration documentation for the Citrix Netscaler (found here) does not discuss changing the load balancer to use a different directory on IIS to monitor the availability of the site, so when you install SP2 for Exchange 2010 and that update resets the root directory to require SSL, your load balancer thinks the site is offline and does not pass through any traffic!

image

image

To fix this issue in the short term, just uncheck the Require SSL option on the root of the Default Web Site on each of your Client Access Servers. Your load balancer should notice within a few seconds and service will resume, for example the Citrix Netscaler checks the root directory via the monitor properties every five seconds for a HTTP success code (and not a HTTPS success code!).

To fix this issue in the long term you should make a new virtual directory on each server covered by the load balancer and get the load balancer to look at this directory to determine if the service is up or down rather than looking at the root directory. Your virtual directory will not be reconfigured by future Exchange service packs (or indeed any other application that you are load balancing that might reset the SSL option on the root directory).

To complete these steps do the following:

1. Create a folder in the inetpub directory called “monitor” or similar (in the examples below the folder is called “netscaler_monitor”).

2. Place an index.htm file in this folder that is a very simple webpage that when browsed returns the page. If you want to make the page more complex to include code (so that issues with the code are picked up by the load balancer then this is fine). A simple page would look like the following:

<html><head>
<title>Netscaler Monitor for Exchange 2010title>
head><body>
<p>This page returns a success code to the netscalers if IIS is running. This page must always work over HTTP and never require an SSL connection.p>
body>html>

3. In IIS require SSL and then uncheck require SSL – this forces a setting into the IIS config file (applicationHost.config) that says that this folder must always be over HTTP and not require SSL. If you do not do this then this folder will take the setting from the parent folder, and as we have already seen, this will cause the monitor folder to require SSL when you apply the service pack.

This SSL change will result in the following configuration at the bottom of applicationHost.config, which can be added directly to the config file rather than in IIS Manager.

    <location path="Default Web Site/netscaler_monitor">
<system.webServer>
<security>
<access sslFlags="None" />
security>
system.webServer>
location>
configuration>

4. Update your load balancer so that it has a new monitor for checking the service state on the managed machine. This monitor would be something like the following for a Citrix Netscaler, each load balancer being different. This monitor checks HEAD /netscaler_monitor/ and expects to get back a 200 status code. You need to change the folder name to match, but ensure the / is before and after the folder name.

image

5. Change the configuration for each client access server in the load balancer so that it uses the new monitor rather than the default HTTP monitor.

image

6. Save your changes to the load balancer. The next time you service pack Exchange 2010 the resetting of the SSL flag on the root directory will not cause you any issues.

Exchange Log Truncation Failure in a DAG

Posted on 2 CommentsPosted in 2010, backup, domain, exchange, networking, windows 2008, x64

Today I visited a client who had noticed that no log files had ever been removed after any backup within Exchange 2010 SP1. It was fortuitous that they had enough log disk space for about eight months of log generations. The disadvantage was that we were four months into this time period, so it was a ticking clock, and that the nightly incremental backups were taking longer and longer.

They were getting the following error in their backup datacentre:

image

Unable to communicate with the Microsoft Exchange Information Store service to coordinate log truncation for database ‘name’ due to an RPC communication failure. Error 3355379671 Extended Error: 0 and Event ID 2136 for the MSExchangeRepl service in the Application event log.

What the error does not clearly say is that the Microsoft Exchange Replication service (MSExchangeRepl) on the server in the DR site (a passive node in the DAG) needs to communicate via RPC to the Microsoft Exchange Information Store service on the server holding the active node of the database.

In the case of my client, the Exchange team is not the same people as the network team or indeed the firewall team, and these teams are in different countries. In the case of the network for this client, the Replication network for the DAG had been opened to allow RPC traffic, but the MAPI (Client) network had not.

When Exchange in the DR site needed to check which logs it could truncate (a process it performs every 15 minutes), it needs to talk to the Microsoft Exchange Information Store service on the server holding the active copy of the database, and name resolution was returning (as expected) the IP address of the server on the MAPI/Client network. This network blocked RPC between servers and so (as one of the many issues they now attribute to this problem) logs could not be truncated and Event ID 2136 was posted once per database on the passive node in the DR site. The two servers in the primary site could RPC each other, so this log is not repeated in the primary site.

To solve this log growth problem without waiting for a response from the firewall team, we added a record to the hosts file on the passive server to override DNS name resolution, and within 15 minutes 2TB of log files instantly disappeared on all servers. Name resolution was reverted to DNS and the firewall team contacted.

GoDaddy SSL Certificate Approval with TXT Records

Posted on 8 CommentsPosted in certificates, exchange, hosting, https, iis, networking, SSL

I had a bit of an issue with Go Daddy yesterday in that they took 5 days to approve a Subject Alternative Name change to a certificate, and as the usual route of adding a file to a website was unavailable to me I decided to prove ownership of the domain by the addition of a new TXT record to the domain.

Go Daddy’s instructions for doing this are only suitable for domains hosted at Go Daddy and there are no clear instructions for doing this if you do not use Go Daddy for your DNS hosting.

So how do you create an SSL approval with TXT record? You do it by creating a TXT record for a subzone. The subzone is DZC and the value of the record is the seven character string that Go Daddy sent you via email. For example dzc.domain.co.uk TXT AbCdEfG.

Once DNS has replicated to ALL of your DNS servers you can return to Go Daddy’s web form and approve your SSL certificate. You can check if all your DNS servers have your new data by using NSLookup or Dig, but preferred is the use of either of these two tools from an independent third party on the internet – for example www.kloth.net/services/nslookup.php or www.dnssy.com/lookup.php.

Configuring an SSTP VPN on Small Business Server 2008

Posted on 8 CommentsPosted in networking, pki, sbs 2008, sstp, vpn

SSL based VPN’s are great. In short it is VPN without firewall or NAT issues (both of which you get with PPTP and IPSec VPN’s). But SBS 2008 does not enable SSTP VPN’s by default. It uses RRAS, so SSTP is possible, but it is not as easy as it first looks! The following is a brief guide to the steps. Exact step by step instructions are not included, as you should be someone with RRAS and certificates experience before approaching this, and if you are not but have a business need for SSTP VPN (and who doesn’t!) then call C7 Solutions in the UK on 0845 257 1777 for help and assistance. This is not at all easy to configure and get working.

  1. Ensure that you have run the connecting to the internet wizard, and that you are using a third party certificate (as there are less steps if you do this). With the default self signed certificate SSTP will not work as the client on the internet will not be able to reach the certificate revocation location. Using the installed Certificate Services and creating your own issued certificate requires publishing to the internet the certificate revocation information and so adds steps that are not entirely necessary given that certificates are inexpensive and would cost less to buy than the time taken to go through all the extra steps needed with your own issued certificates.
  2. Enable remote access from the SBS Console > Network > Connectivity page and choose Configure a Virtual Private Network link under Connectivity Tasks on the right-hand side of the window.
  3. Add some SSTP ports to the VPN in the Routing And Remote Access management program. Right-click Ports and choose Properties and enable SSTP for remote access inbound connections and set the number of connections to a suitable number for your organization. Leave PPTP enabled as Windows XP does not support SSTP VPN tunnels (only Vista SP1 and later will do so).
  4. Create an MMC and add in the local computers Certificate snap-in. View the properties of your trusted certificate that you are using for Remote Web Workplace and note down the Thumbprint value of this certificate.
  5. Ensure that this certificate is associated with 0.0.0.0:443 and [::]:443 network bindings on the server. Type netsh http show ssl from elevated command prompt to get this information. You typically get four entries with IP:port being the first line of each. Check for IP:port reading “0.0.0.0:443” and [::]:443 as this shows the IPv4 and IPv6 mappings for SSL certificates on the server. Ignore the :8172 and :987 entries (these are for IIS Management Service and companyweb).
  6. If the certificate hash is not the same for both the remote web workplace certificate and the netsh bindings information in the previous two steps or if you are missing the IPv6 binding then you need to reset the bindings. If they are same then jump to step 7.
    a) Ensure that the certificate bound to the remote web workplace is correct. From the client machine browse to http://remote.your_domain.com. You should be automatically forwarded to https://remote.your_domain.com/remote and the login page. If you get any certificate errors during this in the web browser you must fix them now before continuing.
    b) If the certificate on the remote web workplace site is incorrect then run the Fix My Network wizard and the Set Up Your Internet Address wizard in the SBS Console (both found in the Network > Connectivity > Connectivity Task pane).
    c) Repeat the test in step a and if the certificate that is now associated with the site is incorrect also run the Add A Trusted Certificate wizard which is found in the same place as above. This step should not be needed if a trusted certificate has already been installed on the server and it matches the remote.your_domain.com name and the wizards in step b will associate the correct certificate to the website.
    d) From an elevated command prompt delete the certificate binding for IPv4 by typing netsh http delete sslcert ipport=0.0.0.0:443. The binding should be deleted successfully.
    e) From an elevated command prompt delete the certificate binding for IPv6 by typing netsh http delete sslcert ipport=[::]:443. The binding should be deleted successfully if an IPv6 binding existed, otherwise expect to see an error which can be ignored.
    f) Delete the certificate binding in the RRAS configuration by deleting these registry keys, if they exist, “HKLM\ System\ CurrentControlSet\ Services\ Sstpsvc\ Parameters\ Sha256CertificateHash” and “HKLM\ System\ CurrentControlSet\ Services\ Sstpsvc\ Parameters\ Sha1CertificateHash
    g) Connect the correct certificate to the IPv4 and IPv6 bindings by typing the following entries from an elevated command prompt where xxx is the certificate hash of the trusted certificate used for the Remote Web Workplace. netsh http add sslcert ipport=0.0.0.0:443 certhash=xxx appid={ba195980-cd49-458b-9e23-c84ee0adcd75} certstorename=MY and netsh http add sslcert ipport=[::]:443 certhash=xxx appid={ba195980-cd49-458b-9e23-c84ee0adcd75} certstorename=MY.
    h) Close any open copy of IIS Manager and restart the program. Ensure that the bindings for the SBS Web Applications site is correctly bound to your trusted remote web workplace certificate.
    i) Note that binding SSTP to the IPv4 and IPv6 listeners on port 443 will cause TS Gateway administration to display error messages (specifically that the certificate is not bound and that the IIS web site is not configured). These errors can be ignored on SBS 2008 but if you click the links to fix the errors then all will work fine. The only condition is that this fixing of errors must be done after SSTP is configured correctly (so ensure SSTP connectivity works and then come back to this step to fix). Future changes to the certificate in IIS or TS Gateway might break the SSTP binding.
  7. From a client machine browse to https://remote.your_domain.com/sra_{BA195980-CD49-458b-9E23-C84EE0ADCD75}/ and ensure that no errors occur. Note that you will not see anything in the web browser. View the properties of the certificate, specifically the CRL Distribution Points (CDP) value. Note that you should not have got any certificate errors when browsing to this site and if you did you need to resolve them before continuing further in these steps.
  8. Browse to the CDP URL in the above certificate – you should be able to reach this location on the web without error. The web browser should attempt to download the CDP file.
  9. On a Vista (SP1 or later) or Windows 7 client create a new VPN connection and in properties of the connection object choose the Security tab and ensure that the Type of VPN is set to SSTP. For regular everyday use set this to Auto, and it will find a working protocol (starting with PPTP) and so if PPTP does not work due to NAT or firewall/proxy issues SSTP will be tried and succeed (but for testing set the VPN connection specifically to SSTP). Also ensure that the name of the server you are connecting to is the same name that the certificate uses for the certificate common name.
  10. Connect the VPN and all should work. Errors regards certificate trust will appear if you have used the self issued certificate, even if you have added the certificate to your certificate store and have the certificate working in Internet Explorer. Once you have connected you can confirm from the RRAS management console that you are connected over an SSTP VPN connection. To confirm this click Ports in the RRAS management console and the active connection should be utilizing an SSTP port.
  11. Congratulate yourself on getting this far – this is not easy!