451 4.7.0 Temporary server error. Please try again later. PRX2

Posted on 1 CommentPosted in DNS, error, Exchange Server

There are a few articles online about this error, but none were correct for the scenario i found a clients network in.

Not that I think the specifics matter, but this was Exchange Server 2016, Windows Domain Controllers running 2012 R2 and Exchange Hybrid. All the mailboxes had already moved to the cloud and the Exchange Server is used for attribute management and SMTP relay.

Sometimes, randomly it would seem, the applications fail to send email and get back the above error. So what does it mean! Lets dive into the Exchange logs to find out more.

In my example, TCP 25 is listening on a number of separate IPs on two different network cards on a server hosted in Azure (maybe all that matters for this case?)

Protocol Logs (Frontend)

In the Exchange Transport logs I turned on Protocol Logging for all connectors and sent some emails and had them rejected with the PRX2 error in the title. After 5 or so minutes the protocol logs contained the erroring session as shown below:

2019-01-31T13:45:09.477Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,0,10.10.10.16:25,10.150.14.108:59877,+,,
2019-01-31T13:45:09.478Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,1,10.10.10.16:25,10.150.14.108:59877,>,220 COMPANY Relay Connector SERVER,
2019-01-31T13:45:09.479Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,2,10.10.10.16:25,10.150.14.108:59877,<,HELO,
2019-01-31T13:45:09.479Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,3,10.10.10.16:25,10.150.14.108:59877,>,250 SERVER.internal.co.uk Hello [10.150.14.108],
2019-01-31T13:45:09.480Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,4,10.10.10.16:25,10.150.14.108:59877,<,MAIL FROM: <appserver@international.com>,
2019-01-31T13:45:09.480Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,5,10.10.10.16:25,10.150.14.108:59877,*,08D68772EDC476C6;2019-01-31T13:45:09.477Z;1,receiving message
2019-01-31T13:45:09.480Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,6,10.10.10.16:25,10.150.14.108:59877,>,250 2.1.0 Sender OK,
2019-01-31T13:45:09.482Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,7,10.10.10.16:25,10.150.14.108:59877,<,RCPT TO: <internal.user@international.com>,
2019-01-31T13:45:09.482Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,8,10.10.10.16:25,10.150.14.108:59877,>,250 2.1.5 Recipient OK,
2019-01-31T13:45:09.483Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,9,10.10.10.16:25,10.150.14.108:59877,<,RCPT TO: <brian@nbconsult.co>,
2019-01-31T13:45:09.483Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,10,10.10.10.16:25,10.150.14.108:59877,>,250 2.1.5 Recipient OK,
2019-01-31T13:45:09.484Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,11,10.10.10.16:25,10.150.14.108:59877,<,DATA,
2019-01-31T13:45:09.484Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,12,10.10.10.16:25,10.150.14.108:59877,>,354 Start mail input; end with <CRLF>.<CRLF>,
2019-01-31T13:45:09.498Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,13,10.10.10.16:25,10.150.14.108:59877,*,,Proxy destination(s) obtained from OnProxyInboundMessage event. Correlation Id:80e0d560-be23-4910-bcb0-43139bee131f
2019-01-31T13:45:09.501Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,14,10.10.10.16:25,10.150.14.108:59877,*,,Message or connection acked with status Retry and response 451 4.4.0 DNS query failed. The error was: DNS query failed with error InfoNoRecords -> DnsQueryFailed: InfoNoRecords
2019-01-31T13:45:09.501Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,15,10.10.10.16:25,10.150.14.108:59877,>,451 4.7.0 Temporary server error. Please try again later. PRX2 ,
2019-01-31T13:45:09.503Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,16,10.10.10.16:25,10.150.14.108:59877,<,QUIT,
2019-01-31T13:45:09.503Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,17,10.10.10.16:25,10.150.14.108:59877,>,221 2.0.0 Service closing transmission channel,
2019-01-31T13:45:09.503Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,18,10.10.10.16:25,10.150.14.108:59877,-,,Local

The protocol logs contain a number of columns to the left. The interesting ones for this are the connector name (“SERVER\From Internal Servers (Relay)”), the session ID (08D68772EDC476C6) and the sequence number (each item on the protocol has a incrementing sequence number, in the above it goes from 0 where the session connects (which is the + at the end) to 18, where it disconnects (the – at the end of the last line).

This log looks no different from a session that works (as it was random as I said above), but we see more about the error. Specifically we see the following:

Proxy destination(s) obtained from OnProxyInboundMessage event. Correlation Id:80e0d560-be23-4910-bcb0-43139bee131f
Message or connection acked with status Retry and response 451 4.4.0 DNS query failed. The error was: DNS query failed with error InfoNoRecords -> DnsQueryFailed: InfoNoRecords
451 4.7.0 Temporary server error. Please try again later. PRX2 ,

So we see that it is DNS. Online there are articles about this being to do with IPv6, AAAA records and invalid responses to those queries and fixes include using external DNS settings or smarthost values. None of this worked in this example.

So lets follow down the logs some more

Connectivity Logs

In the connectivity logs we search the same date/time/hour log for the session number, which in this case is 08D68772EDC476C6 from the above logs. In the connectivity logs we see a session that matches for this ID and its for “internalproxy”

2019-01-31T13:45:09.499Z,08D68772EDC476C7,SMTP,internalproxy,+,Undefined 00000000-0000-0000-0000-000000000000;QueueLength=<no priority counts>. Starting outbound connection for inbound session 08D68772EDC476C6
2019-01-31T13:45:09.501Z,08D68772EDC476C7,SMTP,internalproxy,>,DNS server returned InfoNoRecords reported by 10.10.10.21. [Domain:Result] = SERVER.internal.co.uk:InfoNoRecords;
2019-01-31T13:45:09.501Z,08D68772EDC476C7,SMTP,internalproxy,-,Messages: 0 Bytes: 0 (The DNS query for  'Undefined':'internalproxy':'00000000-0000-0000-0000-000000000000' failed with error : InfoNoRecords)

Internalproxy is what Exchange users to send email from the frontend transport service to the hub transport service. But which hub transport service are we going to use? If does not matter if you have 1 or x number of Exchange Servers in your site, it will use DNS to look up the IP of one of these servers. So even if you have a single Exchange box, DNS is vital.

In the above log we see that DNS 10.10.10.21 returns InfoNoRecords when queried for the Exchange Servers own name.

So I resort to nslookup to check DNS from this Exchange server. I have two DNS server, .20 and .21. The error appears to be related to .21 in this case.

To I enter “nslookup server.internal.co.uk 10.10.10.21” which means look up the name of the server using the DNS server 10.10.10.21. I got back a message saying cannot find server.internal.co.uk: Query refused.

When I tried the other DNS server I got back a successful response and the IP address of the server.

So for immediate fix, I removed 10.10.10.21 as an option for DNS for this server. Exchange immediately went back to work and PRX2 errors where not displayed and email got to its destination.

Now to go and see who has broken DNS!

CannotEnterFinalizationTransientException On Exchange Move Request

Posted on Leave a commentPosted in error, exchange, exchange online, Exchange Server, migration, move

Did not find a lot on the internet on this particular error, so I guess it does not happen very often, but in my case it delayed to move of the mailbox in question by a week or so until I could resolve it.

When a mailbox is moving to a different Exchange organization (cross-forest or to/from Exchange Online) the move process copies the mailbox data to the target database and then right at the end of the move updates Active Directory in both the source and target forest. In the source it changes the object type from mailbox to mailuser (or remotemailbox if Exchange Online is in play, though this is really a special form of mailuser) and in the target, updates the mailuser to become a mailbox.

This particular error occurs at this stage. The Get-MoveRequest cmdlet reports Failed as the status, and Get-MoveRequestStatistics reports FailedOther as the status. If you get the move logs (Get-MoveRequestStatistics <name> -IncludeReport | FL | Out-File <filename.txt>) then in the logs you will see CannotEnterFinalizationTransientException as the error repeated many times until the move fails.

The fix for this issue is as follows:

1. Check that the Exchange System account has permission to the Active Directory object in question. In Active Directory Users and Computers choose View > Advanced to enable the Security tab and then view the security tab on the object in question. Edit > Advanced and then check or click “Enable Inheritance” option or button (depending upon version of AD tools). If inheritance is already set to enabled there is probably no harm in disabling inheritance, copying permissions and then enabling inheritance again.

2. Move the mailbox to a different database in the source Exchange Organization (New-MoveRequest <name>) and waiting for that to complete.

3. Removing and restarting the move in the target forest. If you do not remove and restart the move in the target you will see both MailboxIsNotInExpectedMDBPermanentException and SourceMailboxAlreadyBeingMovedTransientException. The first of these is because the mailbox is not where the target move expects it to be, and the second of these is becuase the source is currently being moved and so cannot be moved to the correct target forest at the same time.

This should resolve your ultimate move request – it did for me! 

Exchange Server Object ID Error With Windows Server 2016 Domain Controllers

Posted on 1 CommentPosted in 2010, 2013, 2016, active directory, ADDS, error, Exchange Server

Saw this error the other day:

image

When you open Exchange Control Panel and view the Mailbox Delegation tab of any user account you get the following:

The object <name> has been corrupted, and it’s in an inconsistent state. The following validation errors happened: The access control entry defines the ObjectType ‘9b026da6-0d3c-465c-8bee-5199d7165cba’ that can’t be resolved..

You do not see this error on any mailboxes that you have moved to Office 365 in hybrid mode, that is you do not see it on any RemoteMailbox objects.

The issue is because ObjectType ‘9b026da6-0d3c-465c-8bee-5199d7165cba’ is the GUID of the DS-Validated-Write-Computer Control Access Right introduced in WS2016 AD DS which is new to your Active Directory upon installing your first 2016 domain controller. Exchange Server reads this access control list when you open the Mailbox Delegation tab in Exchange Control Panel or when you run Get-ADPermission on the mailbox. This error is cosmetic, but to remove it you just need to reboot all your Exchange Servers in turn (relying on your database availability groups and load balancers to maintain service). Once you have rebooted each server, the error goes away when you are connected to that server for administrative functions. There is no impact on user connectivity whilst this error is in place, though it may impact you ability to assign permissions without error.

Therefore recommend that you reboot one server as soon as you can and then use that server as your target for administration until you can reboot the remaining servers.

Slow Virtualization Networking

Posted on 1 CommentPosted in error, ip, ipv4, networking, virtualisation

I had a complaint from a customer today that there new virtual server with lots of resources was too slow on copying files and could I take a look.

So I did! The virtual server was connected to a 1GB switch via two different network cables. One was configured in the Hyper-V virtualization software as available for the guest and one for the host. In the host I could copy a test file across the network from a file server at close to the expected speed for the network:

image

Copying the same file from inside a virtual machine on that same host was considerably slower:

image

Notice the speeds. 85.3 MB/s compared to 355 KB/s!!!

I tried a tracert to the same file server. From the host I got  1    <1 ms    <1 ms    <1 ms  fileserver.domain.local[192.168.1.2] but from the guest I got   1    <200 ms    <201 ms    <38 ms  fileserver.domain.local[192.168.1.2]. Every host to file server tracert had a latency in excess of what I expect of some of the internet!

So how to fix…

On the host I checked which physical network card was being used by the Hyper-V guest (Virtual Switch Manager in the Hyper-V tools) and found it was Broadcom NetXtreme Gigabit Network #4 in this case. So in the host I brought up the properties of that same network card (Network and Sharing Center > Change adapter settings > and hover over each NIC to see the hardware description).

image

Bring up the properties of this network card and click Configure to bring up the configuration of the network card hardware for this specific NIC.

Change to the Advanced tab and select Virtual Machine Queues. Change this to Disabled.

image

Click OK. The network card will reset and come back online in about 3 to 5 seconds. The guest machine will be offline for this time as well. If you are changing the network card that the host uses as well, you will be disconnected from the host for this time as well (though you will be warned about this in the Hyper-V management tools).

Results when I connected to the guest again:

image

Now this is half the speed of the host – and I did not find on this machine other network settings to improve this, so I guess this is the impact of using virtualization over physical machines.

ERROR_REPLICA_SYNC_FAILED_THE TARGET PRINCIPAL NAME IS INCORRECT

Posted on Leave a commentPosted in 2003, 2007, active directory, error, exchange, kerberos, virtual pc, virtual server

This rather imposing message is found if you try to force replication between to Active Directory Domain Controllers when one of the controllers machine account password is out of sync with the password as stored on the other domain controller.

I have seen this a number of times on Virtual PC or Virtual Server Active Directory deployments with more than one DC in the virtual environment.

So, how do you fix it:

  1. On the DC that is broken (the one that when using replmon reports the error above) set the Kerberos Key Distribution Center Service to manual and stop the service.
  2. From a command prompt on the broken DC enter the following:
    netdom resetpwd /s:name_of_working_DC /ud:domain\user /pd:*
    where domain\user is an administrator of the domain in the domain_name\user_name format. You will be prompted to enter your password.
  3. Upon pressing Enter, if the command fails then restart the broken DC and repeat the above command (this restart clears the Kerberos ticket cache and so clears the broken credential attempts that it has stored).
  4. Upon successful completion of the command in step 2 restart the broken DC. You must do this even if done already in step 3.
  5. Check that replication is working, and if so restart the Kerberos Key Distribution Center Service and set the service back to automatic.

This is a summary of Microsoft Knowledgebase Article 325850, with some more specific detail mentioned.