451 4.7.0 Temporary server error. Please try again later. PRX2

There are a few articles online about this error, but none were correct for the scenario i found a clients network in.

Not that I think the specifics matter, but this was Exchange Server 2016, Windows Domain Controllers running 2012 R2 and Exchange Hybrid. All the mailboxes had already moved to the cloud and the Exchange Server is used for attribute management and SMTP relay.

Sometimes, randomly it would seem, the applications fail to send email and get back the above error. So what does it mean! Lets dive into the Exchange logs to find out more.

In my example, TCP 25 is listening on a number of separate IPs on two different network cards on a server hosted in Azure (maybe all that matters for this case?)

Protocol Logs (Frontend)

In the Exchange Transport logs I turned on Protocol Logging for all connectors and sent some emails and had them rejected with the PRX2 error in the title. After 5 or so minutes the protocol logs contained the erroring session as shown below:

2019-01-31T13:45:09.477Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,0,10.10.10.16:25,10.150.14.108:59877,+,,
2019-01-31T13:45:09.478Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,1,10.10.10.16:25,10.150.14.108:59877,>,220 COMPANY Relay Connector SERVER,
2019-01-31T13:45:09.479Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,2,10.10.10.16:25,10.150.14.108:59877,<,HELO,
2019-01-31T13:45:09.479Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,3,10.10.10.16:25,10.150.14.108:59877,>,250 SERVER.internal.co.uk Hello [10.150.14.108],
2019-01-31T13:45:09.480Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,4,10.10.10.16:25,10.150.14.108:59877,<,MAIL FROM: <appserver@international.com>,
2019-01-31T13:45:09.480Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,5,10.10.10.16:25,10.150.14.108:59877,*,08D68772EDC476C6;2019-01-31T13:45:09.477Z;1,receiving message
2019-01-31T13:45:09.480Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,6,10.10.10.16:25,10.150.14.108:59877,>,250 2.1.0 Sender OK,
2019-01-31T13:45:09.482Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,7,10.10.10.16:25,10.150.14.108:59877,<,RCPT TO: <internal.user@international.com>,
2019-01-31T13:45:09.482Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,8,10.10.10.16:25,10.150.14.108:59877,>,250 2.1.5 Recipient OK,
2019-01-31T13:45:09.483Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,9,10.10.10.16:25,10.150.14.108:59877,<,RCPT TO: <brian@nbconsult.co>,
2019-01-31T13:45:09.483Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,10,10.10.10.16:25,10.150.14.108:59877,>,250 2.1.5 Recipient OK,
2019-01-31T13:45:09.484Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,11,10.10.10.16:25,10.150.14.108:59877,<,DATA,
2019-01-31T13:45:09.484Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,12,10.10.10.16:25,10.150.14.108:59877,>,354 Start mail input; end with <CRLF>.<CRLF>,
2019-01-31T13:45:09.498Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,13,10.10.10.16:25,10.150.14.108:59877,*,,Proxy destination(s) obtained from OnProxyInboundMessage event. Correlation Id:80e0d560-be23-4910-bcb0-43139bee131f
2019-01-31T13:45:09.501Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,14,10.10.10.16:25,10.150.14.108:59877,*,,Message or connection acked with status Retry and response 451 4.4.0 DNS query failed. The error was: DNS query failed with error InfoNoRecords -> DnsQueryFailed: InfoNoRecords
2019-01-31T13:45:09.501Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,15,10.10.10.16:25,10.150.14.108:59877,>,451 4.7.0 Temporary server error. Please try again later. PRX2 ,
2019-01-31T13:45:09.503Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,16,10.10.10.16:25,10.150.14.108:59877,<,QUIT,
2019-01-31T13:45:09.503Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,17,10.10.10.16:25,10.150.14.108:59877,>,221 2.0.0 Service closing transmission channel,
2019-01-31T13:45:09.503Z,SERVER\From Internal Servers (Relay),08D68772EDC476C6,18,10.10.10.16:25,10.150.14.108:59877,-,,Local

The protocol logs contain a number of columns to the left. The interesting ones for this are the connector name (“SERVER\From Internal Servers (Relay)”), the session ID (08D68772EDC476C6) and the sequence number (each item on the protocol has a incrementing sequence number, in the above it goes from 0 where the session connects (which is the + at the end) to 18, where it disconnects (the – at the end of the last line).

This log looks no different from a session that works (as it was random as I said above), but we see more about the error. Specifically we see the following:

Proxy destination(s) obtained from OnProxyInboundMessage event. Correlation Id:80e0d560-be23-4910-bcb0-43139bee131f
Message or connection acked with status Retry and response 451 4.4.0 DNS query failed. The error was: DNS query failed with error InfoNoRecords -> DnsQueryFailed: InfoNoRecords
451 4.7.0 Temporary server error. Please try again later. PRX2 ,

So we see that it is DNS. Online there are articles about this being to do with IPv6, AAAA records and invalid responses to those queries and fixes include using external DNS settings or smarthost values. None of this worked in this example.

So lets follow down the logs some more

Connectivity Logs

In the connectivity logs we search the same date/time/hour log for the session number, which in this case is 08D68772EDC476C6 from the above logs. In the connectivity logs we see a session that matches for this ID and its for “internalproxy”

2019-01-31T13:45:09.499Z,08D68772EDC476C7,SMTP,internalproxy,+,Undefined 00000000-0000-0000-0000-000000000000;QueueLength=<no priority counts>. Starting outbound connection for inbound session 08D68772EDC476C6
2019-01-31T13:45:09.501Z,08D68772EDC476C7,SMTP,internalproxy,>,DNS server returned InfoNoRecords reported by 10.10.10.21. [Domain:Result] = SERVER.internal.co.uk:InfoNoRecords;
2019-01-31T13:45:09.501Z,08D68772EDC476C7,SMTP,internalproxy,-,Messages: 0 Bytes: 0 (The DNS query for  'Undefined':'internalproxy':'00000000-0000-0000-0000-000000000000' failed with error : InfoNoRecords)

Internalproxy is what Exchange users to send email from the frontend transport service to the hub transport service. But which hub transport service are we going to use? If does not matter if you have 1 or x number of Exchange Servers in your site, it will use DNS to look up the IP of one of these servers. So even if you have a single Exchange box, DNS is vital.

In the above log we see that DNS 10.10.10.21 returns InfoNoRecords when queried for the Exchange Servers own name.

So I resort to nslookup to check DNS from this Exchange server. I have two DNS server, .20 and .21. The error appears to be related to .21 in this case.

To I enter “nslookup server.internal.co.uk 10.10.10.21” which means look up the name of the server using the DNS server 10.10.10.21. I got back a message saying cannot find server.internal.co.uk: Query refused.

When I tried the other DNS server I got back a successful response and the IP address of the server.

So for immediate fix, I removed 10.10.10.21 as an option for DNS for this server. Exchange immediately went back to work and PRX2 errors where not displayed and email got to its destination.

Now to go and see who has broken DNS!