Tuesday, February 11, 2014

Exchange Transport Abnormal Shutdown

I am working on an issue with one of my all-in-one DAG members. The Transport service keeps shutting down abnormally and Forefront is the only one reporting it.
Log Name:      Application
Source:        FSCController
Date:          2/11/2014 11:18:42 AM
Event ID:      5167
Task Category: Scan Error
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      DAG2S06.domain.local
Microsoft Forefront Protection Monitor detected abnormal EDGETRANSPORT.EXE shutdown

First item I noticed, was that 2 of my send connectors had issues. They were still referencing old Exchange 2007 hub transport servers that are no longer in production.
EventID: 2937 - Process edgetransport.exe () (PID=17792). Object [CN=Outbound SMTP,CN=Connections,CN=Exchange Routing Group (DWBGZMFD01QNBJR),CN=Routing Groups,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=CA,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=domain,DC=local]. Property [HomeMtaServerId] is set to value [domain.local/Configuration/Deleted Objects/Microsoft MTA DEL:e14c2370-b325-47ef-b67f-0f186104bee2], it is pointing to the Deleted Objects container in Active Directory. This property should be fixed as soon as possible.

When you ran a "Get-SendConnector | FL", I saw which 2 connectors needed updating. Instead of updating them, I found it easier to just recreate them

$BadConnector = get-sendConnector | ?{$_.homeMTA -like "*DEL:*"}
$BadConnector | fl name,homemta
Name : Outbound SMTP HomeMTA : domain.local/Configuration/Deleted Objects/Microsoft MTA DEL:e14c2370-b325-47ef-b67f-0f186104bee2
Name : Remote Site TLS Connector HomeMTA : domain.local/Configuration/Deleted Objects/Microsoft MTA DEL:1b846366-974f-498e-b1f8-e5426a695989
$BadConnector | %{New-SendConnector -Name $($_.name.tostring()+"_2014") -SmartHosts $_.smarthosts -AddressSpaces $_.addressspaces -SourceTransportServers $_.sourcetransportservers -whatif}
After they replicated, I went and deleted the bad connectors from the environment. (I was getting the eventid 2937 above until I deleted them, even after disabling them.)

Looking closer, I found that Exchange didn't have permissions to this folder? This was causing an EventID 17104, followed by an Event 7004 "Activation took longer than expected to complete." error. (the the transport service stopped again). To solve this, I simply moved that QUEUE.OLD folder out of the QUEUE root. (It's now on the root of D:\). Restart the Transport service again.

SUCCESS!! Sort of.... 

My server started processing a backlog of messages and quickly exceeded the capacity.

Log Name:      Application
Source:        MSExchangeTransport
Date:          2/11/2014 11:46:49 AM
Event ID:      15004
Task Category: ResourceManager
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      DAG2S06.domain.local
The resource pressure increased from Medium to High.
The following resources are under pressure:
Version buckets = 242 [High] [Normal=80 Medium=120 High=200]
Physical memory load = 98% [limit is 94% to start dehydrating messages.]
The following components are disabled due to back pressure:
Inbound mail submission from Hub Transport servers
Inbound mail submission from the Internet
Mail submission from Pickup directory
Mail submission from Replay directory
Mail submission from Mailbox server
Mail delivery to remote domains
Content aggregation
The following resources are in normal state:
Queue database path ("D:\Program Files\Exchange\TransportRoles\data\Queue\mail.que") = 39% [Normal] [Normal=95% Medium=97% High=99%]
Queue database logging path ("D:\Program Files\Exchange\TransportRoles\data\Queue\") = 39% [Normal] [Normal=94% Medium=96% High=98%]
Private bytes = 1% [Normal] [Normal=71% Medium=73% High=75%]
Batch Point = 0 [Normal] [Normal=2000 Medium=4000 High=8000]
Submission Queue = 0 [Normal] [Normal=1000 Medium=2000 High=4000]
Followed a few minutes later by:
EventID: 17017 - Transport Mail Database: Quota was exceeded while performing a database operation. The Microsoft Exchange Transport service is shutting down. Exception details: Microsoft.Exchange.Isam.IsamVersionStoreOutOfMemoryException: Version store out of memory (cleanup already attempted) (-1069)
5 minutes later, my transport service automatically restarts itself. Now the QUEUE.old folder is back, but the service is still running.

No comments:

Post a Comment