While preparing a test environment (virtual machines) for migration from Lotus Notes to Exchange Server 2007, I came across an interesting issue: All messages sent to recipients internally (within Exchange) or externally (Lotus Notes or Internet), would end up in the Drafts Folder.
No warnings, error messages, or NDR was generated in the UI. This was quite puzzling to say the least. So I decided to do what most of you seasoned IT professionals would consider as the next logical step: Dig deeper.
I began by looking through the event logs on the Exchange Mailbox and Hub Transport Servers. In the application log of the Mailbox Server, I noticed Event ID 1009 – A warning event with the Source and Category labeled as "MSExchangeMailSubmission". The description of the Event was spot on: "The Microsoft Exchange Mail Submission service is currently unable to contact any Hub Transport servers in the local Active Directory site. The servers may be too busy to accept new connections at this time."
I suppose I owe a big "Thank you" to the person at Microsoft who came up with the meaningful labels for Event Source and Category – just the kind of things we take for granted.
Anyways, since there was strong indication that the problem existed at the Hub Transport server, I started examining the application log of the Hub Transport Server. There were no errors logged, but there was one warning: Event ID: 15002, Source: MSExchangeTransport, Category: ResourceManager. In my rush to solve the problem, I ignored the warning event and assumed that the server was just complaining about system resources on the virtual machine – not far from the truth really.
Meanwhile I used Telnet to connect to port 25 of the hub transport server and immediately got a response stating:
452 4.3.1 Insufficient System Resources
Connection to host lost
This got me to take a closer look at the warning Event ID: 15002 in the application log of the hub transport server. I read the description carefully:
"The resource pressure is constant at High. Statistics:
Queue database and disk space ("C:Program FilesMicrosoftExchange ServerTransportRolesdataQueuemail.que") = 63% [High] [Normal=45% MediumHigh=47% High=49%]
Queue database logging disk space ("C:Program FilesMicrosoftExchange ServerTransportRolesdataQueue") = 63% [Normal] [Normal=89% MediumHigh=91% High=93%]
Version buckets = 1 [Normal] [Normal=40 MediumHigh=60 High=100]
Private bytes = 16% [Normal] [Normal=71% MediumHigh=73% High=75%]
Physical memory load = 53% [limit is 94% to start dehydrating messages.]
Inbound mail submission from other Hub Transport servers, the Internet, the Pickup directory, the Replay directory, and the Mailbox server, if it is on a Hub Transport server, has stopped.
Loading of e-mail from the queuing database, if available, continues."
Scrolling down on the description and actually reading the details helped J But wait, I had 2.94 GB Free Space on my virtual machine running the hub transport server role. So why was Exchange complaining about disk space?
After some research, I came across this article on Technet which introduced me to a not so well known, but novel concept introduced in Exchange Server 2007 – Back Pressure. While you can read more details on the subject on referenced pages, essentially back pressure is a system resource monitoring feature built into Exchange Server 2007 Hub Transport and Edge Server roles, and affects message delivery depending on the state of system resources.
After applying the formula Exchange uses to calculate the thresholds for the transport service (100*(hard disk drive size – 4 GB) / hard disk drive size), I realized that the resulting HIGH value for Queue database and disk space in my environment (50%) was higher than the HIGH threshold calculated by Exchange (49%).
Tip: If the available free space is less than 4 GB, the hard disk drive utilization level is considered high.
So my choices to get mail flowing were:
- Increase available disk space (recommended for production environments) OR
- Increase the appropriate threshold (NOT recommended for production environments)
Since I was using a test environment, I decided to take the easy route and override the default calculations for the high level of hard disk drive utilization by specifying a new value in the EdgeTransport.exe.config file which can be found in C:Program FilesMicrosoftExchange ServerBin directory. I changed the default value for PercentageDatabaseDiskSpaceUsedHighThreshold from "0" to "80" and restarted the Microsoft Exchange Transport Service.
I verified that the warning message did not appear again after the service restart and attempted to send a message to myself again using OWA – SUCCESSFUL!! I also noticed that the message which was previously placed into the draft folder (during the issue) was also delivered successfully without any manual intervention.
While Back Pressure is a neat feature in Exchange and is intended to address system resource issues more gracefully, I hope that companies will continue to invest in effective monitoring solutions for Exchange such as System Center Operations Manager to avoid issues such as the one described above. The exception of-course is a scenario where you are testing an Exchange solution in a test environment where you may not have the luxury of having production class system resources.
References:
