Office 365 – Common Exchange Online Hybrid Mail Flow Issues

Exchange Hybrid, when configured properly, can provide almost seamless coexistence between Exchange Online and your on-premises Exchange environment. Part of this concept is that while you technically have two separate Exchange organizations, the mail flow between these organizations appears “internal” so that a message from a cloud user looks no different than a message from an on-premises user.
As a consultant nearly 100% focused on Exchange Online migrations, I’ve come across a variety of situations where hybrid mail flow is not working properly. If you spend any time browsing the Office 365 Community Forums, you’ll see a number of posts on this same issue.
Even if the Hybrid Configuration Wizard completed successfully, it does not mean that hybrid mail flow is setup properly. In some cases, mail will actually be routing between organizations and it may seem like everything is working. A deeper look into the messages and you might find that while they are routing, they are not appearing as “internal”.
Below are some of the things to look for and how to resolve these hybrid mail flow issues.

The Configuration Basics

Before we jump into identifying issues, it’s important to understand how hybrid mail flow is configured. Among other tasks, the Hybrid Configuration Wizard (HCW) creates send and receive connectors in the on-premises Exchange organization and in Exchange Online. These connectors, along with a proper network configuration, are the foundation of hybrid mail flow.
There are some differences in how the configuration is setup in Exchange 2010 versus Exchange 2013 which is why you’ll see some issues on one platform and not the other.

If you’ve modified the connectors created by the HCW in an attempt to “make things work”, there’s a good chance you’ll want to delete them and let the HCW recreate them so you can identify the true root cause of your mail flow problem.

But My Mail Is Routing… So It Must Be Working

There are a number of symptoms that might indicate that hybrid mail flow is not working properly even when messages are routing. If messages between environments occasionally end up in a user’s junk mail, that’s definitely a good sign there is a misconfiguration. Issues booking conference rooms or receiving the wrong out-of-office message are also symptoms.
It basically comes down to whether the messages between environments appear as “internal” or “external”. So just like a conference room is not going to accept a booking request from a random Internet user, it won’t accept a booking from your cloud user if that message appears as external.
The quick way to check is to look at a message received in each environment and check the message headers. If the header “X-MS-Exchange-Organization-AuthAs” says “Internal”, then all is well; if the header says “Anonymous”, you may have one of the issues below.

Issue #1: Incorrectly Configured Hybrid Routing Domain (Exchange 2010)

This issue will actually cause hybrid mail flow to fail to route. In Exchange 2003 environments, the Exchange 2010 HCW does not properly configure the hybrid routing domain (tenant.mail.onmicrosoft.com); this domain should be configured as “Internal Relay” as opposed to “Authoritative”. While this used to be a somewhat undocumented step, it has since been documented in the “Exchange Server Deployment Assistant“.

Issue #2: Not Preserving EOP Source IP (Exchange 2010)

Since the Exchange 2010 HCW creates a new receive connector for Office 365 that is secured by the Exchange Online Protection (EOP) IP addresses, it’s important that the Exchange server sees the proper source IP for mail messages. If your traffic is going through a firewall or load balancer that is replacing the EOP source IP with it’s own, the traffic will end up hitting the default receive connector in Exchange instead of the Office 365 one. The best way to identify this is to enable verbose logging on both receive connectors and see where your connections are landing. Resolving this issue depends on the hardware in the mail path but the final solution needs to be that the Exchange 2010 server sees the EOP IP address and has a proper route back to it.

Issue #3: Improperly Configured “Remote Domains” (Exchange 2013)

Revolutionize Your Business With Generative AI

From product design and software development to virtual agents, content creation, and reporting, GenAI is transforming business. Our AI experts help you unlock GenAI’s full potential and drive growth.

Let’s Get Started

If users are getting the “external” out-of-office instead of the “internal” out-of-office or your “voting buttons” are not working, it’s possibly due to misconfigured remote domains. What I’ve noticed is that some versions of the HCW do not configure your “Remote Domains” properly. I put together a post specific to this issue and the easy resolution: “The Importance of Remote Domains in Exchange Hybrid“.

Issue #4: SSL Certificate Mismatch (Exchange 2013)

After renewing your SSL certificate on Exchange 2013, you may find that you have issues with your hybrid mail flow. Recall that the Exchange 2013 HCW uses the certificate on the on-premises receive connector and the inbound connector in the cloud. Fortunately Microsoft has documented this issue: “Can’t receive mail in an Exchange 2013-based hybrid environment after you install a new certificate on the server (KB2989382)“.

Issue #5: Use Of A Third-Party SMTP Gateway (Exchange 2010 / 2013)

This one comes up on the Office 365 Community Forums quite often. Microsoft does not support any third-party SMTP gateways between EOP and the on-premises hybrid connectors; the only supported device is an Exchange Edge Transport server.

https://technet.microsoft.com/en-us/library/jj659055(v=exchg.150).aspx

While you can leave your non-hybrid traffic routing through a third-party appliance, using it in the middle of your hybrid mail flow may cause messages to appear as external and is not supported.

Issue #6: Failing To Update EOP IP Addresses (Exchange 2010 / 2013)

It’s not uncommon for organizations to secure port 25 on their firewall by locking down the hybrid traffic to just the Exchange Online Protection (EOP) IP addresses. This same list of IPs is on the receive connector in Exchange 2010. As you might imagine, this list of IPs is subject to change and has changed a fair amount recently. Make sure your rules and connectors are current to the Exchange Online Protection IP addresses list. This list has an RSS feed that can be added to your favorite RSS reader or even Outlook.

Issue #7: Firewall Blocking STARTTLS Command (Exchange 2010 / 2013)

Some firewalls have protocol inspection packages that cause issues with the TLS connection. If you telnet from the Exchange server to an Internet target and see a bunch of asterisks in the response, this is likely the issue.

Microsoft has some notes on various firewall vendors and what they each call this feature: “Cannot send or receive e-mail messages behind a Cisco PIX or Cisco ASA firewall (KB320027)“.

Issue #8: Blacklisted IP Address (Exchange 2010 / 2013)

In situations where a new server is deployed for hybrid, it may not have received the proper due diligence from a network planning perspective. I have seen multiple situations where the server with the outbound hybrid connector can send outbound on port 25 but the IP address is the same IP address used by the organization’s general Internet traffic. In many cases, this IP address is already blacklisted or will end up blacklisted and if that occurs, EOP will stop accepting your messages. Make sure there is a source NAT in place for your hybrid server such that the source IP does not appear on any realtime blacklists.

Summary

• Just because mail routes, doesn’t mean hybrid mail flow is working properly.
• Check the “X-MS-Exchange-Organization-AuthAs” message header of cross-premises messages to verify proper configuration.
• Exchange 2010 and Exchange 2013 have different configurations for hybrid mail flow.
• Attempts to create “workarounds” to issues may succeed in routing mail but may break hybrid mail flow.

Did you find this article helpful?
Leave a comment below or follow me on Twitter (@JoePalarchio) for additional posts and information on Office 365.
Looking to do some more reading on Office 365?
Catch up on my past articles here: Joe Palarchio.