Exchange Hybrid, when configured properly, can provide almost seamless coexistence between Exchange Online and your on-premises Exchange environment. Part of this concept is that while you technically have two separate Exchange organizations, the mail flow between these organizations appears “internal” so that a message from a cloud user looks no different than a message from an on-premises user.
As a consultant nearly 100% focused on Exchange Online migrations, I’ve come across a variety of situations where hybrid mail flow is not working properly. If you spend any time browsing the Office 365 Community Forums, you’ll see a number of posts on this same issue.
Even if the Hybrid Configuration Wizard completed successfully, it does not mean that hybrid mail flow is setup properly. In some cases, mail will actually be routing between organizations and it may seem like everything is working. A deeper look into the messages and you might find that while they are routing, they are not appearing as “internal”.
Below are some of the things to look for and how to resolve these hybrid mail flow issues.
The Configuration Basics
Before we jump into identifying issues, it’s important to understand how hybrid mail flow is configured. Among other tasks, the Hybrid Configuration Wizard (HCW) creates send and receive connectors in the on-premises Exchange organization and in Exchange Online. These connectors, along with a proper network configuration, are the foundation of hybrid mail flow.
There are some differences in how the configuration is setup in Exchange 2010 versus Exchange 2013 which is why you’ll see some issues on one platform and not the other.
If you’ve modified the connectors created by the HCW in an attempt to “make things work”, there’s a good chance you’ll want to delete them and let the HCW recreate them so you can identify the true root cause of your mail flow problem.
But My Mail Is Routing… So It Must Be Working
There are a number of symptoms that might indicate that hybrid mail flow is not working properly even when messages are routing. If messages between environments occasionally end up in a user’s junk mail, that’s definitely a good sign there is a misconfiguration. Issues booking conference rooms or receiving the wrong out-of-office message are also symptoms.
It basically comes down to whether the messages between environments appear as “internal” or “external”. So just like a conference room is not going to accept a booking request from a random Internet user, it won’t accept a booking from your cloud user if that message appears as external.
The quick way to check is to look at a message received in each environment and check the message headers. If the header “X-MS-Exchange-Organization-AuthAs” says “Internal”, then all is well; if the header says “Anonymous”, you may have one of the issues below.
Issue #1: Incorrectly Configured Hybrid Routing Domain (Exchange 2010)
This issue will actually cause hybrid mail flow to fail to route. In Exchange 2003 environments, the Exchange 2010 HCW does not properly configure the hybrid routing domain (tenant.mail.onmicrosoft.com); this domain should be configured as “Internal Relay” as opposed to “Authoritative”. While this used to be a somewhat undocumented step, it has since been documented in the “Exchange Server Deployment Assistant“.
Issue #2: Not Preserving EOP Source IP (Exchange 2010)
Since the Exchange 2010 HCW creates a new receive connector for Office 365 that is secured by the Exchange Online Protection (EOP) IP addresses, it’s important that the Exchange server sees the proper source IP for mail messages. If your traffic is going through a firewall or load balancer that is replacing the EOP source IP with it’s own, the traffic will end up hitting the default receive connector in Exchange instead of the Office 365 one. The best way to identify this is to enable verbose logging on both receive connectors and see where your connections are landing. Resolving this issue depends on the hardware in the mail path but the final solution needs to be that the Exchange 2010 server sees the EOP IP address and has a proper route back to it.
Issue #3: Improperly Configured “Remote Domains” (Exchange 2013)
Unleash the Potential of Power Platform With a Center of Excellence
Business innovation often comes from within. Discover how to empower innovation from non-traditional developers with the Microsoft Power Platform.
If users are getting the “external” out-of-office instead of the “internal” out-of-office or your “voting buttons” are not working, it’s possibly due to misconfigured remote domains. What I’ve noticed is that some versions of the HCW do not configure your “Remote Domains” properly. I put together a post specific to this issue and the easy resolution: “The Importance of Remote Domains in Exchange Hybrid“.
Issue #4: SSL Certificate Mismatch (Exchange 2013)
After renewing your SSL certificate on Exchange 2013, you may find that you have issues with your hybrid mail flow. Recall that the Exchange 2013 HCW uses the certificate on the on-premises receive connector and the inbound connector in the cloud. Fortunately Microsoft has documented this issue: “Can’t receive mail in an Exchange 2013-based hybrid environment after you install a new certificate on the server (KB2989382)“.
Issue #5: Use Of A Third-Party SMTP Gateway (Exchange 2010 / 2013)
This one comes up on the Office 365 Community Forums quite often. Microsoft does not support any third-party SMTP gateways between EOP and the on-premises hybrid connectors; the only supported device is an Exchange Edge Transport server.
While you can leave your non-hybrid traffic routing through a third-party appliance, using it in the middle of your hybrid mail flow may cause messages to appear as external and is not supported.
Issue #6: Failing To Update EOP IP Addresses (Exchange 2010 / 2013)
It’s not uncommon for organizations to secure port 25 on their firewall by locking down the hybrid traffic to just the Exchange Online Protection (EOP) IP addresses. This same list of IPs is on the receive connector in Exchange 2010. As you might imagine, this list of IPs is subject to change and has changed a fair amount recently. Make sure your rules and connectors are current to the Exchange Online Protection IP addresses list. This list has an RSS feed that can be added to your favorite RSS reader or even Outlook.
Issue #7: Firewall Blocking STARTTLS Command (Exchange 2010 / 2013)
Some firewalls have protocol inspection packages that cause issues with the TLS connection. If you telnet from the Exchange server to an Internet target and see a bunch of asterisks in the response, this is likely the issue.
Microsoft has some notes on various firewall vendors and what they each call this feature: “Cannot send or receive e-mail messages behind a Cisco PIX or Cisco ASA firewall (KB320027)“.
Issue #8: Blacklisted IP Address (Exchange 2010 / 2013)
In situations where a new server is deployed for hybrid, it may not have received the proper due diligence from a network planning perspective. I have seen multiple situations where the server with the outbound hybrid connector can send outbound on port 25 but the IP address is the same IP address used by the organization’s general Internet traffic. In many cases, this IP address is already blacklisted or will end up blacklisted and if that occurs, EOP will stop accepting your messages. Make sure there is a source NAT in place for your hybrid server such that the source IP does not appear on any realtime blacklists.
- Just because mail routes, doesn’t mean hybrid mail flow is working properly.
- Check the “X-MS-Exchange-Organization-AuthAs” message header of cross-premises messages to verify proper configuration.
- Exchange 2010 and Exchange 2013 have different configurations for hybrid mail flow.
- Attempts to create “workarounds” to issues may succeed in routing mail but may break hybrid mail flow.
Did you find this article helpful?
Leave a comment below or follow me on Twitter (@JoePalarchio) for additional posts and information on Office 365.
Looking to do some more reading on Office 365?
Catch up on my past articles here: Joe Palarchio.
Thank you, post #7 was extremely helpful
Great post! Clear, concise and helpful for various issues.
How does mail flow between 2 internal employees in a hybrid setup? Particularly if I use a 3rd party SMTP gateway that needs to do some compliance and encryption related tasks. Will the 3rd party gateway be able to scan the internal emails?
If the two internal employees are on the same platform (both cloud or both on-premises), then the SMTP gateway will not see the traffic. If one user is on-premises and one in the cloud, then that traffic will traverse the SMTP connector created by the hybrid wizard but that’s where it’s not supported to route that through a third-party gateway.
If we’re talking about traffic to external addresses, then you can use the “Centralized Transport” option to route all outbound emails from cloud users back through the hybrid connector and then out your SMTP gateway to the Internet. Another option, if you need both internal and external traffic, is to use the Journaling feature within Exchange Online.
Thanks for the question!
Another great article.
To make sure issue #6 does not occur, I create a scheduled task to automatically update the IP ranges on the receive connector.
Below is an extract of the script that includes both retrieving the IP ranges and setthe IP ranges.
# Make sure all IP ranges are retrieved
# Catch all current Exchange Online Datacenter IP ranges
$ExOIPranges = (Get-HybridMailflowDatacenterIPs).DatacenterIPs
# Save all current
Set-ReceiveConnector -Identity “= Office 365 receive connector =” -RemoteIPRanges $ExOIPranges
Do you know how, in Exchange 2013, they use the certificate to validate the connection instead of the IPs? I’ve been trying to replicate this for google apps but with no success.
We added gmail to the TlsDomainCapabilities and added the remote and accepted domains but no luck.
Hello Joe, Great Work!! but I have a question. I need to know how much traffic is being sent to the inbound connectors created in o365. Like how we have protocol logging in Exchange. Could we perform the same or find out the traffic coming in O365. Like you configure Hybrid or have relay servers which direct the emails to o365 for Internet delivery.
The message trace functionality and some of the reporting may give you what you’re looking for although there is some traffic that is not exposed in the logs. As you can imagine, there’s a great amount of spam that is trying to come in and if it’s dropped at the edge, you won’t see those messages in your logs.
I seem to have a routing issue that affects a small number of users. In whole the routing seems to act as desired but there are users who only recieve email from myself (migrated) to thier migrated 365 account as opposed from myself to thier on-premise emails. e.g i send an email and rather than hitting their normal inbox it only routes to their 365 account, therefore have to have two email windows open (both outlook and web browser for 365). Any ideas?
Two things that got me. #1. The “Internal Relay” configuration on the domain. #2. Don’t underestimate how long it can take for #1 to take effect in Office365 after setting it. MS support said it could take up to an hour.
That was an hour of banging head until it suddenly started working.