Encrypting ExpressRoute for improved security

When you consider network connectivity to Azure services, authenticating with your IDENTITY over a HTTPS connection would seem to be reasonably secure. You have your identity which has controls and policies to ensure you can do what is allowed of you to do and you’ve gone through numerous hoops to ensure you are who Azure AD has been configured to believe is you.

If we take that one step further and move off of the public internet, ExpressRoute allows for private network connectivity to Azure. There’s a process to set the service up and extend on-premises networks to Azure. Traffic is encapsulated (usually 802.1AD/QinQ, 802.1Q/Dot1Q also available with ER Direct) meaning customers can be assured in the knowledge that no unauthorised 3rd party can intercept network connectivity to and from Azure.

ExpressRoute connects customers to Azure via a 3rd party connectivity provider. Technically this is to the Microsoft Enterprise Edge (MSEE) and not Azure proper just yet. Then there’s also the Azure network backbone/SDN, which is that ‘last mile’ between the MSEE, where ExpressRoute terminates to and customers VNets.

These hops and/or 3rd and 1st party providers mean there potential risk network traffic being compromised, even if that is a very, very, very small risk. If you’re already trusting your 3rd party service provider with network traffic, as well as Azure with secure workloads, this shouldn’t be an issue.

What if you were not so trusting and really wanted an even higher level of network security? What if you wanted encryption of network traffic?

There are really two means of encryption when it comes to ExpressRoute and/or ExpressRoute Direct. However, yes you guessed it, there’s limitations on which option you can use and when.

The two options are as follows:

• ExpressRoute = Configure an IPsec VPN tunnel over ExpressRoute
• ExpressRoute Direct = Configure MACsec and / or configure an IPsec VPN tunnel over an ExpressRoute circuit

MACsec

Let’s start with MACsec as this solution can only apply to ExpressRoute Direct. With ExpressRoute Direct, customers establish a layer 2 private connection from their edge to Microsoft Enterprise Edge in Azure. Once you have connectivity at layer 1, your layer2 is setup which allows you to then create logical ExpressRoute circuits on top of that. MACsec or Media Access Control Security, encrypts traffic from the customers edge to the Microsoft Enterprise Edge using a set of secrets which can be kept in Azure Key Vault for safe keeping.

This solution provides a good deal of security for ExpressRoute connectivity as it ensures that the traffic and network packets between the customer and Microsoft is encrypted and secure. For the most part, Microsoft is a trusted partner, otherwise customers wouldn’t put their secured workloads into Azure. However, as I mentioned earlier there is a small risk in that there is no end-to-end encryption from a customers edge to a Virtual Network (VNet). Depending on the level of security requirements, this solution may not be enough. That’s where the additional option bellow comes into play: IPsec VPN tunnels.

Side note: Microsoft uses MACsec to secure connectivity between zones/data centres within a region, as well as traffic between regions. reference

IPsec VPN

Taking security requirements that step further, over ExpressRoute, end-to-end encryption from a customer’s edge all the way to a VNet can be achieved with the implementation of IPsec VPN tunnels. There’s several options here on how to enable this functionality with the ability to complete the tunnel over Microsoft Peering (previous method, and still available) and Private Peering (GA as of November 2020).

The Microsoft Peering approach leverages the public IPs of a VNet Gateway, while the Private Peering approach completes the tunnel with a VNet VPN Gateway’s private IPs. Moreover, a Network Virtual Appliance (NVA) within the ExpressRoute connected VNet can be used to establish the IPsec tunnel, though additional Route Table/UDR requirements come into play to ensure traffic leaving Azure doesn’t simply go via the ExpressRoute VNet Gateway.

Thoughts and considerations

• This requirement doesn’t come up very often.
  • If we consider that Azure is a secure service, given all the industry standards that the cloud has attained in every geographical region, then its not too many use cases that would need any additional layers of encryption.
  • For the most part, encapsulation works well enough.
    • Furthermore, leveraging MACsec and / or IPsec on ExpressRoute would be reserved for highly secure workloads that would be highly confidential or classified.
    • I don’t see this being used for standard deployments.
• When using IPsec encryption over ExpressRoute, additional infrastructure is leveraged which needs to cater for bandwidth and/or throughput requirements.
  • Incorrectly provisioned connectivity/gateway/edge resources may degrade the service. One example is the VNet VPN Gateway:
  • If incorrectly provisioned at a lower SKU than required, the gateway may not meet ExpressRoute bandwidth requirements and may end up degrading overall performance.
  • Likewise, if using third party resources like Network Virtual Appliances (NVA) in Azure, much like the overhead of the IPsec tunnel over ExpressRoute mentioned above, there’s also the performance of the NVAs themselves that would need to be able to cater for the tunnel. If the NVA vendor has limitations, either in the software itself or how the virtual machine(s) is deployed in Azure to host the NVA, there could again be impacts to throughput due to the encryption/decryption overhead and/or bandwidth thought NIC(s).
• If using third party solutions, like an NVA in Azure, to complete an IPsec VPN tunnel than there’s additional routing requirements that need to be completed within Azure through Route Tables (UDRs) so that traffic if forced down the IPsec tunnel and not just down ExpressRoute.
  • With the routing order of hierarchy, an ExpressRoute circuit is preferred over (Site-to-Site) IPsec VPN when both routes are the same, Azure will use the longest prefix match to choose the route towards the packet’s destination.
• Realistically, specific Government regulations are usually the only times that encryption over encapsulation would be required, likely meaning end-to-end encryption as well.
  • Unless a customer may have a security team that want to align with the best possible standards. Which is also perfectly fine!
• Would I go to this extreme?
  • I say extreme, but it’s more ‘would I go implementing this level of security?’ If I’ve vetted the ExpressRoute service provider to ensure good, secure service. If I’ve vetted Azure as a trusted cloud for my workloads. If my on-premises edge has appropriate controls to not overexpose my network. I would be comfortable with the standard level of network encapsulation associated with ExpressRoute to ensure my network traffic is secure.

Cheers!