Dealing with Wildcard SSL Certificates on Azure and Kubernetes

It is almost certain that any DevOps approaches the challenges of implementing SSL certificates at some time.

Of course, there are free certificates, such as the well-known Lets Encrypt. As with any free solution, it has a number of limitations, all the restrictions are detailed on the certificate provider page for you to read. Some of the inconveniences encountered:

• Certificates must be reissued with a maximum validity period is 3 months
• When using Kubernetes, certificates have to be stored in the K8S itself and constantly regenerated
• There are a number of nuances with using and reissuing wildcard certificates
• Certain other features of the usage of protocols and encryption algorithms

Facing these issues from time to time, I came up with my own customization of the certificate solution, which I would like to share.

You may have heard about cert-manager, let’s install it with helm (my preferred way):

helm repo add jetstack https://charts.jetstack.io
helm repo update
kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.11.2/cert-manager.crds.yaml
helm install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace --version v1.11.2

so that you can create ClusterIssue as below:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-cluster-issuer
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: martin.miles@perficient.com   #replace with your e-mail
    privateKeySecretRef:
      name: letsencrypt-cluster-issuer
    solvers:
      - http01:
          ingress:
            class: nginx

At this stage, you got two options for issuing the certificates:

• via adding kind: certificate
• via ingress

1. In the first case, here’s what my yaml looks like:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: myservice
  namespace: test
Spec:
  duration: 2160h # 30 days
  renewBefore: 72h
  dnsNames:
    - replace_with_your.hostname.com  # replace with yours
  secretName: myservice-tls
  issuerRef:
    name: letsencrypt-cluster-issuer
    kind: ClusterIssuer

In that case, your ingress only references secretName: myservice-tls at the tls section for the desired service. The above file got helpful parameters:

• duration – a lifetime in hours
• renewBefore – how far from the certificate expiration you can renew an existing certificate

Tip: you can inspect the certificate in more detail by using the below kubectl command:

kubectl describe certificates <certificate name> -n <namespace>

2. Working with Let’s Encrypt certificates using Ingress seems to be more comfortable and reliable. In addition to secretName and hostname in tls section, you will also need to add only annotations:

annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-cluster-issuer"
    cert-manager.io/renew-before: 72h

And that’s it! Certificates are reissued automatically (within 3 days prior to the expirations, as stated above), upon the renewal which by default is 90 days.

Azure Key Vault

When developing a project for Azure, you’ll likely store your certificates at Azure Key Vault. Once purchase a certificate from Azure you’ll get prompted on how to add it to Azure Key Vault (also known as AKV, as abbreviated) – there’s nothing specific, just steps to prove and verify your domain ownership. Once completing all stages and collected all the green ticks, your certificate will show up at Secrets from AKV.

That approach benefits from an auto-update of certificates. A year later an updated certificate appears in AKV and automatically synchronizes with Secret in Kubernetes.

However, for Kubernetes to be able to use this cert we need to grant permissions. First, we need to obtain identityProfile.kubeletidentity.objectId of the cluster:

az aks show -g <ResourceGroup> -n <AKS_cluster_name>

the above returns an ID we require to provide in order to grant permission to secrets:

az keyvault set-policy --name <AKV_name> --object-id <identityProfile.kubeletidentity.objectId from the past step> --secret-permissions get

At this stage, we can install akv2k8s – a tool that takes care of Azure Key Vault secrets, certificates, and keys available in Kubernetes and/or your application – in a simple and secure way (here’s the installation guide with helm).

Next, synchronize the certificate from Azure Key Vault to Secret as per the official documentation.

apiVersion: spv.no/v1
kind: AzureKeyVaultSecret
metadata:
  name: wildcard-cert # any name of your preference
  namespace: default
spec:
  vault:
    name: SandboxKeyVault  # you certificate storage name in Azure
    object:
      name: name_object_id #object id from Azure AKV for the certificate
      type: secret
  output:
    secret:
      name: wildcard-cert # any name of secret within your namespace
      type: kubernetes.io/tls
      chainOrder: ensureserverfirst # this line is important - read below!

The last line is extremely important. The original problem was that despite the certificate being passed to Kubernetes correctly, it still did not work, and it appeared to be a non-trivial problem. The reason for it appeared to be while exporting a PFX certificate from Key Vault, the server certificate appears at the end of a chain, rather than at the beginning where you expect it to be. If using it together with ingress-nginx, the certificate won’t get loaded and will default. Specifying chainOrder: ensureserverfirst actually resolves this issue by placing the server certificate first in the chain, which otherwise has the following order:

1. Intermediate
2. Root
3. Server

 Wildcard Certificates

It is possible to purchase a certificate at Azure directly (actually served by GoDaddy) with two potential options:

• for a specific domain
• wildcard certificates

Notable that wildcard certificates only cover one level down, but not two or more – *.domain.com is not equal to *.*.domain.com. For example, this is not convenient when you would like to set up lover-level API endpoints for your subdomain-occupied websites. Without purchasing additional nested certificates, the only way to resolve this is by adding SAN (Subject Alternative Name) records to the certificate. Unfortunately, doing that is not easily possible, even through Azure support, which is hard to believe. That contrasts with AWS Certificate Manager, which in opposite, supports up to 10 SAN with a wildcard (*). Sad but true…

Azure Front Door

Azure Front Door (AFD) is a globally distributes application acceleration service provided by Microsoft. It acts as a cloud-based entry point for applications, allowing you to optimize and secure the delivery of your web applications, APIs, and content to users around the world. Azure Front Door operates at Layer 7 (HTTP/HTTPS) and can handle SSL/TLS encryption/decryption on behalf of your application, offloading the compute overhead from your backend servers. It also supports custom domain and certificate management and that is what we’re interested in.

When working with HTTPS you can also generate the certificate at AFD, upload your own, or sync the one from AKV (however you still require to grant AFD permission to AKV in order to access the certificate). The last approach allows selecting to rely on the latest version of the secret – that, in fact, takes all the pain of auto-upgrading certificates, an updated cert will be in play, once issued.

Tip:

When creating a backend pool and specifying your external AKS cluster IP address, make sure to leave the “Backend host header” field empty. It will fill in automatically with the values from the input box above.

An alternative option would be to route the whole HTTPS traffic from AFD to AKS, without SSL offloading at AFD. In order for AFD to work you must specify DNS name matching your AKS cluster (because of SNI and hc), otherwise it won’t work.

That introduces additional work. Say, you’ve already got AKS clusters without any name, working directly, which you now want routing through AFD. To make this work you need to end up with a separate DNS name for AKS cluster, setup DNS and create a service with a certificate attached to ingress. Only once that is done, HTTPS traffic redirect to AKS cluster would work perfectly well.

Tip: Another thing you may want to do – is to increase security for the above case by restricting AKS access to only AFD IP addresses within your Network Security Group for AKS. In addition, you may instruct ingress to only accept requests having a header from your Azure Front Door by id (X-Azure-FDID).

Lessons Learned

1. Azure Front Door is a pretty flexible routing service with powerful configuring options
2. Wildcard certificates only serve subdomains one level down, for other cases use SAN
3. SAN records are however not supported with Azure-purchase certificates, so use other vendors
4. Lets Encrypt certificates are still ok to use with auto-update, they’re free and allow wildcards