Author : MD TAREQ HASSAN | Updated : 2021/06/24
Prerequisites
- Tools
- Install Azure CLI
- Install jq (optional)
- PowerSehll ISE (already installed in windows)
- AGIC is enabled for AKS
- Create Azure DNS Zone
- Create Alias Record in DNS Zone for Public IP Of Application Gateway
Now, get AKS Credentials To Execute kubectl
Commands
Concepts
See: Understanding TLS and Certificate Authority
What Is Lets Encrypt?
- Let’s Encrypt is a non-profit certificate authority run by Internet Security Research Group that provides X.509 certificates for Transport Layer Security encryption at no charge.
- A nonprofit Certificate Authority providing TLS certificates
- https://letsencrypt.org/
Cert Manager
- cert-manager is a Kubernetes add-on, which automates the creation and management of certificates
- cert-manager uses Lets Encrypt to automatically obtain a TLS/SSL certificate for your domain
- The certificate will be installed on Ingress Controller Gateway (AGIC Application Gateway, Nginx etc.), which will perform SSL/TLS termination for your AKS cluster
Issuer
- Issuers is Kubernetes resources (CRD) that represent certificate authorities (CAs) that are able to generate signed certificates by honoring certificate signing requests
- An Issuer is a namespaced resource, and it is not possible to issue certificates from an Issuer in a different namespace
- You need to create an Issuer in each namespace you wish to obtain Certificates in
ClusterIssuer
- Non-namespaced Issuer that can be used cluster wide
- ClusterIssuer is almost identical to the Issuer resource, however is non-namespaced so it can be used to issue Certificates across all namespaces
Built-in certificate issuers
- cert-manager comes with a number of built-in certificate issuers (denoted by
cert-manager.io
) - You can also install external issuers in addition to the built-in types
- Both built-in and external issuers are treated the same and are configured similarly.
Cluster Resource Namespace
- cert-manager creates custom resources (CRDs) in “Cluster Resource Namespace”
- By default Cluster Resource Namespace is named “cert-manager”
Referencing secret created by ClusterIssuer
- ClusterIssuer resource is cluster scoped but when referencing a secret (certificate issued by ClusterIssuer) via the secretName field, secrets will be looked for in the “Cluster Resource Namespace” which is by default “cert-manager”
- Cluster Resource Namespace can be changed via a flag on the “cert-manager-controller” component:
--cluster-resource-namespace=my-namespace
ACME Challenge
- cert-manager can be used to obtain certificates from a CA using the ACME protocol
- The ACME protocol supports various challenge mechanisms which are used to prove ownership of a domain so that a valid certificate can be issued for that domain
- Challenge mechanisms
- DNS01 (DNS validation): you prove ownership of a domain by proving you control its DNS records. This is done by creating a TXT record with specific content that proves you have control of the domains DNS records
- HTTP01 (HTTP validation): you prove ownership of a domain by ensuring that a particular file is present at the domain. It is assumed that you control the domain if you are able to publish the given file under a given path
Links
- https://cert-manager.io/docs/concepts/
- https://cert-manager.io/docs/installation/kubernetes/
- https://cert-manager.io/docs/configuration/
- https://cert-manager.io/docs/tutorials/
Installing Cert Manager
Namespace
- Cert manager CustomResourceDefinition (CRD) will be installed in “cert-manager” namespace
- Create “cert-manager” namespace and disable resource validation (
cert-manager.io/disable-validation=true
)
#
# Create the namespace for cert-manager
#
kubectl create namespace cert-manager
#
# Label the cert-manager namespace to disable resource validation
#
kubectl label namespace cert-manager cert-manager.io/disable-validation=true
Installing cert-manager using Helm chart
- Add the Jetstack Helm repository (cert-manager should be used from this)
- The version residing in the helm repository is deprecated and should not be used (-> use Jetstack repository)
- You can use “
--set installCRDs=true
” flag to your Helm installation command to automatically install and manage the CRDs as part of your Helm release (alternatively, CRDs can be installed beforehand using kubectl command -> see below)
#
# Add the Jetstack Helm repository
#
helm repo add jetstack https://charts.jetstack.io
#
# Update your local Helm chart repository cache
#
helm repo update
#
# Install cert-manager using helm chart
#
# Check version: https://artifacthub.io/packages/helm/cert-manager/cert-manager
# Install latest version
#
# Automatically install cert-manager CRDs as part of your Helm release
# Make sure 'cert-manager' namespace is created and resource validation is disabled (see above)
# "--create-namespace" -> could be used but need to disable resource validation, that's why namespace is created beforehand
#
helm install cert-manager jetstack/cert-manager \
--namespace cert-manager \
--version v1.4.0 \
--set installCRDs=true
#
# Helm chart installation when CRDs are created beforehand using kubectl command
#
# helm install cert-manager jetstack/cert-manager \
# --namespace cert-manager \
# --version v1.4.0
#
#
# Some additional flag (if needed)
#
--set ingressShim.defaultACMEChallengeType=dns01 \
--set ingressShim.defaultACMEDNS01ChallengeProvider=AzureDNS \
--set ingressShim.defaultIssuerName=letsencrypt-prod \
--set ingressShim.defaultIssuerKind=ClusterIssuer \
Verify installation
kubectl get pods --namespace cert-manager
Alternative CRDs installation (before installing cert-manager helm chart): Installing cert-manager CRDs with kubectl
- cert-manager requires a number of CRDs resources to be installed into your cluster as part of installation
- Check latest version: https://cert-manager.io/docs/installation/supported-releases/
- Use
--validate=false
#
# Install the CustomResourceDefinition resources separately
# Note: --validate=false is required per https://github.com/jetstack/cert-manager/issues/2208#issuecomment-541311021
#
kubectl apply -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.14/deploy/manifests/00-crds.yaml --validate=false
# kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.4.0/cert-manager.crds.yaml
Issuer Resource
- The first thing you need need to do after installing cert-manager is to create a ClusterIssuer or Issuer resource
- ClusterIssuer is required by cert-manager to represent the certificate authority where the signed certificates will be obtained
- Types:
- Issuer (namespaced)
- ClusterIssuer (non-namespaced)
- By using the non-namespaced ClusterIssuer resource, cert-manager will issue certificates that can be consumed from multiple namespaces
ACME Type Issuer
- The ACME Issuer type represents a single account registered with the Automated Certificate Management Environment (ACME) Certificate Authority server
- When you create a new ACME Issuer, cert-manager will generate a private key which is used to identify you with the ACME server
We will use Let’s Encrypt as ACME Type ClusterIssuer (Other issuer types: https://cert-manager.io/docs/configuration/#supported-issuer-types)
ACME Challenge
- Let’s Encrypt uses the ACME protocol to verify that you control a given domain name and to issue you a certificate
- Challenge types:
- HTTP (
HTTP01
) - DNS (
DNS01
)
- HTTP (
We will use ACME DNS01
challenge
- ACME
DNS01
(when configured i.e. using Pod identity) will manage DNS records automatically for ingresses - (ExternalDNS](https://github.com/kubernetes-sigs/external-dns) is better than ACME
DNS01
but currently ExternalDNS is not compatible with cert-manager
ClusterIssuer
- Non-namespaced cluster wide issuer
- Acts as a CA for ingress domain and sub-domain
- Name:
- Ingresses will refer to the ClusterIssuer using it’s name with annotation
- Annotation: “
cert-manager.io/cluster-issuer: fooapp-letsencrypt-staging
”
- ACME Server (in our case ACME Server is Let’s Encrypt Server)
- Let’s Encrypt Staging: “
server: https://acme-staging-v02.api.letsencrypt.org/directory
” - Let’s Encrypt Production: “
server: https://acme-v02.api.letsencrypt.org/directory
”
- Let’s Encrypt Staging: “
- ACME Challenge Solver
- http01
- dns01
- Ingresses will refer to the ACME solver type with annotation “
cert-manager.io/acme-challenge-type: http01
”
See below
- Staging Server And HTTP01 Challenge
- Production Server And DNS01 Challenge
Staging Server And HTTP01 Challenge
- Used for testing only (to check if TLS is working)
- Check prerequisites above
- Check
cert-manager.io
version: https://cert-manager.io/docs/reference/api-docs/ - ACME URL for Let’s Encrypt staging environment: https://letsencrypt.org/docs/staging-environment/
letsencrypt-staging.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-staging
spec:
acme:
# You must replace this email address with your own.
# Let's Encrypt will use this to contact you about expiring
# certificates, and issues related to your account.
email: <YOUR.EMAIL@ADDRESS>
# ACME server URL for Let’s Encrypt’s staging environment.
# The staging environment will not issue trusted certificates but is
# used to ensure that the verification process is working properly
# before moving to production
server: https://acme-staging-v02.api.letsencrypt.org/directory
privateKeySecretRef:
# Secret resource used to store the account's private key.
name: letsencrypt-secret
# Enable the HTTP-01 challenge provider
# you prove ownership of a domain by ensuring that a particular
# file is present at the domain
solvers:
- http01:
ingress:
class: azure/application-gateway
Create ClusterIssuer
#
# Since ClusterIssuer is non-namespaced, no need to specify namespace in yaml or kubectl command
#
kubectl apply -f letsencrypt-staging.yaml
Checking
#
# Check clusterissuer
#
kubectl describe clusterissuer letsencrypt-staging
#
# Issued certificate will be stored as K8s secret in 'cert-manager' namesapce
#
# This secret (tls certificate) will be referred in spec.tls section of ingress manifest yaml
#
kubectl get secret letsencrypt-secret -n cert-manager -o yaml
Create a sample web app, publish to Azure container registry and deploy to AKS. Now create an ingress for that web app.
letsencrypt-staging-ingress.yaml
Production Server And DNS01 Challenge
Managed Indentity For ClusterIssuer
- Pod managed identity (backed by managed identity) will be used:
- ACME DNS01 challenge: to create txt record in Azure DNS Zone for domain ownership verification
- to create DNS records for ingresses
Prepare managed identity and K8s pod identity
#
# Required variables
#
$ResourceGroup = "fusys-poc-rg"
$AksName = "fusys-poc-aks"
$DnsZoneName = "fusyspoc.com"
$CertManagerNamespace = "cert-manager"
$ManagedIdentityName = "dns-zone-contributor-mid"
#
# Create managed identity
#
$ManagedIdentity = $(az identity create --name $ManagedIdentityName --resource-group $ResourceGroup)
#echo $ManagedIdentity
#
# Extract required information from that managed identity
#
$ResourceId = (echo $ManagedIdentity | jq -r ".id")
#echo $ResourceId
$ClientId = (echo $ManagedIdentity | jq -r ".clientId")
#echo $ClientId
$PrincipleId = (echo $ManagedIdentity | jq -r ".principalId")
#echo $PrincipleId
#
# Get DNS Zone resource id
#
$DnsZoneResourceId = $(az network dns zone show --name $DnsZoneName --resource-group $ResourceGroup --query "id" -o tsv)
#echo $DnsZoneResourceId
#
# Create role assignment "DNS Zone Contributor"
#
az role assignment create --role "DNS Zone Contributor" --assignee $PrincipleId --scope $DnsZoneResourceId
#
# Create pod-identity (AzureIdentity,AzureIdentityBinding)
#
az aks pod-identity add `
--resource-group $ResourceGroup `
--cluster-name $AksName `
--namespace $CertManagerNamespace `
--name $ManagedIdentityName `
--identity-resource-id $ResourceId
#
# Check that the identity and identity binding are created
#
# Take note of identity binding name (will be used in cert-manager pod)
#
kubectl get AzureIdentity,AzureIdentityBinding -n $CertManagerNamespace
#echo $ManagedIdentityName-binding
Install cert-manager with pod identity
#
# If you already installed cert-manager (with helm) without pod identity,
# you need to uninstall and then re-install with pod identity
#
# Get releases
# Uninstall cert-manager
#
# helm uninstall RELEASE_NAME
#
helm list --all --all-namespaces
helm uninstall cert-manager
#
# Now install cert-manager
#
# If you already installed cert-manager, you need to re-install with "--set podLabels.aadpodidbinding=xxx-binding"
#
helm install cert-manager jetstack/cert-manager `
--namespace $CertManagerNamespace `
--version v1.4.0 `
--set installCRDs=true `
--set podLabels.aadpodidbinding=$ManagedIdentityName-binding
Gather following information about DNS Zone (will be used in ClusterIssuer dns01 solver)
subscriptionID: AZURE_SUBSCRIPTION_ID
resourceGroupName: AZURE_DNS_ZONE_RESOURCE_GROUP
hostedZoneName: AZURE_DNS_ZONE
#
# Resource group and DNS Zone
#
$DnsZoneName = "fusyspoc.com"
$ResourceGroup = "fusys-poc-rg"
#
# Get subcription id
#
$DnsZoneResourceId = (az resource show -g $ResourceGroup -n $DnsZoneName --resource-type "Microsoft.Network/dnsZones" --query id --output tsv)
echo $DnsZoneResourceId
$SubscriptionId = $DnsZoneResourceId.Split('/')[2]
echo $SubscriptionId
letsencrypt-prod.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
spec:
acme:
email: test@hovermind.com
server: https://acme-v02.api.letsencrypt.org/directory
privateKeySecretRef:
name: letsencrypt-secret
solvers:
- dns01:
azureDNS:
subscriptionID: 2f38b28b-b089-4a76-8109-9ef207934b0e
resourceGroupName: fusys-poc-rg
hostedZoneName: fusyspoc.com
# Azure Cloud Environment, default to AzurePublicCloud
environment: AzurePublicCloud
Create ClusterIssuer
#
# Since ClusterIssuer is non-namespaced, no need to specify namespace in yaml or kubectl command
#
kubectl apply -f letsencrypt-prod.yaml
Checking
#
# Check clusterissuer
#
kubectl describe clusterissuer letsencrypt-prod
#
# Issued certificate will be stored as K8s secret in 'cert-manager' namesapce
#
# This secret (tls certificate) will be referred in spec.tls section of ingress manifest yaml
#
kubectl get secret letsencrypt-secret -n cert-manager -o yaml
letsencrypt-prod-ingress.yaml
Links
- https://azure.github.io/application-gateway-kubernetes-ingress/how-tos/lets-encrypt/
- https://docs.microsoft.com/en-us/azure/application-gateway/ingress-controller-letsencrypt-certificate-application-gateway
- https://docs.microsoft.com/en-us/azure/application-gateway/ingress-controller-expose-service-over-http-https
- https://github.com/fbeltrao/aks-letsencrypt
- https://github.com/Azure/application-gateway-kubernetes-ingress#how-tos
- https://github.com/Azure/application-gateway-kubernetes-ingress/blob/master/docs/how-tos/dns.md
- https://cert-manager.io/docs/configuration/acme/dns01/azuredns/
- https://gist.github.com/joelharkes/e63482541abc5c3a396c5434549b8a58