Author : MD TAREQ HASSAN | Updated : 2021/06/24
Domain and DNS Zone
Create App Service Domain
- Domain i.e. “
mypoc.com
”- New domain from Azure App Service Domain
- Buy custom in Azure Portal
- Go to: https://portal.azure.com/#create/Microsoft.Domain
- Fillup details
- Create
- Microsoft doc: buy an App Service domain
- Buy custom in Azure Portal
- Already have domain
- You own a domain (i.e. in namecheap)
- You can switch the domain DNS resolution to Azure
- See: Use Azure DNS for your domain
- New domain from Azure App Service Domain
- See: Creating app service domain in azure portal
Azure DNS zone:
- An Azure DNS Zone with same name (i.e.
mypoc.com
) will be created automatically if you buy App Service Domain in Azure portal - ExternalDNS K8s Add-on will automate DNS names for K8s Ingresses
Alias Record For Public IP Of Application Gateway
Using Azure cli command: Creating Alias Record In Azure DNS Zone
Using Azure Portal
- Go to Azure DNS Zone (
demo.com
) - Create alias record set for the “public IP” of Application Gateway (mentioned that AKS AGIC is enabled)
- Name:
@
- Target: (Azure Resource) Public IP of AGIC Application Gateway
- Name:
- Check that
demo.com
is hitting AGIC Application Gateway- Visit
demo.com
- Should show message:
- “Microsoft-Azure-Application-Gateway/v2” if no message or ingress is define for root uri “
/
” - The specified page or message if defined for root uri “
/
”
- “Microsoft-Azure-Application-Gateway/v2” if no message or ingress is define for root uri “
- Visit
- Now your domain i.e.
demo.com
will refer to AGIC Application Gateway
Multiple APIs Under Same Domain
Assign domain to AGIC Application Gateway: see previous step
Create deployment and ingress
- Each API will be served under “
/api
”- Example:
- Foo API → “
/api/foo
” - Bar API → “
/api/bar
”
- Foo API → “
- Routing has to be implemented accordingly in API project to make it work
- Example:
- All ingresses will use the same host i.e. “
demo.com
” but different “path” and “backend-path-prefix” accordingly- Example
- host:
host: demo.com
- path:
path: /api/foo*
- backend-path-prefix:
appgw.ingress.kubernetes.io/backend-path-prefix: "/api/foo"
- host:
- Make sure that “backend-path-prefix” (“
/api/foo
”) url responds 200 (means there is corresponding controller action for “/api/foo
” uri), otherwise AGIC Application Gateway health probe will fail
- Example
Deployment and ingress manifest yaml files: Sample - Serving Multiple APIs Under Same Domain
Multiple MVC Apps Under Same Domain
Applications Under Same Domain As Sub-Applications
Relative URLs for MVC Applications
- Relative URLs and URLs for static files would not work if applications are served as sub-application (i.e. serving under DNS
demo.com/xyz-webapp
) - To serve MVC application as sub-application with K8s ingress, make sure relative URLs are defined with “
~/
” (i.e. “~/app.css
”)- Page URL: “
~/about
”, “~/contact
” etc. - Static resources: “
~/app.css
”, “~/app.js
” etc.
- Page URL: “
- MVC projects: https://github.com/hovermind/AKSDeployDemos
- To make it work, see: https://stackoverflow.com/a/57212033/4802664
See: Samples - serving multiple MVC applications under same domain
Automate DNS Records With ExternalDNS
cert-manager ACME DNS01
- ExternalDNS is not needed if you are using cert-manager (ACME DNS01 with Let’s Encrypt)
- ACME DNS01 of cert-manager will serve similar to ExternalDNS (domain verification, DNS record for ingresses etc.)
- Although ExternalDNS is more powerful than ACME DNS01, (as of July, 2021) ExternalDNS is not compatible with cert-manager
How does ExternalDNS work
- ExternalDNS synchronizes exposed Kubernetes Services and Ingresses with DNS providers i.e. Azure DNS Zone
- ExternalDNS makes Kubernetes resources discoverable via public DNS servers
- ExternalDNS is K8s Add-on to automate DNS records in DNS providers i.e. Azure DNS Zones (DNS records for K8s Ingresses)
- If ExternalDNS is deployed to AKS (as pod), ingress dns names (i.e.
api.mypoc.com
) will be registered in Azure DNS Zone by ExternalDNS automatically - Follow steps bellow to automate DNS records for K8s ingresses
Prerequisites for ExternalDNS
- Make sure you have configured domain (i.e.
mypoc.com
) and Azure DNS zone - In order to make DNS records for K8s ingresses, AKS managed identity must be granted permission to Azure DNS Zone (i.e. contributor role)
- ExternalDNS will use AKS managed indentity to create DNS records (in Azure DNS Zone) for K8s ingresses
Login to Azure (PowerShell is being used)
<pre class="highlight"><code><span class="c">#</span><span class="w"> </span><span class="c"># Login to Azure</span><span class="w"> </span><span class="c">#</span><span class="w"> </span><span class="nv">$</span><span class="nn">Env</span><span class="p">:</span><span class="nv">AZ_USERNAME</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"email-here"</span><span class="w"> </span><span class="nv">$</span><span class="nn">Env</span><span class="p">:</span><span class="nv">AZ_PASSWORD</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"pass-here"</span><span class="w">
</span>az login -u $AZ_USERNAME -p $AZ_PASSWORD </code></pre>
Get AKS Credential (PowerShell is being used)
#
# Get AKS cluster credential
#
$Env:RESOURCE_GROUP = "my-poc-rg"
$Env:AKS_CLUSTER_NAME = "my-poc-aks"
az aks get-credentials --resource-group=$RESOURCE_GROUP --name=$AKS_CLUSTER_NAME --overwrite-existing
# Or
az login --use-device-code
Check that Managed Identity is enabled for AKS (if not enabled, then enable it)
#
# Checking that managed identity is enabled for AKS
#
# "clientId": "msi" => indicates that managed identity is enabled
# https://docs.microsoft.com/en-us/azure/aks/use-managed-identity#obtain-and-use-the-system-assigned-managed-identity-for-your-aks-cluster
#
az aks show -g my-poc-rg -n my-poc-aks --query "servicePrincipalProfile"
# To get control plane system-assigned identity's object ID:
# az aks show -g my-poc-rg -n my-poc-aks --query "identity"
#
# If managed identity is not enabled, then enable it
#
# az aks update -g <RGName> -n <AKSName> --enable-managed-identity
#
az aks update -g my-poc-rg -n my-poc-aks --enable-managed-identity
Grant permission (for AKS managed identity) to Azure DNS (PowerShell is being used)
$RESOURCE_GROUP = "my-poc-rg"
$AKS_CLUSTER_NAME = "my-poc-aks"
$DNS_ZONE_DOMAIN = "mypoc.com"
# To check your Azure DNS Zone: az network dns zone list --query "[?name=='$AZ_DNS_DOMAIN']"
#
# Get Scope (will be used to grant permission to Azure DNS)
#
# Format: /subscriptions/<subscription id>/resourceGroups/<resource group name>/providers/Microsoft.Network/dnszones/<zone name>/
#
$DNS_ZONE_SCOPE=$(az network dns zone list --query "[?name=='$DNS_ZONE_DOMAIN'].id" --output tsv)
#echo $DNS_ZONE_SCOPE
#
# Get principalId (objectId) of AKS managed identity
#
$MANAGED_IDENTITY_PRINCIPAL_ID = $(az aks show -g $RESOURCE_GROUP -n $AKS_CLUSTER_NAME --query "identityProfile.kubeletidentity.objectId" --output tsv)
echo $MANAGED_IDENTITY_PRINCIPAL_ID
#
# Assign contributor role to AKS managed identity
#
az role assignment create --assignee "$MANAGED_IDENTITY_PRINCIPAL_ID" --role "DNS Zone Contributor" --scope "$DNS_ZONE_SCOPE"
Create Kubernetes Secret For ExternalDNS
Using PowerShell
#
# Run PowerShell ISE as Admin (otherwise, PowerShell ISE would not be able to create azure.json file)
# Make sure that you are logged in to azure using 'az login'
#
#
# Set required variables
#
$AzureJsonFileLocation = "C:\azure.json"
$ResourceGroupName = "my-poc-rg"
#
# Get tenantId and subscriptionId
#
$TenantId = $(az account show --query "tenantId" --output tsv)
$SubscriptionId = $(az account show --query "id" --output tsv)
#
# JSON representation
#
$jsonRepresentation = @"
{
"tenantId": "$TenantId",
"subscriptionId" : "$SubscriptionId",
"resourceGroup" : "$ResourceGroupName",
"useManagedIdentityExtension" : true
}
"@
#
# Create azure.json file at specified location ($AzureJsonFileLocation)
#
$jsonRepresentation | Out-File $AzureJsonFileLocation
# For complex object to json => $jsonRepresentation | ConvertTo-Json -depth 100 | Out-File "C:\azure.json"
#
# Now create kubenetes secret base on azure.json
# This secret will be used by ExternalDNS to create DNS records for K8s ingresses in Azure DNS Zones
# Make sure: AKS managed indentity has been granted permission to Azure DNS Zone
#
kubectl create secret generic azure-config-file --from-file=$AzureJsonFileLocation
Using commands
#
# Gather information for `azure.json` file (K8s secret will be created using `azure.json`)
#
#
# Azure Tenant ID
#
az account show --query "tenantId"
#
# Azure Subscription ID
#
az account show --query "id"
#
# Create `azure.json` file for K8s Secret** (this secret will be used by ExternalDNS to modify Azure DNS Zone records)
#
# {
# "tenantId": "01234abc-xyz-abc-abc-xyz",
# "subscriptionId": "01234abc-xyz-abc-abc-xyz",
# "resourceGroup": "my-poc-rg",
# "useManagedIdentityExtension": true
# }
#
#
# Create K8s secret using azure.json
#
kubectl create secret generic azure-config-file --from-file=/local/path/to/azure.json
Deploy ExternalDNS
- Make sure you followed the previous steps (AKS managed identity, K8s secret for ExternalDNS etc.)
- Check latest release: https://github.com/kubernetes-sigs/external-dns/releases
- Docker image:
k8s.gcr.io/external-dns/external-dns:vx.y.z
(i.e.k8s.gcr.io/external-dns/external-dns:v0.8.0
)
external-dns.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: external-dns
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: external-dns
rules:
- apiGroups: [""]
resources: ["services","endpoints","pods"]
verbs: ["get","watch","list"]
- apiGroups: ["extensions","networking.k8s.io"]
resources: ["ingresses"]
verbs: ["get","watch","list"]
- apiGroups: [""]
resources: ["nodes"]
verbs: ["list"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: external-dns-viewer
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: external-dns
subjects:
- kind: ServiceAccount
name: external-dns
namespace: default
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: external-dns
spec:
strategy:
type: Recreate
selector:
matchLabels:
app: external-dns
template:
metadata:
labels:
app: external-dns
spec:
serviceAccountName: external-dns
containers:
- name: external-dns
image: k8s.gcr.io/external-dns/external-dns:v0.8.0
args:
- --source=service
- --source=ingress
#- --domain-filter=example.com # (optional) limit to only example.com domains; change to match the zone created above.
- --provider=azure
#- --azure-resource-group=externaldns # (optional) use the DNS zones from the specific resource group
volumeMounts:
- name: azure-config-file
mountPath: /etc/kubernetes
readOnly: true
volumes:
- name: azure-config-file
secret:
secretName: azure-config-file
Testing ExternalDNS With A Simple Ingress
Deploy sample ingresses: Foo and Bar ingresses for testing ExternalDNS
Verify that DNS record for ingress is created
az network dns record-set a list -g my-poc-rg -z mypoc.com
Next
Links
- https://azure.github.io/application-gateway-kubernetes-ingress/how-tos/dns/
- https://github.com/kubernetes-sigs/external-dns/blob/master/docs/tutorials/azure.md
- https://www.stacksimplify.com/azure-aks/azure-kubernetes-service-externaldns/
- https://joachim8675309.medium.com/extending-aks-with-external-dns-3da2703b9d52