Using Existing VNet

• Existing VNet can be used to deploy AKS cluster
• Create a dedicated subnet for AKS cluster
• Nodes will use CNI (container network interface) plugin to communicate with other nodes

Network Prerequisites

• The VNET for the AKS cluster must allow outbound internet connectivity
• Do not create more than one AKS cluster in the same subnet
• Advanced networking for AKS does not support VNETs that use Azure Private DNS Zones
• AKS clusters may not use 169.254.0.0/16, 172.30.0.0/16, or 172.31.0.0/16 for the Kubernetes service address range
• The service principal used for the AKS cluster must have Contributor permissions to the resource group containing the existing VNET

Service Address Range

• Kubernetes service address range (CIDR) is a set of virtual IPs that Kubernetes assigns to internal services in your cluster
• Service Object will have virtual IP and services are only visible with cluster by default
• K8s service range is not a routable range and only K8s knows about it, routing packets from containers to a service within the cluster
• This range should not be used by any network element on or connected to this virtual network
• You can reuse this range across different AKS clusters
• Service address CIDR must be smaller than ‘/12’

Exposing cluster

• Expose service to internet via HTTP(S) -> use ingress controller, in case of Azure it’s Application Gateway Ingress Controller (AGIC)
• Expose service to internet via “non HTTP(s)” -> service type: loadBalancer

IP Address Range Calculation

• Container Networking Interface (CNI) will be used
• With CNI, every pod gets an IP address from the subnet and can be accessed directly. These IP addresses must be unique across your network space, and must be planned in advance
• Virtual network:
  • VNet can be as large as /8, but is limited to 65,536 configured IP addresses
  • Consider all your networking needs, including communicating with services in other virtual networks, on-premise etc.
• Subnet:
  • minimum subnet size: (number of nodes + 1) + ((number of nodes + 1) * maximum pods per node that you configure)
  • Example for a 50 node cluster: (51) + (51 * 30 (default, 30 pods per node)) = 1,581 (/21 or larger)
  • Example for a 50 node cluster that also includes provision to scale up an additional 10 nodes: (61) + (61 * 30 (default)) = 1,891 (/21 or larger)
  • “/21 or larger” -> Larger means, less mask bits i.e. /21, /20 …
• Kubernetes service address CIDR
  • Service address CIDR must be smaller than ‘/12’
  • smaller than /12 -> means more musk bits, i.e. /13, ‘/14’ …
  • Example service IP address range: 10.2.0.0/24 (virtual IPs and confined within the cluster by default)
• See: https://docs.microsoft.com/en-us/azure/aks/configure-azure-cni

Design IP Address Spaces

VNet Address Space (feel free to design according to your need)

10.8.0.0/16

Subnet for AKS Cluster (AKSClusterSubnet)

10.8.0.0/21

Service Address CIDR

10.0.0.0/16

Docket bridge IP range (use default)

172.17.0.1/16

While creating AKS in Azure portal, you can use the defaults (automatically filled up) for followings

• Kubernetes service address range
• Kubernetes DNS service IP address
• Docker Bridge address
• DNS name prefix

Example (ARM template from here):

"networkProfile": {
  "loadBalancerSku": "standard",
  "outboundType": "loadBalancer",
  "networkPlugin": "azure",
  "networkPolicy": "calico",
  "serviceCidr": "10.0.0.0/16",
  "dnsServiceIp": "10.0.0.10",
  "dockerBridgeCidr": "172.17.0.1/16"
}

Network Policy

• Network policies allow you to define rules for ingress and egress traffic between pods in a cluster, improving your cluster security by restricting access to certain pods
• The way NSG applies to NIC and subnets, Network Policy in AKS applies to Pods
• Why do we need network policy?
  • The principle of least privilege - to control inter-pod communictaion
  • To control ingress and egress traffic
  • To block traffic directly to back-end applications
  • To define rules for ingress (from cluster to pod) and egress (from pod to cluster i.e. node VM metadata endpoint) traffic between pods in a cluster
• Network policy is a kubernetes feature
• Policy can not be changed after provisioning AKS (decide carefully while provisioning AKS)
• Policy options:
  • Azure
  • Calico

Kubernetes authentication and authorization

• Role-based access control (RBAC): Set Enabled
• AKS-managed Azure Active Directory: Set Enabled
  • Configure Kubernetes role-based access control (RBAC) using Azure Active Directory group membership
  • Once this feature is enabled, it cannot be disabled
• You can optionally add Azure Active Directory groups that will have admin access within the cluster. You will be able to add or remove groups at any time

Links

• https://argonsys.com/microsoft-cloud/library/azure-kubernetes-service-cluster-capacity-planning/
• https://www.danielstechblog.io/build-azure-kubernetes-service-cluster-with-bring-your-own-virtual-network-on-azure/
• https://docs.microsoft.com/en-us/answers/questions/243407/azure-aks-network-plan.html