Scenario

• Network topology: Hub-Spoke
• On-premise network is connected via Express Route or VPN gateway
• “GatewaySubnet” is in Hub VNet
• “GatewaySubnet” address space: 10.150.192.0/24
• “mssql:1433” traffic coming from on-premise network (i.e. “10.1.16.0/23”) must go through Azure Firewall
• Azure Firewall:
  • AzureFirewallSubnet: 10.150.193.0/24
  • Firewall private IP: 10.150.193.4
• Azure SQL Managed Instance:
  • “AzureSQLManagedInstanceSubnet” is in Spoke VNet
  • AzureSQLManagedInstanceSubnet address space: 10.150.40.0/21

Let’s say we have workload deployed in Spoke VNet (i.e. Azure SQL Managed Instance, AKS etc.) and Hub VNet has VMs, Application Gateway, Firewall etc. On-premise applications, SQL clients (i.e. SSMS) will connect to workload. For security purposes, we need to inspect all traffic. Therefore incoming traffic from on-premise must go through Azure Firewall. Azure Firewall is deployed in Hub VNet’s AzureFirewallSubnet.

Route Table for GatewaySubnet

Create a route table with required settings & routes and assign it to GatewaySubnet:

• Create a route table i.e. “rt-gwsubnet-xsystem-dev-japaneast”
• Route Table Settings
  • Configuration > Propagate gateway routes: Yes (Gateway will not function properly if set to “No”)
  • Do not override “0.0.0.0/0” route, because Gateway must have direct internet connection
  • Subnets > “+ Associate” > GatewaySubnet
• Add routes
  • If destination is entire Spoke VNet (i.e. 10.150.0.0/18)
    • Route name: ToSpokeVNet-ThroughFirewall
    • Address prefix: 10.150.0.0/18
    • Next hop type: Virtual appliance
    • Next hop address: 10.150.193.4 (Firewall private IP)
  • If destination is specific subnet in Spoke (i.e. PrivateLinkSubnet: 10.150.36.0/24)
    • Route name: ToPrivateLinkSubnet-ThroughFirewall
    • Address prefix: 10.150.36.0/24
    • Next hop type: Virtual appliance
    • Next hop address: 10.150.193.4 (Firewall private IP)
  • If destination is specific subnet in Hub (i.e. UtilitySubnet: 10.150.196.0/24)
    • Route name: ToHubUtilitySubnet-ThroughFirewall
    • Address prefix: 10.150.196.0/24
    • Next hop type: Virtual appliance
    • Next hop address: 10.150.193.4 (Firewall private IP)

Firewall Rule Collection Groups and Rule Collections

• Go to firewall policy
• Settings: Rule collections > “+ Add”
• Create rule collection groups and rule collections

Network Rule Collection Groups

• OnPremiseNetworkRuleCollectionGroup
  • Name: OnPremiseNetworkRuleCollectionGroup
  • Priority: 205
• HubUtilityServiceNetworkRuleCollectionGroup
  • Name: HubUtilityServiceNetworkRuleCollectionGroup
  • Priority: 210

Application Rule Collection Groups

• OnPremiseApplicationRuleCollectionGroup
  • Name: OnPremiseApplicationRuleCollectionGroup
  • Priority: 305
• AksApplicationRuleCollectionGroup
  • Name: AksApplicationRuleCollectionGroup
  • Priority: 310
• SpokePaasServiceApplicationRuleCollectionGroup
  • Name: SpokePaasServiceApplicationRuleCollectionGroup
  • Priority: 315
• HubUtilityServiceApplicationRuleCollectionGroup
  • Name: HubUtilityServiceApplicationRuleCollectionGroup
  • Priority: 320

Example Rule Collections

• Add network rule collection to rule collection group
  • Name: OnPremiseNetworkRuleCollection
  • Rule collection type: Network
  • Priority: 200
  • Rule collection action: Allow
  • Rule collection group: OnPremiseNetworkRuleCollectionGroup
• Add application rule collection to rule collection group
  • Name: OnPremiseApplicationRuleCollection
  • Rule collection type: Application
  • Priority: 200
  • Rule collection action: Allow
  • Rule collection group: OnPremiseApplicationRuleCollectionGroup

Azure SQM Managed Instance

Managed Instance creates all necessary NSG rules and routes during provisioning. Therefore, no need worry about adding firewall rules for management traffic when overriding “0.0.0.0/0” with “Propagate gateway routes: No”. Firewall rules are needed to allow incoming on-premise traffic. Required firewall rules vary depending on Managed Instance connection type.

Azure SQL Managed Instance connection types

• Proxy (Default)
• Redirect

Redirect mode enables direct connectivity to Managed Instance resulting in improved latency and throughput and it requires ports 11000-11999 and 1433 to be open to the inbound traffic. Proxy mode enables proxy connectivity to Managed Instance and it requires only port 1433 to be open to the inbound traffic. Configuring this option applies to private endpoint only and does not affect public endpoint which always defaults to Proxy connection mode.

Step-1

“Route Table for GatewaySubnet” which is described above.

Step-2

Since the traffic is coming from on-premise to Firewall because of UDR in the Route Table of GatewaySubnet, now we need to add required Firewall rules to allow it so that traffic goes to Managed Instance (Firewall denies by default).

Proxy Mode: Azure Firewall Application Rule with FQDN filtering

• Application Rule with FQDN filtering is supported in proxy-mode only (port 1433)
• For FQDN filtering:
  • Since Application Rules are always SNATed (with transparent proxy), “asymmetric routing” would not happen
  • We do not need to add UDR for target on-premise network to Managed Instance subnet’s route table
• Add Application Rule to allow MSSQL traffic (“mssql:1433” is layer-7 traffic)
  • Rule collection group: OnPremiseApplicationRuleCollectionGroup
  • Rule collection: OnPremiseApplicationRuleCollection
  • Name: AllowMssql-FromOnPremiseNet001-ToManagedInstance
  • Source type: IP Address
  • Source IP Address: 10.1.16.0/23 (address prefix of target on-premise network)
  • Destination type: FQDN
  • Target FQDNs: *.database.windows.net
  • Protocol: Mssql:1433

Redirect Mode: Additional Azure Firewall Network Rule

• In addition to Application Rule mentioned above in case of proxy mode, we also need to add network rule (TCP, port: 11000-11999)
• Instead of creating single rule for all on-premise networks, add one rule for each on-premise network that is trying to connect to Managed Instance
• Add Network Rule
  • Rule collection group: OnPremiseNetworkRuleCollectionGroup
  • Rule collection: OnPremiseNetworkRuleCollection
  • Name: AllowMssqlRedirect-FromOnPremiseNet001-ToManagedInstance
  • Source Type: IP address
  • Source IP address: 10.1.16.0/23 (on-premise network CIDR)
  • Destination type: IP addresses
  • Destination IP addresses: 10.150.40.0/21 (Managed Instance subnet CIDR)
  • Protocol: TCP
  • Port: 11000-11999
• Note: Application Rule mentioned above in case of proxy mode is also required (because redirect mode requires both 1433 and 11000-11999)

Step-3

Managed Instance Network Security Group:

• Proxy mode:
  • “allow_tds_inbound” and “allow_linkedserver_outbound” (TCP, 1433) are automatically created when provisioning from Azure Portal
  • “allow_tds_inbound” and “allow_linkedserver_outbound” allow traffic from within same VNet, peered VNets and connected on-premise networks (because of NSG rule with “VirtualNetwork” tag)
  • In case of IaC i.e. Pulumi, you might need to add those rules manually (Managed Instance creates it’s own necessary rules only)
• Redirect mode:
  • Requires ports 11000-11999 and 1433 open for traffic. Therefore, “allow_tds_inbound” and “allow_linkedserver_outbound” (TCP, 1433) are also required
  • “allow_redirect_inbound” and “allow_redirect_outbound” (TCP, 11000-11999) are automatically created when provisioning from Azure Portal
  • “allow_redirect_inbound” and “allow_redirect_outbound” allow traffic from within same VNet, peered VNets and connected on-premise networks (because of NSG rule with “VirtualNetwork” tag)
  • In case of IaC i.e. Pulumi, you might need to add those rules manually (Managed Instance creates it’s own necessary rules only)
• Hardening inbound NSG rules “allow_tds_inbound” and “allow_redirect_inbound”
  • “VirtualNetwork” service tag is used for source, which will allow traffic from within same VNet, peered VNets and connected on-premise networks
  • Instead of “VirtualNetwork” service tag, use “IP Addresses” as source (to allow specific networks only)
    • Set address space of AzureFirewallSubnet 10.150.193.0/24 (since on-premise traffic will come through Firewall)
    • Set address space of AksAppClusterSubnet 10.150.0.0/19 (since applications running in AKS will access Managed Instance)
    • Set address space of target on-premise network i.e. 10.1.16.0/23
    • Update inbound rule with multiple CIDRs (comma separated). Example: 10.150.193.0/24,10.150.0.0/19,10.1.16.0/23

Managed Instance Route Table (only in case of “Redirect Mode”):

• In case of “Redirect Mode”, returning traffic (response) from Managed Instance to on-premise network must go through Firewall, otherwise “asymmetric routing” will happen
• To avoid “asymmetric routing”:
  • Settings: Configuration > Propagate gateway routes: No
  • Override “0.0.0.0/0” route (Managed Instance adds all necessary routes for management traffic, so would not be any problem)
    • Route name: ToOnPremise-ThroughFirewall
    • Address prefix: 0.0.0.0/0
    • Next hop type: Virtual appliance
    • Next hop address: 10.150.193.4
• Alternative to overriding “0.0.0.0/0” with “Propagate gateway routes: No”
  • You need to add UDRs (to force traffic through Firewall) for all on-premise network that are connecting to Managed Instance (otherwise “asymmetric routing” will happen)
  • Example: UDR for 10.1.16.0/23 on-premise network
    • Route Name: ToOnPremiseNet001-ThroughFirewall
    • Address prefix: 10.1.16.0/23
    • Next hop type: Virtual appliance
    • Next hop address: 10.150.193.4

Step-4

Test from on-premise network

• Install SSMS and upload bakcup (.bak) file in Azure Storage Account blob container. Details: Restoring a sample database to Managed Instance
• Test database access using SSMS

AKS

• First, force outbound traffic through firewall, see Hardening AKS egress traffic
• Application Cluster
  • If AGIC is used, then applications will be accessed from on-premise via application gateway private IP, therefore no need to add Firewall rule targeting AksAppClusterSubnet
  • See “Application Gateway” section below
• Kafka/Other Cluster
  • Kafka/Other type of workload will be served using internal load balancer
  • Add necessary application rules and network rules to Firewall policy

Note: no need to add NSG for AKS subnet

• Add Application Rule with “AKS FQDN tag” to Firewall policy to allow management traffic
• When selecting FQDN Tag in an application rule, the “protocol:port: field must be set to “https”

Application Gateway Private IP

Route for AKS with AGIC and WAF

• No TLS inspection:
  • If Application Gateway have WAF, and Application Gateway and Firewall are in parallel, then no need to add firewall policy
  • No need to force traffic (destined to Firewall private IP) through Firewall, on-premise clients can directly connect to private IP of application gateway
• When TLS inspection is required:
  • Add TLS certificate to Application Gateway
  • Setup mTLS between Firewall Premium and Application Gateway
  • Traffic to private IP of Application Gateway must go through Firewall. Force traffic through Firewall by adding UDR to Route Table of GatewaySubnet (traffic destined for ApplicationGatewaySubnet):
    • Route Name: ToApplicationGatewaySubnet-ThroughFirewall
    • Address prefix: ApplicationGatewaySubnet address space (i.e. 10.150.194.0/19)
    • Next hop type: Virtual appliance
    • Next hop address: Firewall private IP (i.e. 10.150.193.4)
  • Add Application Rules (https:443, http:80) to Firewall policy

PaaS Services

Route for PaaS Services Connected by Private Endpoint

• PaaS services:
  • Storage Account
  • Key Vault
  • ACR
• Use single subnet i.e. “PrivateLinkSubnet” to connect all private endpoints
• Add Application rule with FQDN to Firewall policy, so that applications from on-premise can access private DNS url of PaaS services through Firewall