Configure Azure IoT Layered Network Management Preview on level 4 cluster

Article
10/31/2024

Important

Azure IoT Operations Preview – enabled by Azure Arc is currently in preview. You shouldn't use this preview software in production environments.

You'll need to deploy a new Azure IoT Operations installation when a generally available release becomes available. You won't be able to upgrade a preview installation.

For legal terms that apply to Azure features that are in beta, in preview, or otherwise not yet released into general availability, see the Supplemental Terms of Use for Microsoft Azure Previews.

Azure IoT Layered Network Management Preview is a component to support Azure IoT Operations Preview. However, it needs to be deployed individually to the top network layer for supporting the Azure IoT Operations in the lower layer. In the top level of your network layers (usually level 4 of the ISA-95 network architecture), the cluster and Layered Network Management service have direct internet access. Once the setup is completed, the Layered Network Management service is ready for receiving network traffic from the child layer and forwards it to Azure Arc.

Prerequisites

Meet the following minimum requirements for deploying the Layered Network Management individually on the system.

AKS Edge Essentials - Arc-connected cluster and GitOps category in AKS Edge Essentials requirements and support matrix
K3S Kubernetes cluster - Azure Arc-enabled Kubernetes system requirements

Set up Kubernetes cluster in Level 4

To set up only Layered Network Management, the prerequisites are simpler than an Azure IoT Operations deployment. It's optional to fulfill the general requirements for Azure IoT Operations in Prepare your Kubernetes cluster.

The following steps for setting up AKS Edge Essentials and K3S Kubernetes cluster are verified by Microsoft.

K3S Cluster
AKS Edge Essentials

Prepare an Ubuntu machine

Ubuntu 22.04 LTS is the recommended version for the host machine.
Install Helm 3.8.0 or later.
Install Kubectl.
Install the Azure CLI. You can install the Azure CLI directly onto the level 4 machine or on another developer or jumpbox machine if you plan to access the level 3 cluster remotely. If you choose to access the Kubernetes cluster remotely to keep the cluster host clean, you run the kubectl and az" related commands from the developer machine for the rest of the steps in this article.
- Install Azure CLI. Follow the steps in Install Azure CLI on Linux.
- Install connectedk8s and other extensions.
```
az extension add --name connectedk8s
az extension add --name k8s-extension
```

Create the K3S cluster

Install K3S with the following command:
```
curl -sfL https://get.k3s.io | sh -s - --disable=traefik --write-kubeconfig-mode 644
```
Refer to the K3s quick-start guide for more detail.

Important

Be sure to use the --disable=traefik parameter to disable treafik. Otherwise, you might have an issue when you try to allocate public IP for the Layered Network Management service in later steps.

Copy the K3s configuration yaml file to .kube/config.

mkdir ~/.kube
cp ~/.kube/config ~/.kube/config.back
sudo KUBECONFIG=~/.kube/config:/etc/rancher/k3s/k3s.yaml kubectl config view --flatten > ~/.kube/merged
mv ~/.kube/merged ~/.kube/config
chmod  0600 ~/.kube/config
export KUBECONFIG=~/.kube/config
#switch to k3s context
kubectl config use-context default

For hosting the Layered Network Management service, you need a single machine deployment of AKS Edge Essentials. You can follow the AKS Edge Essentials documentation to create your cluster with default configurations.

Prepare Windows 11

Follow the steps in Prepare your machines for AKS Edge Essentials to set up your Windows machine.
In addition, you need to install the Azure CLI and extensions for later steps.
1. Install Azure CLI. Follow the steps in Install Azure CLI on Windows.
2. Install connectedk8s using the following command:
```
az extension add --name connectedk8s
az extension add --name k8s-extension
```

Create the AKS Edge Essentials cluster

Follow the steps in Single machine deployment to create your cluster.

You need to complete step 1-3 in this document.
In aksedge-config.json from step 1, you only need to make the following adjustment for Layered Network Management. You can keep the default value for the rest of the parameters. Otherwise, make proper adjustments based on your environment.
```
"Init": {
    "ServiceIPRangeSize": 10
  },
```

Sign in with Azure CLI. To avoid permission issues later, it's important that you sign in interactively using a browser window:
```
az login
```

Set environment variables for the setup steps. Replace values in <> with valid values or names of your choice. The CLUSTER_NAME and RESOURCE_GROUP are created based on the names you provide. Refer to Azure IoT Operations supported regions for choosing the LOCATION.

# Id of the subscription where your resource group and Arc-enabled cluster will be created
$SUBSCRIPTION_ID = "<subscription-id>"
# Azure region where the created resource group will be located
$LOCATION = "<region>"
# Name of a new resource group to create which will hold the Arc-enabled cluster and Azure IoT Operations resources
$RESOURCE_GROUP = "<resource-group-name>"
# Name of the Arc-enabled cluster to create in your resource group
$CLUSTER_NAME = "<cluster-name>"

Set the Azure subscription context for all commands:
```
az account set -s $SUBSCRIPTION_ID
```
Register the required resource providers in your subscription:

Note

This step only needs to be run once per subscription. To register resource providers, you need permission to do the /register/action operation, which is included in subscription Contributor and Owner roles. For more information, see Azure resource providers and types.
```
az provider register -n "Microsoft.ExtendedLocation"
az provider register -n "Microsoft.Kubernetes"
az provider register -n "Microsoft.KubernetesConfiguration"
```
Use the az group create command to create a resource group in your Azure subscription to store all the resources:
```
az group create --location $LOCATION --resource-group $RESOURCE_GROUP --subscription $SUBSCRIPTION_ID
```
Use the az connectedk8s connect command to Arc-enable your Kubernetes cluster and manage it in the resource group you created in the previous step:
```
az connectedk8s connect -n $CLUSTER_NAME -l $LOCATION -g $RESOURCE_GROUP --subscription $SUBSCRIPTION_ID
```

Deploy Layered Network Management Preview Service to the cluster

Once your Kubernetes cluster is Arc-enabled, you can deploy the Layered Network Management service to the cluster.

Install the Layered Network Management Preview operator

Run the following command. Replace the placeholders <RESOURCE GROUP> and <CLUSTER NAME> with your Arc onboarding information from an earlier step.

az login

az k8s-extension create --resource-group <RESOURCE GROUP> --name kind-lnm-extension --cluster-type connectedClusters --cluster-name <CLUSTER NAME> --auto-upgrade false --extension-type Microsoft.IoTOperations.LayeredNetworkManagement --version 0.1.0-preview --release-train preview

Use the kubectl command to verify the Layered Network Management operator is running.

kubectl get pods

NAME                                   READY   STATUS    RESTARTS   AGE
azedge-lnm-operator-598cc495c-5428j   1/1     Running   0          28h

Configure Layered Network Management Preview Service

Create the Layered Network Management custom resource.

Create a lnm-cr.yaml file as specified:

apiVersion: layerednetworkmgmt.iotoperations.azure.com/v1beta1
kind: Lnm
metadata:
  name: level4
  namespace: default
spec:
  image:
    pullPolicy: IfNotPresent
    repository: mcr.microsoft.com/oss/envoyproxy/envoy-distroless
    tag: v1.27.0
  replicas: 1
  logLevel: "debug"
  openTelemetryMetricsCollectorAddr: "http://aio-otel-collector.azure-iot-operations.svc.cluster.local:4317"
  level: 4
  allowList:
    enableArcDomains: true
    domains:
    - destinationUrl: "*.arc.azure.net"
      destinationType: external
    - destinationUrl: "*.data.mcr.microsoft.com"
      destinationType: external
    - destinationUrl: "*.dp.kubernetesconfiguration.azure.com"
      destinationType: external
    - destinationUrl: "*.guestnotificationservice.azure.com"
      destinationType: external
    - destinationUrl: "*.his.arc.azure.com"
      destinationType: external
    - destinationUrl: "*.login.microsoft.com"
      destinationType: external
    - destinationUrl: "*.login.microsoftonline.com"
      destinationType: external
    - destinationUrl: "*.obo.arc.azure.com"
      destinationType: external
    - destinationUrl: "*.servicebus.windows.net"
      destinationType: external
    - destinationUrl: "graph.microsoft.com"
      destinationType: external
    - destinationUrl: "login.windows.net"
      destinationType: external
    - destinationUrl: "management.azure.com"
      destinationType: external
    - destinationUrl: "mcr.microsoft.com"
      destinationType: external
    - destinationUrl: "sts.windows.net"
      destinationType: external
    - destinationUrl: "*.ods.opinsights.azure.com"
      destinationType: external
    - destinationUrl: "graph.windows.net"
      destinationType: external
    - destinationUrl: "msit-onelake.pbidedicated.windows.net"
      destinationType: external
    - destinationUrl: "*.azurecr.io"
      destinationType: external
    - destinationUrl: "*.azureedge.net"
      destinationType: external
    - destinationUrl: "*.blob.core.windows.net"
      destinationType: external
    - destinationUrl: "*.prod.hot.ingestion.msftcloudes.com"
      destinationType: external
    - destinationUrl: "*.prod.microsoftmetrics.com"
      destinationType: external
    - destinationUrl: "adhs.events.data.microsoft.com"
      destinationType: external
    - destinationUrl: "dc.services.visualstudio.com"
      destinationType: external
    - destinationUrl: "go.microsoft.com"
      destinationType: external
    - destinationUrl: "packages.microsoft.com"
      destinationType: external
    - destinationUrl: "www.powershellgallery.com"
      destinationType: external
    - destinationUrl: "*.gw.arc.azure.com"
      destinationType: external
    - destinationUrl: "*.gcs.prod.monitoring.core.windows.net"
      destinationType: external
    - destinationUrl: "*.prod.warm.ingest.monitor.core.windows.net"
      destinationType: external
    - destinationUrl: "*.prod.hot.ingest.monitor.core.windows.net"
      destinationType: external
    - destinationUrl: "azure.archive.ubuntu.com"
      destinationType: external
    - destinationUrl: "crl.microsoft.com"
      destinationType: external
    - destinationUrl: "*.table.core.windows.net"
      destinationType: external
    - destinationUrl: "*.blob.storage.azure.net"
      destinationType: external
    - destinationUrl: "*.docker.com"
      destinationType: external
    - destinationUrl: "*.docker.io"
      destinationType: external
    - destinationUrl: "*.googleapis.com"
      destinationType: external
    - destinationUrl: "github.com"
      destinationType: external
    - destinationUrl: "collect.traefik.io"
      destinationType: external
    - destinationUrl: "contracts.canonical.com"
      destinationType: external
    - destinationUrl: "database.clamav.net"
      destinationType: external
    - destinationUrl: "esm.ubuntu.com"
      destinationType: external
    - destinationUrl: "livepatch.canonical.com"
      destinationType: external
    - destinationUrl: "motd.ubuntu.com"
      destinationType: external
    - destinationUrl: "update.traefik.io"
      destinationType: external
    sourceIpRange:
    - addressPrefix: "0.0.0.0"
      prefixLen: 0

For debugging or experimentation, you can change the value of loglevel parameter to debug.

Create the Custom Resource to create a Layered Network Management instance.
```
kubectl apply -f lnm-cr.yaml
```

View the Layered Network Management Kubernetes service:

kubectl get services -n azure-iot-operations

NAME           TYPE           CLUSTER-IP    EXTERNAL-IP   PORT(S)                                      AGE
lnm-level-4   LoadBalancer   10.43.91.54   192.168.0.4   80:30530/TCP,443:31117/TCP,10000:31914/TCP   95s

Add iptables configuration for AKS Edge Essentials

Important

This step is applicable only when hosting the Layered Network Management in an AKS Edge Essentials cluster.

The Layered Network Management deployment creates a Kubernetes service of type LoadBalancer. To ensure that the service is accessible from outside the Kubernetes cluster, you need to map the underlying Windows host's ports to the appropriate ports on the Layered Network Management service.

netsh interface portproxy add v4tov4 listenport=443 listenaddress=0.0.0.0 connectport=443 connectaddress=192.168.0.4
netsh interface portproxy add v4tov4 listenport=10000 listenaddress=0.0.0.0 connectport=10000 connectaddress=192.168.0.4

After these commands are run successfully, traffic received on ports 443 and 10000 on the Windows host is routed through to the Kubernetes service. When configuring customized DNS for the child level network layer, you direct the network traffic to the IP of this Windows host and then to the Layered Network Management service running on it.

Create sample network environment

Share via

Configure Azure IoT Layered Network Management Preview on level 4 cluster

Prerequisites

Set up Kubernetes cluster in Level 4

Prepare an Ubuntu machine

Create the K3S cluster

Prepare Windows 11

Create the AKS Edge Essentials cluster

Arc enable the cluster

Deploy Layered Network Management Preview Service to the cluster

Install the Layered Network Management Preview operator

Configure Layered Network Management Preview Service

Add iptables configuration for AKS Edge Essentials

Feedback

Additional resources

Share via

Configure Azure IoT Layered Network Management Preview on level 4 cluster

Prerequisites

Set up Kubernetes cluster in Level 4

Prepare an Ubuntu machine

Create the K3S cluster

Arc enable the cluster

Deploy Layered Network Management Preview Service to the cluster

Install the Layered Network Management Preview operator

Configure Layered Network Management Preview Service

Add iptables configuration for AKS Edge Essentials

Related content

Feedback

Additional resources