Quickstart: Create an Azure Nexus Kubernetes cluster by using Azure CLI

• Deploy an Azure Nexus Kubernetes cluster using Azure CLI.

Before you begin

If you don't have an Azure subscription, create an Azure free account before you begin.

• Use the Bash environment in Azure Cloud Shell. For more information, see Quickstart for Bash in Azure Cloud Shell.

  

• If you prefer to run CLI reference commands locally, install the Azure CLI. If you're running on Windows or macOS, consider running Azure CLI in a Docker container. For more information, see How to run the Azure CLI in a Docker container.

  • If you're using a local installation, sign in to the Azure CLI by using the az login command. To finish the authentication process, follow the steps displayed in your terminal. For other sign-in options, see Sign in with the Azure CLI.

  • When you're prompted, install the Azure CLI extension on first use. For more information about extensions, see Use extensions with the Azure CLI.

  • Run az version to find the version and dependent libraries that are installed. To upgrade to the latest version, run az upgrade.

• Install the latest version of the necessary Azure CLI extensions.

• This article requires version 2.61.0 or later of the Azure CLI. If using Azure Cloud Shell, the latest version is already installed.

• If you have multiple Azure subscriptions, select the appropriate subscription ID in which the resources should be billed using the az account command.

• Refer the VM SKU table in the reference section for the list of supported VM SKUs.

• Refer the supported Kubernetes versions for the list of supported Kubernetes versions.

• Create a resource group using the az group create command. An Azure resource group is a logical group in which Azure resources are deployed and managed. When you create a resource group, you're prompted to specify a location. This location is the storage location of your resource group metadata and where your resources run in Azure if you don't specify another region during resource creation. The following example creates a resource group named myResourceGroup in the eastus location.

  az group create --name myResourceGroup --location eastus
  

  The following output example resembles successful creation of the resource group:

  {
    "id": "/subscriptions/<guid>/resourceGroups/myResourceGroup",
    "location": "eastus",
    "managedBy": null,
    "name": "myResourceGroup",
    "properties": {
      "provisioningState": "Succeeded"
    },
    "tags": null
  }
  

• To deploy a Bicep file or ARM template, you need write access on the resources you're deploying and access to all operations on the Microsoft.Resources/deployments resource type. For example, to deploy a cluster, you need Microsoft.NetworkCloud/kubernetesclusters/write and Microsoft.Resources/deployments/* permissions. For a list of roles and permissions, see Azure built-in roles.

• You need the custom location resource ID of your Azure Operator Nexus cluster.

• You need to create various networks according to your specific workload requirements, and it's essential to have the appropriate IP addresses available for your workloads. To ensure a smooth implementation, it's advisable to consult the relevant support teams for assistance.

• This quickstart assumes a basic understanding of Kubernetes concepts. For more information, see Kubernetes core concepts for Azure Kubernetes Service (AKS).

Create an Azure Nexus Kubernetes cluster

The following example creates a cluster named myNexusK8sCluster in resource group myResourceGroup in the eastus location.

Before you run the commands, you need to set several variables to define the configuration for your cluster. Here are the variables you need to set, along with some default values you can use for certain variables:

Once you've defined these variables, you can run the Azure CLI command to create the cluster. Add the --debug flag at the end to provide more detailed output for troubleshooting purposes.

To define these variables, use the following set commands and replace the example values with your preferred values. You can also use the default values for some of the variables, as shown in the following example:

RESOURCE_GROUP="myResourceGroup"
SUBSCRIPTION_ID="<Azure subscription ID>"
LOCATION="$(az group show --name $RESOURCE_GROUP --query location --subscription $SUBSCRIPTION_ID -o tsv)"
CUSTOM_LOCATION="/subscriptions/<subscription_id>/resourceGroups/<managed_resource_group>/providers/microsoft.extendedlocation/customlocations/<custom-location-name>"
CSN_ARM_ID="/subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.NetworkCloud/cloudServicesNetworks/<csn-name>"
CNI_ARM_ID="/subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.NetworkCloud/l3Networks/<l3Network-name>"
AAD_ADMIN_GROUP_OBJECT_ID="00000000-0000-0000-0000-000000000000"
CLUSTER_NAME="myNexusK8sCluster"
K8S_VERSION="v1.24.9"
ADMIN_USERNAME="azureuser"
SSH_PUBLIC_KEY="$(cat ~/.ssh/id_rsa.pub)"
CONTROL_PLANE_SSH_PUBLIC_KEY="$(cat ~/.ssh/id_rsa.pub)"
AGENT_POOL_SSH_PUBLIC_KEY="$(cat ~/.ssh/id_rsa.pub)"
CONTROL_PLANE_COUNT="1"
CONTROL_PLANE_VM_SIZE="NC_G6_28_v1"
INITIAL_AGENT_POOL_NAME="${CLUSTER_NAME}-nodepool-1"
INITIAL_AGENT_POOL_COUNT="1"
INITIAL_AGENT_POOL_VM_SIZE="NC_P10_56_v1"
POD_CIDR="10.244.0.0/16"
SERVICE_CIDR="10.96.0.0/16"
DNS_SERVICE_IP="10.96.0.10"

After defining these variables, you can create the Kubernetes cluster by executing the following Azure CLI command:

az networkcloud kubernetescluster create \
  --name "${CLUSTER_NAME}" \
  --resource-group "${RESOURCE_GROUP}" \
  --subscription "${SUBSCRIPTION_ID}" \
  --extended-location name="${CUSTOM_LOCATION}" type=CustomLocation \
  --location "${LOCATION}" \
  --kubernetes-version "${K8S_VERSION}" \
  --aad-configuration admin-group-object-ids="[${AAD_ADMIN_GROUP_OBJECT_ID}]" \
  --admin-username "${ADMIN_USERNAME}" \
  --ssh-key-values "${SSH_PUBLIC_KEY}" \
  --control-plane-node-configuration \
    count="${CONTROL_PLANE_COUNT}" \
    vm-sku-name="${CONTROL_PLANE_VM_SIZE}" \
    ssh-key-values='["${CONTROL_PLANE_SSH_PUBLIC_KEY}"]' \
  --initial-agent-pool-configurations "[{count:${INITIAL_AGENT_POOL_COUNT},mode:System,name:${INITIAL_AGENT_POOL_NAME},vm-sku-name:${INITIAL_AGENT_POOL_VM_SIZE},ssh-key-values:['${AGENT_POOL_SSH_PUBLIC_KEY}']}]" \
  --network-configuration \
    cloud-services-network-id="${CSN_ARM_ID}" \
    cni-network-id="${CNI_ARM_ID}" \
    pod-cidrs="[${POD_CIDR}]" \
    service-cidrs="[${SERVICE_CIDR}]" \
    dns-service-ip="${DNS_SERVICE_IP}"

If there isn't enough capacity to deploy requested cluster nodes, an error message appears. However, this message doesn't provide any details about the available capacity. It states that the cluster creation can't proceed due to insufficient capacity.

After a few minutes, the command completes and returns information about the cluster. For more advanced options, see Quickstart: Deploy an Azure Nexus Kubernetes cluster using Bicep.

Review deployed resources

After the deployment finishes, you can view the resources using the CLI or the Azure portal.

To view the details of the myNexusK8sCluster cluster in the myResourceGroup resource group, execute the following Azure CLI command:

az networkcloud kubernetescluster show \
  --name myNexusK8sCluster \
  --resource-group myResourceGroup

Additionally, to get a list of agent pool names associated with the myNexusK8sCluster cluster in the myResourceGroup resource group, you can use the following Azure CLI command.

az networkcloud kubernetescluster agentpool list \
  --kubernetes-cluster-name myNexusK8sCluster \
  --resource-group myResourceGroup \
  --output table

Connect to the cluster

Now that the Nexus Kubernetes cluster has been successfully created and connected to Azure Arc, you can easily connect to it using the cluster connect feature. Cluster connect allows you to securely access and manage your cluster from anywhere, making it convenient for interactive development, debugging, and cluster administration tasks.

For more detailed information about available options, see Connect to an Azure Operator Nexus Kubernetes cluster.

To access your cluster, you need to set up the cluster connect kubeconfig. After logging into Azure CLI with the relevant Microsoft Entra entity, you can obtain the kubeconfig necessary to communicate with the cluster from anywhere, even outside the firewall that surrounds it.

1. Set CLUSTER_NAME, RESOURCE_GROUP and SUBSCRIPTION_ID variables.

   CLUSTER_NAME="myNexusK8sCluster"
   RESOURCE_GROUP="myResourceGroup"
   SUBSCRIPTION_ID=<set the correct subscription_id>
   

2. Query managed resource group with az and store in MANAGED_RESOURCE_GROUP

    az account set -s $SUBSCRIPTION_ID
    MANAGED_RESOURCE_GROUP=$(az networkcloud kubernetescluster show -n $CLUSTER_NAME -g $RESOURCE_GROUP --output tsv --query managedResourceGroupConfiguration.name)
   

3. The following command starts a connectedk8s proxy that allows you to connect to the Kubernetes API server for the specified Nexus Kubernetes cluster.

   az connectedk8s proxy -n $CLUSTER_NAME  -g $MANAGED_RESOURCE_GROUP &
   

4. Use kubectl to send requests to the cluster:

   kubectl get pods -A
   

   You should now see a response from the cluster containing the list of all nodes.

Add an agent pool

The cluster created in the previous step has a single node pool. Let's add a second agent pool using the az networkcloud kubernetescluster agentpool create command. The following example creates an agent pool named myNexusK8sCluster-nodepool-2:

You can also use the default values for some of the variables, as shown in the following example:

RESOURCE_GROUP="myResourceGroup"
CUSTOM_LOCATION="/subscriptions/<subscription_id>/resourceGroups/<managed_resource_group>/providers/microsoft.extendedlocation/customlocations/<custom-location-name>"
CLUSTER_NAME="myNexusK8sCluster"
AGENT_POOL_NAME="${CLUSTER_NAME}-nodepool-2"
AGENT_POOL_VM_SIZE="NC_P10_56_v1"
AGENT_POOL_COUNT="1"
AGENT_POOL_MODE="User"

After defining these variables, you can add an agent pool by executing the following Azure CLI command:

az networkcloud kubernetescluster agentpool create \
  --name "${AGENT_POOL_NAME}" \
  --kubernetes-cluster-name "${CLUSTER_NAME}" \
  --resource-group "${RESOURCE_GROUP}" \
  --subscription "${SUBSCRIPTION_ID}" \
  --extended-location name="${CUSTOM_LOCATION}" type=CustomLocation \
  --count "${AGENT_POOL_COUNT}" \
  --mode "${AGENT_POOL_MODE}" \
  --vm-sku-name "${AGENT_POOL_VM_SIZE}"

After a few minutes, the command completes and returns information about the agent pool. For more advanced options, see Quickstart: Deploy an Azure Nexus Kubernetes cluster using Bicep.

The following output example resembles successful creation of the agent pool.

$ az networkcloud kubernetescluster agentpool list --kubernetes-cluster-name myNexusK8sCluster --resource-group myResourceGroup --output table
This command is experimental and under development. Reference and support levels: https://aka.ms/CLI_refstatus
Count    Location    Mode    Name                          ProvisioningState    ResourceGroup    VmSkuName
-------  ----------  ------  ----------------------------  -------------------  ---------------  -----------
1        eastus      System  myNexusK8sCluster-nodepool-1  Succeeded            myResourceGroup  NC_P10_56_v1
1        eastus      User    myNexusK8sCluster-nodepool-2  Succeeded            myResourceGroup  NC_P10_56_v1

Clean up resources

When no longer needed, delete the resource group. The resource group and all the resources in the resource group are deleted.

az group delete --name myResourceGroup --yes --no-wait

Remove-AzResourceGroup -Name myResourceGroup

Next steps

You can now deploy the CNFs either directly via cluster connect or via Azure Operator Service Manager.