New-AzureRmHDInsightCluster

Reference

Module:: AzureRM.HDInsight

Creates an Azure HDInsight cluster in the specified resource group for the current subscription.

Warning

The AzureRM PowerShell module has been officially deprecated as of February 29, 2024. Users are advised to migrate from AzureRM to the Az PowerShell module to ensure continued support and updates.

Although the AzureRM module may still function, it's no longer maintained or supported, placing any continued use at the user's discretion and risk. Please refer to our migration resources for guidance on transitioning to the Az module.

Syntax

New-AzureRmHDInsightCluster
   [-Location] <String>
   [-ResourceGroupName] <String>
   [-ClusterName] <String>
   [-ClusterSizeInNodes] <Int32>
   [-HttpCredential] <PSCredential>
   [[-DefaultStorageAccountName] <String>]
   [[-DefaultStorageAccountKey] <String>]
   [-DefaultStorageAccountType <StorageType>]
   [-Config <AzureHDInsightConfig>]
   [-OozieMetastore <AzureHDInsightMetastore>]
   [-HiveMetastore <AzureHDInsightMetastore>]
   [-AdditionalStorageAccounts <System.Collections.Generic.Dictionary`2[System.String,System.String]>]
   [-Configurations <System.Collections.Generic.Dictionary`2[System.String,System.Collections.Generic.Dictionary`2[System.String,System.String]]>]
   [-ScriptActions <System.Collections.Generic.Dictionary`2[Microsoft.Azure.Management.HDInsight.Models.ClusterNodeType,System.Collections.Generic.List`1[Microsoft.Azure.Commands.HDInsight.Models.Management.AzureHDInsightScriptAction]]>]
   [-DefaultStorageContainer <String>]
   [-DefaultStorageRootPath <String>]
   [-Version <String>]
   [-HeadNodeSize <String>]
   [-WorkerNodeSize <String>]
   [-EdgeNodeSize <String>]
   [-ZookeeperNodeSize <String>]
   [-ClusterType <String>]
   [-ComponentVersion <System.Collections.Generic.Dictionary`2[System.String,System.String]>]
   [-VirtualNetworkId <String>]
   [-SubnetName <String>]
   [-OSType <OSType>]
   [-ClusterTier <Tier>]
   [-SshCredential <PSCredential>]
   [-SshPublicKey <String>]
   [-RdpCredential <PSCredential>]
   [-RdpAccessExpiry <DateTime>]
   [-ObjectId <Guid>]
   [-CertificatePassword <String>]
   [-AadTenantId <Guid>]
   [-SecurityProfile <AzureHDInsightSecurityProfile>]
   [-DisksPerWorkerNode <Int32>]
   [-DefaultProfile <IAzureContextContainer>]
   [<CommonParameters>]

New-AzureRmHDInsightCluster
   [-Location] <String>
   [-ResourceGroupName] <String>
   [-ClusterName] <String>
   [-ClusterSizeInNodes] <Int32>
   [-HttpCredential] <PSCredential>
   [[-DefaultStorageAccountName] <String>]
   [[-DefaultStorageAccountKey] <String>]
   [-DefaultStorageAccountType <StorageType>]
   [-Config <AzureHDInsightConfig>]
   [-OozieMetastore <AzureHDInsightMetastore>]
   [-HiveMetastore <AzureHDInsightMetastore>]
   [-AdditionalStorageAccounts <System.Collections.Generic.Dictionary`2[System.String,System.String]>]
   [-Configurations <System.Collections.Generic.Dictionary`2[System.String,System.Collections.Generic.Dictionary`2[System.String,System.String]]>]
   [-ScriptActions <System.Collections.Generic.Dictionary`2[Microsoft.Azure.Management.HDInsight.Models.ClusterNodeType,System.Collections.Generic.List`1[Microsoft.Azure.Commands.HDInsight.Models.Management.AzureHDInsightScriptAction]]>]
   [-DefaultStorageContainer <String>]
   [-DefaultStorageRootPath <String>]
   [-Version <String>]
   [-HeadNodeSize <String>]
   [-WorkerNodeSize <String>]
   [-EdgeNodeSize <String>]
   [-ZookeeperNodeSize <String>]
   [-ClusterType <String>]
   [-ComponentVersion <System.Collections.Generic.Dictionary`2[System.String,System.String]>]
   [-VirtualNetworkId <String>]
   [-SubnetName <String>]
   [-OSType <OSType>]
   [-ClusterTier <Tier>]
   [-SshCredential <PSCredential>]
   [-SshPublicKey <String>]
   [-RdpCredential <PSCredential>]
   [-RdpAccessExpiry <DateTime>]
   [-ObjectId <Guid>]
   [-CertificateFilePath <String>]
   [-CertificatePassword <String>]
   [-AadTenantId <Guid>]
   [-SecurityProfile <AzureHDInsightSecurityProfile>]
   [-DisksPerWorkerNode <Int32>]
   [-DefaultProfile <IAzureContextContainer>]
   [<CommonParameters>]

New-AzureRmHDInsightCluster
   [-Location] <String>
   [-ResourceGroupName] <String>
   [-ClusterName] <String>
   [-ClusterSizeInNodes] <Int32>
   [-HttpCredential] <PSCredential>
   [[-DefaultStorageAccountName] <String>]
   [[-DefaultStorageAccountKey] <String>]
   [-DefaultStorageAccountType <StorageType>]
   [-Config <AzureHDInsightConfig>]
   [-OozieMetastore <AzureHDInsightMetastore>]
   [-HiveMetastore <AzureHDInsightMetastore>]
   [-AdditionalStorageAccounts <System.Collections.Generic.Dictionary`2[System.String,System.String]>]
   [-Configurations <System.Collections.Generic.Dictionary`2[System.String,System.Collections.Generic.Dictionary`2[System.String,System.String]]>]
   [-ScriptActions <System.Collections.Generic.Dictionary`2[Microsoft.Azure.Management.HDInsight.Models.ClusterNodeType,System.Collections.Generic.List`1[Microsoft.Azure.Commands.HDInsight.Models.Management.AzureHDInsightScriptAction]]>]
   [-DefaultStorageContainer <String>]
   [-DefaultStorageRootPath <String>]
   [-Version <String>]
   [-HeadNodeSize <String>]
   [-WorkerNodeSize <String>]
   [-EdgeNodeSize <String>]
   [-ZookeeperNodeSize <String>]
   [-ClusterType <String>]
   [-ComponentVersion <System.Collections.Generic.Dictionary`2[System.String,System.String]>]
   [-VirtualNetworkId <String>]
   [-SubnetName <String>]
   [-OSType <OSType>]
   [-ClusterTier <Tier>]
   [-SshCredential <PSCredential>]
   [-SshPublicKey <String>]
   [-RdpCredential <PSCredential>]
   [-RdpAccessExpiry <DateTime>]
   [-ObjectId <Guid>]
   [-CertificateFileContents <Byte[]>]
   [-CertificatePassword <String>]
   [-AadTenantId <Guid>]
   [-SecurityProfile <AzureHDInsightSecurityProfile>]
   [-DisksPerWorkerNode <Int32>]
   [-DefaultProfile <IAzureContextContainer>]
   [<CommonParameters>]

Description

The New-AzureHDInsightCluster creates an Azure HDInsight cluster by using the specified parameters or by using a configuration object that is created by using the New-AzureRmHDInsightClusterConfig cmdlet.

Examples

Example 1: Create an Azure HDInsight cluster

PS C:\> # Primary storage account info
        $storageAccountResourceGroupName = "Group"
        $storageAccountName = "yourstorageacct001"
        $storageAccountKey = Get-AzureStorageAccountKey `
            -ResourceGroupName $storageAccountResourceGroupName `
            -Name $storageAccountName | %{ $_.Key1 }
        $storageContainer = "container002"

        # Cluster configuration info
        $location = "East US 2"
        $clusterResourceGroupName = "Group"
        $clusterName = "your-hadoop-002"
        $clusterCreds = Get-Credential

        # If the cluster's resource group doesn't exist yet, run:
        #   New-AzureRMResourceGroup -Name $clusterResourceGroupName -Location $location

        # Create the cluster
        New-AzureRmHDInsightCluster `
            -ClusterType Hadoop `
            -OSType Windows `
            -ClusterSizeInNodes 4 `
            -ResourceGroupName $clusterResourceGroupName `
            -ClusterName $clusterName `
            -HttpCredential $clusterCreds `
            -Location $location `
            -DefaultStorageAccountName "$storageAccountName.blob.core.contoso.net" `
            -DefaultStorageAccountKey $storageAccountKey `
            -DefaultStorageContainer $storageContainer

This command creates a cluster in the current subscription.

Parameters

-AadTenantId

Specifies the Microsoft Entra tenant ID that will be used when accessing Azure Data Lake Store.

Type:	Guid
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-AdditionalStorageAccounts

Specifies the additional Azure Storage accounts for the cluster. You can alternatively use the Add-AzureRmHDInsightStorage cmdlet.

Type:	Dictionary<TKey,TValue>[System.String,System.String]
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-CertificateFileContents

Specifies file contents of the certificate that will be used when accessing Azure Data Lake Store.

Type:	Byte[]
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-CertificateFilePath

Specifies the file path to the certificate that will be used to authenticate as the Service Principal. The cluster will use this when accessing Azure Data Lake Store.

Type:	String
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-CertificatePassword

Specifies the password for the certificate that will be used to authenticate as the Service Principal. The cluster will use this when accessing Azure Data Lake Store.

Type:	String
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-ClusterName

Specifies the name of the cluster.

Type:	String
Position:	2
Default value:	None
Required:	True
Accept pipeline input:	False
Accept wildcard characters:	False

-ClusterSizeInNodes

Specifies the number of Worker nodes for the cluster.

Type:	Int32
Position:	3
Default value:	None
Required:	True
Accept pipeline input:	False
Accept wildcard characters:	False

-ClusterTier

Specifies the HDInsight cluster tier. By default, this is Standard. The Premium tier can only be used with Linux clusters, and it enables the use of some new features.

Type:	Tier
Accepted values:	Standard, Premium
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-ClusterType

Specifies the type of cluster to create. Options are: Hadoop, HBase, Storm, Spark

Type:	String
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-ComponentVersion

Type:	Dictionary<TKey,TValue>[System.String,System.String]
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-Config

Specifies the cluster object to be used to create the cluster. This object can be created by using the New-AzureRmHDInsightClusterConfig cmdlet.

Type:	AzureHDInsightConfig
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	True
Accept wildcard characters:	False

-Configurations

Specifies the configurations of this HDInsight cluster. You can alternatively use the Add-AzureRmHDInsightConfigValues cmdlet.

Type:	Dictionary<TKey,TValue>[System.String,System.Collections.Generic.Dictionary`2[System.String,System.String]]
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-DefaultProfile

The credentials, account, tenant, and subscription used for communication with azure

Type:	IAzureContextContainer
Aliases:	AzureRmContext, AzureCredential
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-DefaultStorageAccountKey

Specifies the account key for the default Azure Storage account that the HDInsight cluster will use. You can alternatively use the Set-AzureRmHDInsightDefaultStorage cmdlet.

Type:	String
Position:	6
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-DefaultStorageAccountName

Specifies the name of the default Azure Storage account that the HDInsight cluster will use. You can alternatively use the Set-AzureRmHDInsightDefaultStorage cmdlet.

Type:	String
Position:	5
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-DefaultStorageAccountType

Specifies the type of the default storage account that the HDInsight cluster will use. Possible values are AzureStorage and AzureDataLakeStore. Defaults to AzureStorage if not specified.

Type:	Nullable<T>[StorageType]
Accepted values:	AzureStorage, AzureDataLakeStore
Position:	Named
Default value:	AzureStorage
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-DefaultStorageContainer

Specifies the name of the default container in the default Azure storage account that the HDInsight cluster will use. You can alternatively use the Set-AzureRmHDInsightDefaultStorage cmdlet.

Type:	String
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-DefaultStorageRootPath

Specifies the path-prefix in the Data Lake Store Account that the HDInsight cluster will use as the default filesystem.

Type:	String
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-DisksPerWorkerNode

Specifies the number of disks for worker node role in the cluster.

Type:	Int32
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-EdgeNodeSize

Specifies the size of the virtual machine for the edge node. Use Get-AzureRmVMSize for acceptable VM sizes, and see HDInsight's pricing page. This parameter is valid only for RServer clusters.

Type:	String
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-HeadNodeSize

Specifies the size of the virtual machine for the Head node. Use Get-AzureRmVMSize for acceptable VM sizes, and see HDInsight's pricing page.

Type:	String
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-HiveMetastore

Specifies the SQL Database to store Hive metadata. You can alternatively use the Add-AzureRmHDInsightMetastore cmdlet.

Type:	AzureHDInsightMetastore
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-HttpCredential

Specifies the cluster login (HTTP) credentials for the cluster.

Type:	PSCredential
Position:	4
Default value:	None
Required:	True
Accept pipeline input:	False
Accept wildcard characters:	False

-Location

Specifies the location for the cluster.

Type:	String
Position:	0
Default value:	None
Required:	True
Accept pipeline input:	False
Accept wildcard characters:	False

-ObjectId

Specifies the Microsoft Entra object ID (a GUID) of the Microsoft Entra service principal that represents the cluster. The cluster will use this when accessing Azure Data Lake Store.

Type:	Guid
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-OozieMetastore

Specifies the SQL Database to store Oozie metadata. You can alternatively use the Add-AzureRmHDInsightMetastore cmdlet.

Type:	AzureHDInsightMetastore
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-OSType

Specifies the operating system for the cluster. Options are: Windows, Linux

Type:	OSType
Accepted values:	Windows, Linux
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-RdpAccessExpiry

Specifies the expiration, as a DateTime object, for Remote Desktop Protocol (RDP) access to a cluster.

Type:	DateTime
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-RdpCredential

Specifies the Remote Desktop (RDP) credentials for the cluster. This is only for Windows clusters.

Type:	PSCredential
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-ResourceGroupName

Specifies the name of the resource group.

Type:	String
Position:	1
Default value:	None
Required:	True
Accept pipeline input:	False
Accept wildcard characters:	False

-ScriptActions

Specifies the script actions to run on the cluster at the end of cluster creation. You can alternatively use Add-AzureRmHDInsightScriptAction.

Type:	Dictionary<TKey,TValue>[Microsoft.Azure.Management.HDInsight.Models.ClusterNodeType,System.Collections.Generic.List`1[AzureHDInsightScriptAction]]
Accepted values:	HeadNode, WorkerNode, ZookeeperNode, EdgeNode
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-SecurityProfile

Specifies the security related properties used to create a secure cluster. You can alternatively use the Add-AzureRmHDInsightSecurityProfile cmdlet.

Type:	AzureHDInsightSecurityProfile
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-SshCredential

Specifies the SSH credential to be used for SSH connections. This is only for Linux clusters.

Type:	PSCredential
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-SshPublicKey

Specifies the public key to be used for SSH connections. This is only for Linux clusters.

Type:	String
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-SubnetName

Specifies the name of a subnet within the chosen virtual network for the cluster.

Type:	String
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-Version

Specifies the HDI version of the HDInsight cluster.

Type:	String
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-VirtualNetworkId

Specifies the ID of the virtual network into which to provision the cluster.

Type:	String
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-WorkerNodeSize

Specifies the size of the virtual machine for the Worker node. Use Get-AzureRmVMSize for acceptable VM sizes, and see HDInsight's pricing page.

Type:	String
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

-ZookeeperNodeSize

Specifies the size of the virtual machine for the Zookeeper node. Use Get-AzureRmVMSize for acceptable VM sizes, and see HDInsight's pricing page. This parameter is valid only for HBase or Storm clusters.

Type:	String
Position:	Named
Default value:	None
Required:	False
Accept pipeline input:	False
Accept wildcard characters:	False

Inputs

AzureHDInsightConfig

Parameters: Config (ByValue)

Outputs

AzureHDInsightCluster

Notes

Keywords: azure, azurerm, arm, resource, management, manager, hadoop, hdinsight, hd, insight

New-AzureRmHDInsightClusterConfig

Share via