Submit-AzSynapseSparkJob
Submits a Synapse Analytics Spark job.
Syntax
Submit-AzSynapseSparkJob
-WorkspaceName <String>
-SparkPoolName <String>
-Language <String>
-Name <String>
-MainDefinitionFile <String>
[-MainClassName <String>]
[-CommandLineArgument <String[]>]
[-ReferenceFile <String[]>]
-ExecutorCount <Int32>
-ExecutorSize <String>
[-Configuration <Hashtable>]
[-DefaultProfile <IAzureContextContainer>]
[-WhatIf]
[-Confirm]
[<CommonParameters>] Submit-AzSynapseSparkJob
-SparkPoolObject <PSSynapseSparkPool>
-Language <String>
-Name <String>
-MainDefinitionFile <String>
[-MainClassName <String>]
[-CommandLineArgument <String[]>]
[-ReferenceFile <String[]>]
-ExecutorCount <Int32>
-ExecutorSize <String>
[-Configuration <Hashtable>]
[-DefaultProfile <IAzureContextContainer>]
[-WhatIf]
[-Confirm]
[<CommonParameters>] Description
The Submit-AzSynapseSparkJob cmdlet submits a Synapse Analytics Spark job.
Examples
Example 1
Submit-AzSynapseSparkJob -WorkspaceName ContosoWorkspace -SparkPoolName ContosoSparkPool -Language Spark -Name WordCount_Java -MainDefinitionFile abfss://ContosoFileSystem@ContosoGen2Storage.dfs.core.windows.net/samples/java/wordcount/wordcount.jar -MainClassName WordCount -CommandLineArgument abfss://ContosoFileSystem@ContosoGen2Storage.dfs.core.windows.net/samples/java/wordcount/shakespeare.txt,abfss://ContosoFileSystem@ContosoGen2Storage.dfs.core.windows.net/samples/java/wordcount/result/ -ExecutorCount 2 -ExecutorSize SmallThis command submits a Synapse Analytics Spark job.
Example 2
Submit-AzSynapseSparkJob -WorkspaceName ContosoWorkspace -SparkPoolName ContosoSparkPool -Language SparkDotNet -Name WordCount_Dotnet -MainDefinitionFile abfss://ContosoFileSystem@ContosoGen2Storage.dfs.core.windows.net/samples/dotnet/wordcount/wordcount.zip -MainExecutableFile WordCount -CommandLineArgument abfss://ContosoFileSystem@ContosoGen2Storage.dfs.core.windows.net/samples/dotnet/wordcount/shakespeare.txt,abfss://ContosoFileSystem@ContosoGen2Storage.dfs.core.windows.net/samples/dotnet/wordcount/result -ExecutorCount 2 -ExecutorSize SmallThis command submits a Synapse Analytics Spark .NET job.
Example 3
Submit-AzSynapseSparkJob -WorkspaceName ContosoWorkspace -SparkPoolName ContosoSparkPool -Language PySpark -Name WordCount_Python -MainDefinitionFile abfss://ContosoFileSystem@ContosoGen2Storage.blob.core.windows.net/samples/python/wordcount/wordcount.py -CommandLineArgument abfss://ContosoFileSystem@ContosoGen2Storage.blob.core.windows.net/samples/python/wordcount/shakespeare.txt,abfss://ContosoFileSystem@ContosoGen2Storage.blob.core.windows.net/samples/python/wordcount/result/ -ExecutorCount 2 -ExecutorSize SmallThis command submits a Synapse Analytics PySpark job.
Parameters
-CommandLineArgument
Optional arguments to the job. e.g. "--iteration 10000 --timeout 20s"
| Type: | String[] |
| Position: | Named |
| Default value: | None |
| Required: | False |
| Accept pipeline input: | False |
| Accept wildcard characters: | False |
-Configuration
Spark configuration properties.
| Type: | Hashtable |
| Position: | Named |
| Default value: | None |
| Required: | False |
| Accept pipeline input: | False |
| Accept wildcard characters: | False |
-Confirm
Prompts you for confirmation before running the cmdlet.
| Type: | SwitchParameter |
| Aliases: | cf |
| Position: | Named |
| Default value: | None |
| Required: | False |
| Accept pipeline input: | False |
| Accept wildcard characters: | False |
-DefaultProfile
The credentials, account, tenant, and subscription used for communication with Azure.
| Type: | IAzureContextContainer |
| Aliases: | AzContext, AzureRmContext, AzureCredential |
| Position: | Named |
| Default value: | None |
| Required: | False |
| Accept pipeline input: | False |
| Accept wildcard characters: | False |
-ExecutorCount
Number of executors to be allocated in the specified Spark pool for the job.
| Type: | Int32 |
| Position: | Named |
| Default value: | None |
| Required: | True |
| Accept pipeline input: | False |
| Accept wildcard characters: | False |
-ExecutorSize
Number of core and memory to be used for executors allocated in the specified Spark pool for the job.
| Type: | String |
| Accepted values: | Small, Medium, Large, XLarge, XXLarge, XXXLarge |
| Position: | Named |
| Default value: | None |
| Required: | True |
| Accept pipeline input: | False |
| Accept wildcard characters: | False |
-Language
Language of the job to submit.
| Type: | String |
| Accepted values: | Spark, Scala, PySpark, Python, SparkDotNet, CSharp |
| Position: | Named |
| Default value: | None |
| Required: | True |
| Accept pipeline input: | False |
| Accept wildcard characters: | False |
-MainClassName
The fully-qualified identifier or the main class that is in the main definition file. Required for Spark and .NET Spark job. e.g. "org.apache.spark.examples.SparkPi"
| Type: | String |
| Aliases: | MainExecutableFile |
| Position: | Named |
| Default value: | None |
| Required: | False |
| Accept pipeline input: | False |
| Accept wildcard characters: | False |
-MainDefinitionFile
The main file used for the job. e.g. "abfss://filesystem@account.dfs.core.windows.net/mySpark.jar"
| Type: | String |
| Position: | Named |
| Default value: | None |
| Required: | True |
| Accept pipeline input: | False |
| Accept wildcard characters: | False |
-Name
Name of Spark job.
| Type: | String |
| Position: | Named |
| Default value: | None |
| Required: | True |
| Accept pipeline input: | False |
| Accept wildcard characters: | False |
-ReferenceFile
Additional files used for reference in the main definition file. Comma-separated storage URI list. e.g. "abfss://filesystem@account.dfs.core.windows.net/file1.txt,abfss://filesystem@account.dfs.core.windows.net/result/"
| Type: | String[] |
| Position: | Named |
| Default value: | None |
| Required: | False |
| Accept pipeline input: | False |
| Accept wildcard characters: | False |
-SparkPoolName
Name of Synapse Spark pool.
| Type: | String |
| Position: | Named |
| Default value: | None |
| Required: | True |
| Accept pipeline input: | False |
| Accept wildcard characters: | False |
-SparkPoolObject
Spark pool input object, usually passed through the pipeline.
| Type: | PSSynapseSparkPool |
| Position: | Named |
| Default value: | None |
| Required: | True |
| Accept pipeline input: | True |
| Accept wildcard characters: | False |
-WhatIf
Shows what would happen if the cmdlet runs. The cmdlet is not run.
| Type: | SwitchParameter |
| Aliases: | wi |
| Position: | Named |
| Default value: | None |
| Required: | False |
| Accept pipeline input: | False |
| Accept wildcard characters: | False |
-WorkspaceName
Name of Synapse workspace.
| Type: | String |
| Position: | Named |
| Default value: | None |
| Required: | True |
| Accept pipeline input: | False |
| Accept wildcard characters: | False |
