SynapseSparkJobDefinitionActivity Class

Reference

Execute spark job activity.

All required parameters must be populated in order to send to server.

Inheritance: azure.mgmt.datafactory.models._models_py3.ExecutionActivity

SynapseSparkJobDefinitionActivity

Constructor

SynapseSparkJobDefinitionActivity(*, name: str, spark_job: _models.SynapseSparkJobReference, additional_properties: Dict[str, MutableMapping[str, Any]] | None = None, description: str | None = None, state: str | _models.ActivityState | None = None, on_inactive_mark_as: str | _models.ActivityOnInactiveMarkAs | None = None, depends_on: List[_models.ActivityDependency] | None = None, user_properties: List[_models.UserProperty] | None = None, linked_service_name: _models.LinkedServiceReference | None = None, policy: _models.ActivityPolicy | None = None, arguments: List[Any] | None = None, file: MutableMapping[str, Any] | None = None, scan_folder: MutableMapping[str, Any] | None = None, class_name: MutableMapping[str, Any] | None = None, files: List[MutableMapping[str, Any]] | None = None, python_code_reference: List[MutableMapping[str, Any]] | None = None, files_v2: List[MutableMapping[str, Any]] | None = None, target_big_data_pool: _models.BigDataPoolParametrizationReference | None = None, executor_size: MutableMapping[str, Any] | None = None, conf: MutableMapping[str, Any] | None = None, driver_size: MutableMapping[str, Any] | None = None, num_executors: MutableMapping[str, Any] | None = None, configuration_type: str | _models.ConfigurationType | None = None, target_spark_configuration: _models.SparkConfigurationParametrizationReference | None = None, spark_config: Dict[str, MutableMapping[str, Any]] | None = None, **kwargs: Any)

Keyword-Only Parameters

Name	Description
additional_properties	dict[str, <xref:JSON>] Unmatched properties from the message are deserialized to this collection.
name	str Activity name. Required.
description	str Activity description.
state	str or ActivityState Activity state. This is an optional property and if not provided, the state will be Active by default. Known values are: "Active" and "Inactive".
on_inactive_mark_as	str or ActivityOnInactiveMarkAs Status result of the activity when the state is set to Inactive. This is an optional property and if not provided when the activity is inactive, the status will be Succeeded by default. Known values are: "Succeeded", "Failed", and "Skipped".
depends_on	list[ActivityDependency] Activity depends on condition.
user_properties	list[UserProperty] Activity user properties.
linked_service_name	LinkedServiceReference Linked service reference.
policy	ActivityPolicy Activity policy.
spark_job	SynapseSparkJobReference Synapse spark job reference. Required.
arguments	list[any] User specified arguments to SynapseSparkJobDefinitionActivity.
file	<xref:JSON> The main file used for the job, which will override the 'file' of the spark job definition you provide. Type: string (or Expression with resultType string).
scan_folder	<xref:JSON> Scanning subfolders from the root folder of the main definition file, these files will be added as reference files. The folders named 'jars', 'pyFiles', 'files' or 'archives' will be scanned, and the folders name are case sensitive. Type: boolean (or Expression with resultType boolean).
class_name	<xref:JSON> The fully-qualified identifier or the main class that is in the main definition file, which will override the 'className' of the spark job definition you provide. Type: string (or Expression with resultType string).
files	list[<xref:JSON>] (Deprecated. Please use pythonCodeReference and filesV2) Additional files used for reference in the main definition file, which will override the 'files' of the spark job definition you provide.
python_code_reference	list[<xref:JSON>] Additional python code files used for reference in the main definition file, which will override the 'pyFiles' of the spark job definition you provide.
files_v2	list[<xref:JSON>] Additional files used for reference in the main definition file, which will override the 'jars' and 'files' of the spark job definition you provide.
target_big_data_pool	BigDataPoolParametrizationReference The name of the big data pool which will be used to execute the spark batch job, which will override the 'targetBigDataPool' of the spark job definition you provide.
executor_size	<xref:JSON> Number of core and memory to be used for executors allocated in the specified Spark pool for the job, which will be used for overriding 'executorCores' and 'executorMemory' of the spark job definition you provide. Type: string (or Expression with resultType string).
conf	<xref:JSON> Spark configuration properties, which will override the 'conf' of the spark job definition you provide.
driver_size	<xref:JSON> Number of core and memory to be used for driver allocated in the specified Spark pool for the job, which will be used for overriding 'driverCores' and 'driverMemory' of the spark job definition you provide. Type: string (or Expression with resultType string).
num_executors	<xref:JSON> Number of executors to launch for this job, which will override the 'numExecutors' of the spark job definition you provide. Type: integer (or Expression with resultType integer).
configuration_type	str or ConfigurationType The type of the spark config. Known values are: "Default", "Customized", and "Artifact".
target_spark_configuration	SparkConfigurationParametrizationReference The spark configuration of the spark job.
spark_config	dict[str, <xref:JSON>] Spark configuration property.

Variables

Name	Description
additional_properties	dict[str, <xref:JSON>] Unmatched properties from the message are deserialized to this collection.
name	str Activity name. Required.
type	str Type of activity. Required.
description	str Activity description.
state	str or ActivityState Activity state. This is an optional property and if not provided, the state will be Active by default. Known values are: "Active" and "Inactive".
on_inactive_mark_as	str or ActivityOnInactiveMarkAs Status result of the activity when the state is set to Inactive. This is an optional property and if not provided when the activity is inactive, the status will be Succeeded by default. Known values are: "Succeeded", "Failed", and "Skipped".
depends_on	list[ActivityDependency] Activity depends on condition.
user_properties	list[UserProperty] Activity user properties.
linked_service_name	LinkedServiceReference Linked service reference.
policy	ActivityPolicy Activity policy.
spark_job	SynapseSparkJobReference Synapse spark job reference. Required.
arguments	list[any] User specified arguments to SynapseSparkJobDefinitionActivity.
file	<xref:JSON> The main file used for the job, which will override the 'file' of the spark job definition you provide. Type: string (or Expression with resultType string).
scan_folder	<xref:JSON> Scanning subfolders from the root folder of the main definition file, these files will be added as reference files. The folders named 'jars', 'pyFiles', 'files' or 'archives' will be scanned, and the folders name are case sensitive. Type: boolean (or Expression with resultType boolean).
class_name	<xref:JSON> The fully-qualified identifier or the main class that is in the main definition file, which will override the 'className' of the spark job definition you provide. Type: string (or Expression with resultType string).
files	list[<xref:JSON>] (Deprecated. Please use pythonCodeReference and filesV2) Additional files used for reference in the main definition file, which will override the 'files' of the spark job definition you provide.
python_code_reference	list[<xref:JSON>] Additional python code files used for reference in the main definition file, which will override the 'pyFiles' of the spark job definition you provide.
files_v2	list[<xref:JSON>] Additional files used for reference in the main definition file, which will override the 'jars' and 'files' of the spark job definition you provide.
target_big_data_pool	BigDataPoolParametrizationReference The name of the big data pool which will be used to execute the spark batch job, which will override the 'targetBigDataPool' of the spark job definition you provide.
executor_size	<xref:JSON> Number of core and memory to be used for executors allocated in the specified Spark pool for the job, which will be used for overriding 'executorCores' and 'executorMemory' of the spark job definition you provide. Type: string (or Expression with resultType string).
conf	<xref:JSON> Spark configuration properties, which will override the 'conf' of the spark job definition you provide.
driver_size	<xref:JSON> Number of core and memory to be used for driver allocated in the specified Spark pool for the job, which will be used for overriding 'driverCores' and 'driverMemory' of the spark job definition you provide. Type: string (or Expression with resultType string).
num_executors	<xref:JSON> Number of executors to launch for this job, which will override the 'numExecutors' of the spark job definition you provide. Type: integer (or Expression with resultType integer).
configuration_type	str or ConfigurationType The type of the spark config. Known values are: "Default", "Customized", and "Artifact".
target_spark_configuration	SparkConfigurationParametrizationReference The spark configuration of the spark job.
spark_config	dict[str, <xref:JSON>] Spark configuration property.

Share via

SynapseSparkJobDefinitionActivity Class

Constructor

Keyword-Only Parameters

Variables

Additional resources