Share via

SparkComponent Class

  • Reference

Spark component version, used to define a Spark Component or Job.

Inheritance
azure.ai.ml.entities._component.component.Component
SparkComponent
azure.ai.ml.entities._job.parameterized_spark.ParameterizedSpark
SparkComponent
azure.ai.ml.entities._job.spark_job_entry_mixin.SparkJobEntryMixin
SparkComponent
azure.ai.ml.entities._component.code.ComponentCodeMixin
SparkComponent

Constructor

SparkComponent(*, code: PathLike | str | None = '.', entry: Dict[str, str] | SparkJobEntry | None = None, py_files: List[str] | None = None, jars: List[str] | None = None, files: List[str] | None = None, archives: List[str] | None = None, driver_cores: int | str | None = None, driver_memory: str | None = None, executor_cores: int | str | None = None, executor_memory: str | None = None, executor_instances: int | str | None = None, dynamic_allocation_enabled: bool | str | None = None, dynamic_allocation_min_executors: int | str | None = None, dynamic_allocation_max_executors: int | str | None = None, conf: Dict[str, str] | None = None, environment: Environment | str | None = None, inputs: Dict | None = None, outputs: Dict | None = None, args: str | None = None, **kwargs: Any)

Keyword-Only Parameters

Name Description
code

The source code to run the job. Can be a local path or "http:", "https:", or "azureml:" url pointing to a remote location. Defaults to ".", indicating the current directory.

Default value: .
entry
Optional[Union[dict[str, str], SparkJobEntry]]

The file or class entry point.
py_files
Optional[List[str]]

The list of .zip, .egg or .py files to place on the PYTHONPATH for Python apps. Defaults to None.
jars
Optional[List[str]]

The list of .JAR files to include on the driver and executor classpaths. Defaults to None.
files
Optional[List[str]]

The list of files to be placed in the working directory of each executor. Defaults to None.
archives
Optional[List[str]]

The list of archives to be extracted into the working directory of each executor. Defaults to None.
driver_cores
Optional[int]

The number of cores to use for the driver process, only in cluster mode.
driver_memory
Optional[str]

The amount of memory to use for the driver process, formatted as strings with a size unit suffix ("k", "m", "g" or "t") (e.g. "512m", "2g").
executor_cores
Optional[int]

The number of cores to use on each executor.
executor_memory
Optional[str]

The amount of memory to use per executor process, formatted as strings with a size unit suffix ("k", "m", "g" or "t") (e.g. "512m", "2g").
executor_instances
Optional[int]

The initial number of executors.
dynamic_allocation_enabled
Optional[bool]

Whether to use dynamic resource allocation, which scales the number of executors registered with this application up and down based on the workload. Defaults to False.
dynamic_allocation_min_executors
Optional[int]

The lower bound for the number of executors if dynamic allocation is enabled.
dynamic_allocation_max_executors
Optional[int]

The upper bound for the number of executors if dynamic allocation is enabled.
conf
Optional[dict[str, str]]

A dictionary with pre-defined Spark configurations key and values. Defaults to None.
environment
Optional[Union[str, Environment]]

The Azure ML environment to run the job in.
inputs
Optional[dict[str, Union[ <xref:azure.ai.ml.entities._job.pipeline._io.NodeOutput>, Input, str, bool, int, float, <xref:Enum>, ]]]

A mapping of input names to input data sources used in the job. Defaults to None.
outputs
Optional[dict[str, Union[str, Output]]]

A mapping of output names to output data sources used in the job. Defaults to None.
args
Optional[str]

The arguments for the job. Defaults to None.

Examples

Creating SparkComponent.


   from azure.ai.ml.entities import SparkComponent

   component = SparkComponent(
       name="add_greeting_column_spark_component",
       display_name="Aml Spark add greeting column test module",
       description="Aml Spark add greeting column test module",
       version="1",
       inputs={
           "file_input": {"type": "uri_file", "mode": "direct"},
       },
       driver_cores=2,
       driver_memory="1g",
       executor_cores=1,
       executor_memory="1g",
       executor_instances=1,
       code="./src",
       entry={"file": "add_greeting_column.py"},
       py_files=["utils.zip"],
       files=["my_files.txt"],
       args="--file_input ${{inputs.file_input}}",
       base_path="./sdk/ml/azure-ai-ml/tests/test_configs/dsl_pipeline/spark_job_in_pipeline",
   )

Methods

dump

Dump the component content into a file in yaml format.

dump

Dump the component content into a file in yaml format.

dump(dest: str | PathLike | IO, **kwargs: Any) -> None

Parameters

Name Description
dest
Required
Union[<xref:PathLike>, str, IO[AnyStr]]

The destination to receive this component's content. Must be either a path to a local file, or an already-open file stream. If dest is a file path, a new file will be created, and an exception is raised if the file exists. If dest is an open file, the file will be written to directly, and an exception will be raised if the file is not writable.

Attributes

base_path

The base path of the resource.

Returns

Type Description
str

The base path of the resource.

creation_context

The creation context of the resource.

Returns

Type Description
Optional[SystemData]

The creation metadata for the resource.

display_name

Display name of the component.

Returns

Type Description
str

Display name of the component.

entry

environment

The Azure ML environment to run the Spark component or job in.

Returns

Type Description
Optional[Union[str, Environment]]

The Azure ML environment to run the Spark component or job in.

id

The resource ID.

Returns

Type Description
Optional[str]

The global ID of the resource, an Azure Resource Manager (ARM) ID.

inputs

Inputs of the component.

Returns

Type Description
dict

Inputs of the component.

is_deterministic

Whether the component is deterministic.

Returns

Type Description
bool

Whether the component is deterministic

outputs

Outputs of the component.

Returns

Type Description
dict

Outputs of the component.

type

Type of the component, default is 'command'.

Returns

Type Description
str

Type of the component.

version

Version of the component.

Returns

Type Description
str

Version of the component.

CODE_ID_RE_PATTERN 

CODE_ID_RE_PATTERN = re.compile('\\/subscriptions\\/(?P<subscription>[\\w,-]+)\\/resourceGroups\\/(?P<resource_group>[\\w,-]+)\\/providers\\/Microsoft\\.MachineLearningServices\\/workspaces\\/(?P<workspace>[\\w,-]+)\\/codes\\/(?P<co)