Mlflow and Spark

Bakhruz Dzhafarov 60

Hi, I encountered the following problem when I tried to use a model for spark inference (via mlflow.pyfunc.spark_udf) that I had previously trained in pandas and saved in mlflow.

I saved a model via

from mlflow.tracking import MlflowClient
from azureml.core import Workspace

# Connect to your Azure ML workspace
ws = Workspace.from_config()  # Make sure you have a config.json file

# Set the tracking URI to Azure ML workspace
mlflow.set_tracking_uri(ws.get_mlflow_tracking_uri())

client = MlflowClient()

# Start an MLflow run
experiment_name = "CatBoost_Experiment"
experiment_id = client.create_experiment(experiment_name)
run = client.create_run(experiment_id)

# Log the CatBoost model using the client
mlflow.catboost.log_model(model, "catboost_model_20", registered_model_name= "catboost_model")

and read

model_uri = "models:/catboost_model/latest"  # Use the latest version of the registered model

# Load the model as a PySpark UDF
mlflow.pyfunc.get_model_dependencies(model_uri)
loaded_model_udf = mlflow.pyfunc.spark_udf(spark, model_uri, env_manager="conda")

logs in the attachment stderr.txt

  File "/home/trusted-service-user/cluster-env/env/lib/python3.10/site-packages/mlflow/pyfunc/__init__.py", line 1069, in udf
    pyfunc_backend.prepare_env(
  File "/home/trusted-service-user/cluster-env/env/lib/python3.10/site-packages/mlflow/pyfunc/backend.py", line 89, in prepare_env
    conda_env_path = os.path.join(local_path, self._config[ENV])
  File "/home/trusted-service-user/cluster-env/env/lib/python3.10/posixpath.py", line 90, in join
    genericpath._check_arg_types('join', a, *p)
  File "/home/trusted-service-user/cluster-env/env/lib/python3.10/genericpath.py", line 152, in _check_arg_types
    raise TypeError(f'{funcname}() argument must be str, bytes, or '
TypeError: join() argument must be str, bytes, or os.PathLike object, not 'dict'

I was able to find a relatively similar error

https://community.databricks.com/t5/machine-learning/logging-model-to-mlflow-using-feature-store-api-getting/td-p/7890

dupammi 8,540 Reputation points Microsoft Vendor

2024-06-13T09:59:50.3066667+00:00
Hi @Bakhruz Dzhafarov

The error message suggests that the os.path.join() function is expecting a string, bytes, or os.PathLike object, but it received a dictionary instead. Specifically, the error seems to be coming from the mlflow.pyfunc.spark_udf() function.

To fix this error, you may want to check the model_uri object that you are passing to mlflow.pyfunc.spark_udf(). It's possible that there is a dictionary in this object that is causing the issue. You may need to modify the model_uri object to ensure that it only contains strings, bytes, or os.PathLike objects.

Additionally, you can refer to the following Azure documentation links for more information on using MLflow with Spark:

https://docs.microsoft.com/en-us/azure/machine-learning/how-to-deploy-mlflow-model-spark-jobs

https://docs.microsoft.com/en-us/azure/databricks/applications/mlflow/mlflow-tracking-azure-databricks
Bakhruz Dzhafarov 60 Reputation points

2024-06-14T13:52:41.0633333+00:00
Hi, thank you for the answer.

The second link returns a 404 error.

I understand the text of the error message. However, I'm confused because the model_uri in my code is a string (two lines above mlflow.pyfunc.spark_udf).
Rivera, Kevin 0 Reputation points

2024-12-23T06:01:22.2066667+00:00

@Bakhruz Dzhafarov Were you able to solve it? Or at least point to what is causing the issue? I'm having similar issues but nobody else has had a similar problem to this before in my research (aside from the link you posted).

Share via

Mlflow and Spark

Your answer