How to fix memory has been exhausted error in Azure Machine Learning Studio in Convert to Indicator Values module

Question

When I am running a pipeline that I created in Azure Machine Learning Studio, it is for some reason giving me the following error:

Execution failed. User process '/azureml-envs/azureml_862fbd3b8df44d2c582aa46cf5a23700/bin/python' exited with status code 1. Please check log file 'user_logs/std_log.txt' for error details. Error:   File "/azureml-envs/azureml_862fbd3b8df44d2c582aa46cf5a23700/lib/python3.8/site-packages/azureml/studio/modulehost/module_invoker.py", line 7, in      execute(sys.argv)   File "/azureml-envs/azureml_862fbd3b8df44d2c582aa46cf5a23700/lib/python3.8/site-packages/azureml/studio/modulehost/module_host_executor.py", line 41, in execute     return execute_with_cli(original_args)   File "/azureml-envs/azureml_862fbd3b8df44d2c582aa46cf5a23700/lib/python3.8/site-packages/azureml/studio/core/logger.py", line 214, in wrapper     ret = func(*args, **kwargs)   File "/azureml-envs/azureml_862fbd3b8df44d2c582aa46cf5a23700/lib/python3.8/site-packages/azureml/studio/modulehost/module_host_executor.py", line 52, in execute_with_cli     do_execute_with_env(parser, FolderRuntimeEnv())   File "/azureml-envs/azureml_862fbd3b8df44d2c582aa46cf5a23700/lib/python3.8/site-packages/azureml/studio/modulehost/module_host_executor.py", line 63, in do_execute_with_env     ModuleReflector(parser.module_entry, env).exec(   File "/azureml-envs/azureml_862fbd3b8df44d2c582aa46cf5a23700/lib/python3.8/site-packages/azureml/studio/modulehost/module_reflector.py", line 397, in exec     self._handle_exception(bex)   File "/azureml-envs/azureml_862fbd3b8df44d2c582aa46cf5a23700/lib/python3.8/site-packages/azureml/studio/modulehost/module_reflector.py", line 471, in _handle_exception     raise exception   File "/azureml-envs/azureml_862fbd3b8df44d2c582aa46cf5a23700/lib/python3.8/site-packages/azureml/studio/modulehost/module_reflector.py", line 394, in exec     ErrorMapping.rethrow(e=mem_error,   File "/azureml-envs/azureml_862fbd3b8df44d2c582aa46cf5a23700/lib/python3.8/site-packages/azureml/studio/common/error.py", line 842, in rethrow     raise err from e azureml.studio.common.error.ModuleOutOfMemoryError: Memory has been exhausted, unable to complete running of module.

I am not sure why am I getting this error.

My dataset has 2 million+ entries with 12 columns out of which 5 columns are being dropped, and I am converting 3 columns as categorical data using Convert to Indicator Values module.

Please help. Thanks in advance!!

Answer

Hello Lakshya Jain,

Welcome to the Microsoft Q&A and thank you for posting your questions here.

I understand that you would like to fix memory has been exhausted error that you encountered in Azure Machine Learning Studio in Convert to Indicator Values module.

Each of the steps below are independent guideline to resolve the issue:

Review detailed logs (user_logs/std_log.txt) for insights into memory usage and specific bottlenecks.
Optimize datasets locally before uploading to Azure ML Studio to reduce resource usage. - https://pandas.pydata.org/docs and https://scikit-learn.org/stable/documentation.html
For sustained operations, consider higher-tier instances or GPU-enabled compute for demanding tasks. - https://zcusa.951200.xyz/en-us/azure/machine-learning
Split datasets into manageable chunks to prevent memory exhaustion and handle intermediate outputs efficiently.
Use Azure ML's Data Prep SDK to manipulate large datasets efficiently and avoid memory-related issues in the pipeline. - https://zcusa.951200.xyz/en-us/azure/machine-learning/how-to-data-prep-sdk

I hope this is helpful! Do not hesitate to let me know if you have any other questions.

Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful.

Share via

How to fix memory has been exhausted error in Azure Machine Learning Studio in Convert to Indicator Values module

1 answer

Your answer