Databricks Runtime 10.0 per ML (EoS)
Nota
Il supporto per questa versione di Databricks Runtime è terminato. Per la data di fine del supporto, vedere Cronologia di fine del supporto. Per tutte le versioni supportate di Databricks Runtime, vedere Versioni e compatibilità delle note sulla versione di Databricks Runtime.
Databricks Runtime 10.0 per Machine Learning è un ambiente pronto all’uso ottimizzato per l'esecuzione di processi di apprendimento automatico e data science basato su Databricks Runtime 10.0 (EoS). Databricks Runtime ML contiene molte di queste popolari librerie per l’apprendimento automatico, tra cui TensorFlow, PyTorch e XGBoost. È inoltre supportato il training distribuito con Horovod.
Per altre informazioni, incluse le istruzioni per la creazione di un cluster di Machine Learning di Databricks Runtime, vedere Intelligenza artificiale e Machine Learning in Databricks.
Miglioramenti e nuove funzionalità
Databricks Runtime 10.0 ML è basato su Databricks Runtime 10.0. Per informazioni sulle novità di Databricks Runtime 10.0, tra cui Apache Spark MLlib e SparkR, vedere le note sulla versione di Databricks Runtime 10.0 (EoS).
Previsione di serie temporali con AutoML
AutoML supporta ora la previsione delle serie temporali. Per altre informazioni, vedere la documentazione di AutoML.
Modifiche principali all'ambiente Python di Databricks Runtime ML
Pacchetti Python aggiunti
- databricks-automl-runtime 0.1.0
- imbalanced-learn 0.8.0
- transformers 4.9.2
Ambiente di sistema
È stata aggiornata la versione inclusa di RStudio Server Open Source alla versione 1.4.
L'ambiente di sistema in Databricks Runtime 10.0 ML differisce da Databricks Runtime 10.0 come indicato di seguito:
- DBUtils: Databricks Runtime ML non include l'utilità libreria (dbutils.library) (legacy).
Usare
%pipinvece i comandi. Vedere Librerie Python con ambito notebook. - Per i cluster GPU, Databricks Runtime ML include le librerie GPU NVIDIA seguenti:
- CUDA 11.0
- CUDNN 8.0.5.39
- NCCL 2.10.3
- TensorRT 7.2.2
Librerie
Le sezioni seguenti elencano le librerie incluse in Databricks Runtime 10.0 ML che differiscono da quelle incluse in Databricks Runtime 10.0.
Contenuto della sezione:
Librerie di livello superiore
Databricks Runtime 10.0 ML include le librerie di livello superiore seguenti:
- GraphFrames
- Horovod e HorovodRunner
- MLflow
- PyTorch
- spark-tensorflow-connector
- TensorFlow
- TensorBoard
Librerie Python
Databricks Runtime 10.0 ML usa Virtualenv per la gestione dei pacchetti Python e include molti dei pacchetti ML più diffusi.
Oltre ai pacchetti specificati nelle sezioni seguenti, Databricks Runtime 10.0 ML include anche i pacchetti seguenti:
- hyperopt 0.2.5.db2
- sparkdl 2.2.0_db3
- feature_store 0.3.4
- automl 1.3.1
Librerie Python nei cluster CPU
| Library | Versione | Library | Versione | Library | Versione |
|---|---|---|---|---|---|
| absl-py | 0.11.0 | Antergos Linux | 2015.10 (ISO-Rolling) | appdirs | 1.4.4 |
| argon2-cffi | 20.1.0 | astor | 0.8.1 | astunparse | 1.6.3 |
| async-generator | 1.10 | attrs | 20.3.0 | backcall | 0.2.0 |
| bcrypt | 3.2.0 | bleach | 3.3.0 | blis | 0.7.4 |
| boto3 | 1.16.7 | botocore | 1.19.7 | Bottleneck | 1.3.2 |
| cachetools | 4.2.2 | catalogue | 2.0.6 | certifi | 2020.12.5 |
| cffi | 1.14.5 | chardet | 4.0.0 | Clang | 5.0 |
| click | 7.1.2 | cloudpickle | 1.6.0 | cmdstanpy | 0.9.68 |
| configparser | 5.0.1 | convertdate | 2.3.2 | cryptography | 3.4.7 |
| cycler | 0.10.0 | cymem | 2.0.5 | Cython | 0.29.23 |
| databricks-automl-runtime | 0.1.0 | databricks-cli | 0.14.3 | dbus-python | 1.2.16 |
| decorator | 5.0.6 | defusedxml | 0.7.1 | dill | 0.3.2 |
| diskcache | 5.2.1 | distlib | 0.3.2 | distro-info | 0.23ubuntu1 |
| entrypoints | 0.3 | ephem | 4.0.0.2 | facet-overview | 1.0.0 |
| filelock | 3.0.12 | Flask | 1.1.2 | flatbuffers | 1.12 |
| fsspec | 0.9.0 | future | 0.18.2 | gast | 0.4.0 |
| gitdb | 4.0.7 | gitpython | 3.1.12 | google-auth | 1.22.1 |
| google-auth-oauthlib | 0.4.2 | google-pasta | 0.2.0 | grpcio | 1.39.0 |
| gunicorn | 20.0.4 | h5py | 3.1.0 | hijri-converter | 2.2.1 |
| holidays | 0.11.2 | horovod | 0.22.1 | htmlmin | 0.1.12 |
| huggingface-hub | 0.0.12 | idna | 2.10 | ImageHash | 4.2.1 |
| imbalanced-learn | 0.8.0 | importlib-metadata | 3.10.0 | ipykernel | 5.3.4 |
| ipython | 7.22.0 | ipython-genutils | 0.2.0 | ipywidgets | 7.6.3 |
| isodate | 0.6.0 | itsdangerous | 1.1.0 | jedi | 0.17.2 |
| Jinja2 | 2.11.3 | jmespath | 0.10.0 | joblib | 1.0.1 |
| joblibspark | 0.3.0 | jsonschema | 3.2.0 | jupyter-client | 6.1.12 |
| jupyter-core | 4.7.1 | jupyterlab-pygments | 0.1.2 | jupyterlab-widgets | 1.0.0 |
| keras | 2.6.0 | keras-preprocessing | 1.1.2 | kiwisolver | 1.3.1 |
| koalas | 1.8.1 | korean-lunar-calendar | 0.2.1 | lightgbm | 3.1.1 |
| llvmlite | 0.37.0 | LunarCalendar | 0.0.9 | Mako | 1.1.3 |
| Markdown | 3.3.3 | MarkupSafe | 1.1.1 | matplotlib | 3.4.2 |
| missingno | 0.5.0 | mistune | 0.8.4 | mleap | 0.17.0 |
| mlflow-skinny | 1.20.2 | multimethod | 1.4 | mormurhash | 1.0.5 |
| nbclient | 0.5.3 | nbconvert | 6.0.7 | nbformat | 5.1.3 |
| nest-asyncio | 1.5.1 | networkx | 2.5 | nltk | 3.6.1 |
| notebook | 6.3.0 | numba | 0.54.0 | numpy | 1.19.2 |
| oauthlib | 3.1.0 | opt-einsum | 3.3.0 | packaging | 20.9 |
| pandas | 1.2.4 | pandas-profiling | 3.0.0 | pandocfilters | 1.4.3 |
| paramiko | 2.7.2 | parso | 0.7.0 | patia | 0.6.0 |
| patsy | 0.5.1 | petastorm | 0.11.2 | pexpect | 4.8.0 |
| phik | 0.12.0 | pickleshare | 0.7.5 | Pillow | 8.2.0 |
| pip | 21.0.1 | plotly | 5.1.0 | preshed | 3.0.5 |
| prometheus-client | 0.10.1 | prompt-toolkit | 3.0.17 | prophet | 1.0.1 |
| protobuf | 3.17.2 | psutil | 5.8.0 | Psycopg2 | 2.8.5 |
| ptyprocess | 0.7.0 | pyarrow | 4.0.0 | pyasn1 | 0.4.8 |
| pyasn1-modules | 0.2.8 | pycparser | 2.20 | pydantic | 1.8.2 |
| Pygments | 2.8.1 | pygobject | 3.36.0 | PyMeeus | 0.5.11 |
| PyNaCl | 1.3.0 | pyodbc | 4.0.30 | pyparsing | 2.4.7 |
| pirsistente | 0.17.3 | pystan | 2.19.1.1 | API Python | 2.0.0+ubuntu0.20.4.6 |
| python-dateutil | 2.8.1 | python-editor | 1.0.4 | pytz | 2020.5 |
| PyWavelets | 1.1.1 | PyYAML | 5.4.1 | pyzmq | 20.0.0 |
| regex | 2021.4.4 | requests | 2.25.1 | requests-oauthlib | 1.3.0 |
| requests-unixsocket | 0.2.0 | rsa | 4.7.2 | s3transfer | 0.3.7 |
| sacrimoses | 0.0.45 | scikit-learn | 0.24.1 | scipy | 1.6.2 |
| seaborn | 0.11.1 | Send2Trash | 1.5.0 | setuptools | 52.0.0 |
| setuptools-git | 1.2 | shap | 0.39.0 | simplejson | 3.17.2 |
| six | 1.15.0 | slicer | 0.0.7 | Ritaglio intelligente | 5.2.0 |
| smmap | 3.0.5 | spacy | 3.1.2 | spacy-legacy | 3.0.8 |
| spark-tensorflow-distributor | 1.0.0 | sqlparse | 0.4.1 | srsly | 2.4.1 |
| ssh-import-id | 5.10 | statsmodels | 0.12.2 | tabulate | 0.8.7 |
| tangled-up-in-unicode | 0.1.0 | tenacity | 6.2.0 | tensorboard | 2.6.0 |
| tensorboard-data-server | 0.6.1 | tensorboard-plugin-wit | 1.8.0 | tensorflow-cpu | 2.6.0 |
| tensorflow-estimator | 2.6.0 | termcolor | 1.1.0 | terminado | 0.9.4 |
| testpath | 0.4.4 | thinc | 8.0.8 | threadpoolctl | 2.1.0 |
| tokenizers | (0.10.3) | torch | 1.9.0+cpu | torchvision | 0.10.0+cpu |
| tornado | 6.1 | tqdm | 4.59.0 | traitlets | 5.0.5 |
| transformers | 4.9.2 | typer | 0.3.2 | typing-extensions | 3.7.4.3 |
| ujson | 4.0.2 | aggiornamenti automatici | 0.1 | urllib3 | 1.25.11 |
| virtualenv | 20.4.1 | visions | 0.7.1 | wasabi | 0.8.2 |
| wcwidth | 0.2.5 | webencodings | 0.5.1 | websocket-client | 0.57.0 |
| Werkzeug | 1.0.1 | wheel | 0.36.2 | widgetsnbextension | 3.5.1 |
| wrapt | 1.12.1 | xgboost | 1.4.2 | zipp | 3.4.1 |
Librerie Python nei cluster GPU
| Library | Versione | Library | Versione | Library | Versione |
|---|---|---|---|---|---|
| absl-py | 0.11.0 | Antergos Linux | 2015.10 (ISO-Rolling) | appdirs | 1.4.4 |
| argon2-cffi | 20.1.0 | astor | 0.8.1 | astunparse | 1.6.3 |
| async-generator | 1.10 | attrs | 20.3.0 | backcall | 0.2.0 |
| bcrypt | 3.2.0 | bleach | 3.3.0 | blis | 0.7.4 |
| boto3 | 1.16.7 | botocore | 1.19.7 | Bottleneck | 1.3.2 |
| cachetools | 4.2.2 | catalogue | 2.0.6 | certifi | 2020.12.5 |
| cffi | 1.14.5 | chardet | 4.0.0 | Clang | 5.0 |
| click | 7.1.2 | cloudpickle | 1.6.0 | cmdstanpy | 0.9.68 |
| configparser | 5.0.1 | convertdate | 2.3.2 | cryptography | 3.4.7 |
| cycler | 0.10.0 | cymem | 2.0.5 | Cython | 0.29.23 |
| databricks-automl-runtime | 0.1.0 | databricks-cli | 0.14.3 | dbus-python | 1.2.16 |
| decorator | 5.0.6 | defusedxml | 0.7.1 | dill | 0.3.2 |
| diskcache | 5.2.1 | distlib | 0.3.2 | distro-info | 0.23ubuntu1 |
| entrypoints | 0.3 | ephem | 4.0.0.2 | facet-overview | 1.0.0 |
| filelock | 3.0.12 | Flask | 1.1.2 | flatbuffers | 1.12 |
| fsspec | 0.9.0 | future | 0.18.2 | gast | 0.4.0 |
| gitdb | 4.0.7 | gitpython | 3.1.12 | google-auth | 1.22.1 |
| google-auth-oauthlib | 0.4.2 | google-pasta | 0.2.0 | grpcio | 1.39.0 |
| gunicorn | 20.0.4 | h5py | 3.1.0 | hijri-converter | 2.2.1 |
| holidays | 0.11.2 | horovod | 0.22.1 | htmlmin | 0.1.12 |
| huggingface-hub | 0.0.12 | idna | 2.10 | ImageHash | 4.2.1 |
| imbalanced-learn | 0.8.0 | importlib-metadata | 3.10.0 | ipykernel | 5.3.4 |
| ipython | 7.22.0 | ipython-genutils | 0.2.0 | ipywidgets | 7.6.3 |
| isodate | 0.6.0 | itsdangerous | 1.1.0 | jedi | 0.17.2 |
| Jinja2 | 2.11.3 | jmespath | 0.10.0 | joblib | 1.0.1 |
| joblibspark | 0.3.0 | jsonschema | 3.2.0 | jupyter-client | 6.1.12 |
| jupyter-core | 4.7.1 | jupyterlab-pygments | 0.1.2 | jupyterlab-widgets | 1.0.0 |
| keras | 2.6.0 | keras-preprocessing | 1.1.2 | kiwisolver | 1.3.1 |
| koalas | 1.8.1 | korean-lunar-calendar | 0.2.1 | lightgbm | 3.1.1 |
| llvmlite | 0.37.0 | LunarCalendar | 0.0.9 | Mako | 1.1.3 |
| Markdown | 3.3.3 | MarkupSafe | 1.1.1 | matplotlib | 3.4.2 |
| missingno | 0.5.0 | mistune | 0.8.4 | mleap | 0.17.0 |
| mlflow-skinny | 1.20.2 | multimethod | 1.4 | mormurhash | 1.0.5 |
| nbclient | 0.5.3 | nbconvert | 6.0.7 | nbformat | 5.1.3 |
| nest-asyncio | 1.5.1 | networkx | 2.5 | nltk | 3.6.1 |
| notebook | 6.3.0 | numba | 0.54.0 | numpy | 1.19.2 |
| oauthlib | 3.1.0 | opt-einsum | 3.3.0 | packaging | 20.9 |
| pandas | 1.2.4 | pandas-profiling | 3.0.0 | pandocfilters | 1.4.3 |
| paramiko | 2.7.2 | parso | 0.7.0 | patia | 0.6.0 |
| patsy | 0.5.1 | petastorm | 0.11.2 | pexpect | 4.8.0 |
| phik | 0.12.0 | pickleshare | 0.7.5 | Pillow | 8.2.0 |
| pip | 21.0.1 | plotly | 5.1.0 | preshed | 3.0.5 |
| prompt-toolkit | 3.0.17 | prophet | 1.0.1 | protobuf | 3.17.2 |
| psutil | 5.8.0 | Psycopg2 | 2.8.5 | ptyprocess | 0.7.0 |
| pyarrow | 4.0.0 | pyasn1 | 0.4.8 | pyasn1-modules | 0.2.8 |
| pycparser | 2.20 | pydantic | 1.8.2 | Pygments | 2.8.1 |
| pygobject | 3.36.0 | PyMeeus | 0.5.11 | PyNaCl | 1.3.0 |
| pyodbc | 4.0.30 | pyparsing | 2.4.7 | pirsistente | 0.17.3 |
| pystan | 2.19.1.1 | API Python | 2.0.0+ubuntu0.20.4.6 | python-dateutil | 2.8.1 |
| python-editor | 1.0.4 | pytz | 2020.5 | PyWavelets | 1.1.1 |
| PyYAML | 5.4.1 | pyzmq | 20.0.0 | regex | 2021.4.4 |
| requests | 2.25.1 | requests-oauthlib | 1.3.0 | requests-unixsocket | 0.2.0 |
| rsa | 4.7.2 | s3transfer | 0.3.7 | sacrimoses | 0.0.45 |
| scikit-learn | 0.24.1 | scipy | 1.6.2 | seaborn | 0.11.1 |
| Send2Trash | 1.5.0 | setuptools | 52.0.0 | setuptools-git | 1.2 |
| shap | 0.39.0 | simplejson | 3.17.2 | six | 1.15.0 |
| slicer | 0.0.7 | Ritaglio intelligente | 5.2.0 | smmap | 3.0.5 |
| spacy | 3.1.2 | spacy-legacy | 3.0.8 | spark-tensorflow-distributor | 1.0.0 |
| sqlparse | 0.4.1 | srsly | 2.4.1 | ssh-import-id | 5.10 |
| statsmodels | 0.12.2 | tabulate | 0.8.7 | tangled-up-in-unicode | 0.1.0 |
| tenacity | 6.2.0 | tensorboard | 2.6.0 | tensorboard-data-server | 0.6.1 |
| tensorboard-plugin-wit | 1.8.0 | tensorflow | 2.6.0 | tensorflow-estimator | 2.6.0 |
| termcolor | 1.1.0 | terminado | 0.9.4 | testpath | 0.4.4 |
| thinc | 8.0.8 | threadpoolctl | 2.1.0 | tokenizers | (0.10.3) |
| torch | 1.9.0+cu111 | torchvision | 0.10.0+cu111 | tornado | 6.1 |
| tqdm | 4.59.0 | traitlets | 5.0.5 | transformers | 4.9.2 |
| typer | 0.3.2 | typing-extensions | 3.7.4.3 | ujson | 4.0.2 |
| aggiornamenti automatici | 0.1 | urllib3 | 1.25.11 | virtualenv | 20.4.1 |
| visions | 0.7.1 | wasabi | 0.8.2 | wcwidth | 0.2.5 |
| webencodings | 0.5.1 | websocket-client | 0.57.0 | Werkzeug | 1.0.1 |
| wheel | 0.36.2 | widgetsnbextension | 3.5.1 | wrapt | 1.12.1 |
| xgboost | 1.4.2 | zipp | 3.4.1 |
Pacchetti Spark contenenti moduli Python
| Pacchetti Spark | Modulo Python | Versione |
|---|---|---|
| GraphFrames | graphframes | 0.8.1-db6-spark3.2 |
Librerie R
Le librerie R sono identiche alle librerie R in Databricks Runtime 10.0.
Librerie Java e Scala (cluster Scala 2.12)
Oltre alle librerie Java e Scala in Databricks Runtime 10.0, Databricks Runtime 10.0 ML contiene i file JAR seguenti:
Cluster CPU
| ID gruppo | ID artefatto | Versione |
|---|---|---|
| com.typesafe.akka | akka-actor_2.12 | 2.5.23 |
| ml.combust.mleap | mleap-databricks-runtime_2.12 | 0.17.0-4882dc3 |
| ml.dmlc | xgboost4j-spark_2.12 | 1.4.1 |
| ml.dmlc | xgboost4j_2.12 | 1.4.1 |
| org.graphframes | graphframes_2.12 | 0.8.1-db6-spark3.2 |
| org.mlflow | mlflow-client | 1.20.2 |
| org.mlflow | mlflow-spark | 1.20.2 |
| org.scala-lang.modules | scala-java8-compat_2.12 | 0.8.0 |
| org.tensorflow | spark-tensorflow-connector_2.12 | 1.15.0 |
Cluster GPU
| ID gruppo | ID artefatto | Versione |
|---|---|---|
| com.typesafe.akka | akka-actor_2.12 | 2.5.23 |
| ml.combust.mleap | mleap-databricks-runtime_2.12 | 0.17.0-4882dc3 |
| ml.dmlc | xgboost4j-gpu_2.12 | 1.4.1 |
| ml.dmlc | xgboost4j-spark-gpu_2.12 | 1.4.1 |
| org.graphframes | graphframes_2.12 | 0.8.1-db6-spark3.2 |
| org.mlflow | mlflow-client | 1.20.2 |
| org.mlflow | mlflow-spark | 1.20.2 |
| org.scala-lang.modules | scala-java8-compat_2.12 | 0.8.0 |
| org.tensorflow | spark-tensorflow-connector_2.12 | 1.15.0 |