Connect to serverless compute
This article explains the multiple serverless offerings available on Azure Databricks. Serverless compute allows you to quickly connect to on-demand computing resources.
The articles in this section focus on serverless compute for notebooks, jobs, and Delta Live Tables. For information on serverless SQL warehouses, see What are Serverless SQL warehouses?. For information on Model Serving, see Model serving with Azure Databricks.
For information on serverless compute plane architecture, see Serverless compute plane.
What is serverless compute?
Serverless compute enhances productivity, cost efficiency, and reliability in the following ways:
- Productivity: Cloud resources are managed by Azure Databricks, reducing management overhead and providing instant compute to enhance user productivity.
- Efficiency: Serverless compute offers rapid start-up and scaling times, minimizing idle time and ensuring you only pay for the compute you use.
- Reliability: With serverless compute, capacity handling, security, patching, and upgrades are managed automatically, alleviating concerns about security policies and capacity shortages.
What types of serverless compute are available on Azure Databricks?
Azure Databricks currently offers the following types of serverless compute:
- Serverless compute for notebooks: On-demand, scalable compute used to execute SQL and Python code in notebooks.
- Serverless compute for jobs: On-demand, scalable compute used to run your Databricks jobs without configuring and deploying infrastructure.
- Serverless SQL warehouses: On-demand elastic compute used to run SQL commands on data objects in the SQL editor or interactive notebooks. You can create SQL warehouses using the UI, CLI, or REST API.
- Serverless DLT pipelines: Optimized and scalable compute for your Delta Live Tables pipeline updates.
- Mosaic AI Model Serving: Highly available and low-latency service for deploying AI models.
Enable serverless compute
To access serverless compute for notebooks, jobs, and Delta Live Tables, an account admin must enable the feature at the account level. For details on enablement, see Enable serverless compute.
Enablement for serverless SQL warehouses and Model Serving is a separate process. See Enable serverless SQL warehouses.
Serverless compute limitations
For a list of limitations, see Serverless compute limitations.
Frequently asked questions (FAQ)
- How are releases rolled out?
- How do I determine which serverless version I am running?
- How do I estimate costs for serverless?
- How do I analyze DBU usage for a specific workload?
- I haven’t enabled serverless compute for jobs and notebooks, why do I see billing records for serverless jobs?
- Does serverless compute support private repos?
- How do I install libraries for my job tasks?
- Can I connect to custom data sources?
- How does the serverless compute plane networking work?
- Can I configure serverless compute for jobs with Databricks Asset Bundles?
How are releases rolled out?
Serverless compute is a versionless product, which means that Databricks automatically upgrades the serverless compute runtime to support enhancements and upgrades to the platform. All users get the same updates, rolled out over a short period of time.
How do I determine which serverless version I am running?
Serverless workloads always run on the latest runtime version. See Release notes for the most recent version.
How do I estimate costs for serverless?
Databricks recommends running and benchmarking a representative or specific workload and then analyzing the billing system table. See Billable usage system table reference.
How do I analyze DBU usage for a specific workload?
To see the cost for a specific workload, query the system.billing.usage
system table. See Monitor the cost of serverless compute for sample queries and to download our cost observability dashboard.
I haven’t enabled serverless compute for jobs and notebooks, why do I see billing records for serverless jobs?
Lakehouse Monitoring and predictive optimization are also billed under the serverless jobs SKU.
Serverless compute does not have to be enabled to use these two features.
Does serverless compute support private repos?
Repositories can be private or require authentication. For security reasons, a pre-signed URL is required when accessing authenticated repositories.
How do I install libraries for my job tasks?
Databricks recommends using environments to install and manage libraries for your jobs. See Configure environments and dependencies for non-notebook tasks.
Can I connect to custom data sources?
No, only the sources using Lakehouse Federation are supported. See Supported data sources.
How does the serverless compute plane networking work?
Serverless compute resources run in the serverless compute plane, which is managed by Azure Databricks. For more details on the network and architecture, see Serverless compute plane networking.
Can I configure serverless compute for jobs with Databricks Asset Bundles?
Yes, Databricks Asset Bundles can be used to configure jobs that use serverless compute. See Configure a job that uses serverless compute.