Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,303 questions
0 answers
How to resolve New-AzResourceGroupDeployment: - The deployment 'DataBricksDeployment' failed with error(s). Showing 1 out of 1 error(s). Status Message: The definition of application 'Databricks25' cannot be null. (Code:InvalidApplicationDefi
God day Does anyone know what could be causing this error. New-AzResourceGroupDeployment: 10:06:59 - The deployment 'DataBricksDeployment' failed with error(s). Showing 1 out of 1 error(s). Status Message: The definition of application…
asked 2025-01-14T08:32:04.5133333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(102.4, 51%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETM%3C/text%3E%3C/svg%3E)
Tsholofelo Mautla
21
Reputation points
edited the question 2025-01-14T21:05:16.9666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(12.8, 76%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EV%3C/text%3E%3C/svg%3E)
VarunTha
10,775
Reputation points Microsoft Vendor
1 answer
One of the answers was accepted by the question author.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 52%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EW%3C/text%3E%3C/svg%3E)
1 answer
unity catalog issue, i build 2 databricks workspace, but can not manage accounts
I created two workspaces in Azure Databricks, but I cannot find the 'Manage Accounts' option in my account. How can I access this service?" I can not use unity catalog service , any action I can fix this issue? thank you
asked 2025-01-13T17:19:54.9666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(153.6, 79%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
莎琦 陳
0
Reputation points
edited a comment 2025-01-13T18:48:18.2233333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 65%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECB%3C/text%3E%3C/svg%3E)
Chandra Boorla
6,460
Reputation points Microsoft Vendor
1 answer
One of the answers was accepted by the question author.
How can I reliably automate granting admin consent for an Azure Databricks SCIM app using Terraform and Azure CLI without manual intervention?
I am trying to automate the creation of an Azure AD application (specifically, an Azure Databricks SCIM app) and grant admin consent for its API permissions using Terraform. The required API permissions include Application.ReadWrite.All,…
asked 2025-01-05T13:46:37.9466667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(25.6, 41%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKS%3C/text%3E%3C/svg%3E)
Kaniganti Sai Sumanth
20
Reputation points
commented 2025-01-13T15:32:20.98+00:00
Kaniganti Sai Sumanth
20
Reputation points
1 answer
Read AppServiceConsoleLogs and push to Blob Storage container/table
Hello Team, we have a webapp python Django based. We are generating custom logs for user actions using code below. Settings.py: LOGGING = { "version": 1, "disable_existing_loggers": False, "formatters": { …
asked 2024-11-20T09:31:14.9233333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(272, 35%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBM%3C/text%3E%3C/svg%3E)
BalaKrishna Mahamkali
0
Reputation points
commented 2025-01-13T08:16:50.2066667+00:00
BalaKrishna Mahamkali
0
Reputation points
1 answer
One of the answers was accepted by the question author.
Passing Boolean from Data Factory to Databricks Notebook
I have a pipeline in Azure Data Factory that includes an activity to call a Databricks notebook. I want to pass a Boolean parameter from Data Factory and use it in the notebook for conditional logic. To achieve this, I passed a string parameter with the…
asked 2025-01-10T12:23:07.9566667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 32%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES5%3C/text%3E%3C/svg%3E)
Stephen-5002
105
Reputation points
accepted 2025-01-13T08:04:21.8233333+00:00
Stephen-5002
105
Reputation points
1 answer
Create global temporary stored procedure or global temporary table
Hi, In ADF I am using self-hosted to create linked service. In SSIS, I know we can create global temporary stored proc & table. I want to know can we do similar to that in ADF too. When a pipeline executes, it needs to create global temporary stored…
asked 2025-01-12T06:59:36.4533333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(166.4, 45%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ER%3C/text%3E%3C/svg%3E)
rookie_pookie
0
Reputation points
commented 2025-01-13T07:02:44.4733333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 61%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
phemanth
12,900
Reputation points Microsoft Vendor
1 answer
How to fetch CRM Dynamic Application Audit Logs from MS Purview through Azure Databricks?
Hi, We are looking to fetch AuditLogs of Dynamics CRM from MS Purview through Azure Databricks. I know we can fetch through 'Microsoft Graph Rest API beta version' but in the page linked below it said it is not supported for Production…
asked 2025-01-12T09:56:08.6733333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(9.6, 27%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKB%3C/text%3E%3C/svg%3E)
Kalyani Bhumireddy
0
Reputation points
answered 2025-01-12T16:05:05.36+00:00
Vasil Michev
111.7K
Reputation points MVP
1 answer
Using Azure Databricks Clusters with Unity Catalog and Managed Identity Support.
We are leveraging Azure Databricks for our Data Engineering project and intend to utilize Databricks clusters that support both Unity Catalog and Managed Identity. However, according to the Databricks documentation…
asked 2025-01-02T13:10:23.9933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(12.8, 34%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHK%3C/text%3E%3C/svg%3E)
Hemanth Kumar
0
Reputation points
commented 2025-01-10T13:52:14.9133333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(201.6, 18%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGG%3C/text%3E%3C/svg%3E)
Ganesh Gurram
3,025
Reputation points Microsoft Vendor
1 answer
How to extract ProcessInstanceHistory object records using new Salesforce connector from ADF
We are trying to extract the data from Salesforce for the ProcessInstanceHistory object through the new Salesforce connector. There is a limitation for the new connector as it doesn't allow inner SOQL query for Bulk API. We have tried using REST API but…
asked 2025-01-07T12:19:33.4766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 47%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESD%3C/text%3E%3C/svg%3E)
S D, Sudevika
40
Reputation points
commented 2025-01-10T09:19:03.98+00:00
Ganesh Gurram
3,025
Reputation points Microsoft Vendor
0 answers
Lineage was extracted from unity but duplicate process created for same table In Purview
Hi Microsoft Team, I have registered and scanned databricks unity catalog in azure purview and then all tables,views metadata came into purview and that time lineage is not pulled into azure purview and now i have enabled access schema in unity catalog…
asked 2025-01-07T12:52:16.9333333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(297.6, 5%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESL%3C/text%3E%3C/svg%3E)
Sri Lakshman Velugubantla
20
Reputation points
commented 2025-01-09T20:42:22.1133333+00:00
Sri Lakshman Velugubantla
20
Reputation points
1 answer
All Azure services cost report
Hi friends, I need to develop a report that shows the cost accrued to each Azure service such as ADF, Databricks, and other services. The report should include high-level and granular costs against Azure services. I understand Azure cost management does…
asked 2025-01-08T16:35:26.3966667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(243.2, 88%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
azure_learner
525
Reputation points
answered 2025-01-08T16:46:53.88+00:00
Nandan Hegde
33,241
Reputation points MVP
1 answer
Parallel Writing of Data to ADLS Delta Storage Causes "The specified path already exists.", 409, PUT" Error
I have a Spark (Spark 3.1.2, Scala 2.12) application that reads json records from a table. The records are distributed across cluster executors. In the application, I have used foreach function to loop through the records in the table. For each record,…
asked 2023-11-06T19:54:40.9533333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 7.000000000000001%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EOA%3C/text%3E%3C/svg%3E)
Orowole, Ayebogbon-XT
0
Reputation points
answered 2025-01-08T14:00:23.99+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(64, 41%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVS%3C/text%3E%3C/svg%3E)
Vidhya Sagar Karthikeyan
396
Reputation points
1 answer
Databricks Libraries Installation SSL Error
HELP ME TO SOLVE THIS ERROR- PLEASE ELABORATE IN DETAIL... Library installation attempted on the driver node of cluster 0606-104958-63c5i7so and failed. Please refer to the following error message to fix the library or contact Databricks support. Error…
asked 2024-10-09T06:16:35.06+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(35.2, 55.00000000000001%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBS%3C/text%3E%3C/svg%3E)
Balwant Singh Chouhan
55
Reputation points
answered 2025-01-08T13:39:22.7666667+00:00
Vidhya Sagar Karthikeyan
396
Reputation points
1 answer
Query Regional Core Usage by Resource
I very frequently run in to the "Operation could not be completed as it results in exceeding approved Total Regional Cores quota" error. I understand what this means and how to increase my subscription's limit, but first I would like to…
asked 2022-06-22T18:56:58.1+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(41.6, 27%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EN%3C/text%3E%3C/svg%3E)
NPizzuti
56
Reputation points
answered 2025-01-08T13:15:38.8+00:00
Vidhya Sagar Karthikeyan
396
Reputation points
1 answer
One of the answers was accepted by the question author.
Merits and Demerits of Handling Large Data set from Azure Data bricks in Integration with Power BI
Hi, I would like to know about what are the various factors to consider when handling Larger Data set from Azure Data bricks in Integration with Power BI. The complete list of Merits and Demerits of Azure Data Bricks with Power BI. Thanks
asked 2025-01-06T06:44:56.97+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 63%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPS%3C/text%3E%3C/svg%3E)
Pradeep1 S
20
Reputation points
accepted 2025-01-08T05:17:27.0666667+00:00
Pradeep1 S
20
Reputation points
1 answer
One of the answers was accepted by the question author.
Can I take the Azure Data Engineer exam from US, while I booked it from other region?
Hi, I have purchased Azure Data Engineer from India, but had to move to US for work. Can I attempt it from US as per the time zone I booked the exam ?
asked 2025-01-07T02:28:07.4966667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(25.6, 7.000000000000001%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBD%3C/text%3E%3C/svg%3E)
Bharadwaj Dhornala
20
Reputation points
accepted 2025-01-08T01:21:06.89+00:00
Bharadwaj Dhornala
20
Reputation points
2 answers
One of the answers was accepted by the question author.
Azure Databricks Usage Cost
Hi Team, We have an Azure Databricks subscription with reservations, and multiple workspaces for different projects. We want to monitor our cost usage in the best way possible. I've explored two options: Creating a consumption dashboard in the Account…
asked 2024-12-18T16:43:33.6566667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(38.4, 15%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMT%3C/text%3E%3C/svg%3E)
Mukesh Thakor
40
Reputation points
commented 2025-01-07T18:10:42.2633333+00:00
Mukesh Thakor
40
Reputation points
1 answer
One of the answers was accepted by the question author.
Constantly Running Interactive Clusters Best Practices
Hello there, I’ve been creating an ETL/ELT Pipeline with Azure Databricks Workflows, Spark and Azure Data Lake. It should process in Near Real Time changes (A Change Data Capture process) from an Azure SQL Database. For that purpose, I will have…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(224.00000000000003, 36%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EM%3C/text%3E%3C/svg%3E)
1 answer
One of the answers was accepted by the question author.
Delta Table Partitioning - why only for tables bigger than 1 TB
Hello there, I’ve been creating an ETL/ELT Pipeline with Azure Databricks Workflows, Spark and Azure Data Lake. It should process in Near Real Time changes (A Change Data Capture process) from an Azure SQL Server Database. I have a bunch of big tables…