Azure Synapse pipeline
Hi, I have a question regarding Synapse. I want to create a pipeline that can delta update the target table “quotes_target” from source table “quotes_source” with filtered quotes data. I have a list of quotes id of the updated quotes after I extracted it…
Updating an existing ADF pipeline - ( Importing new code )
We use different adf instances for dev and production and I was wondering how to update an existing pipeline in prod instance. The updated pipeline sitting in the dev instance and if we try to import again it creates a duplicate pipelines and datasets.…
I can't get diagnostics logging working on a dedicated sql pool
I want to see the exec requests and waits on a dedicated pool. So I created a log analytics workspace in the same workspace as the sql pool and I went to the dedicated sql pool and in the Diagnostics Settings option I created a new one for the logs exec…
Spark Diagnostic Emitter Not Working with Event Hub
I am following this documentation to send logs and metrics for Spark applications running on Synapse Spark pools to Event Hub using Azure Key Vault (AKV). I am using the configuration below and have added the Service Principal secret to AKV. I also tried…
Sometimes, none of my sql query terminate (even the smallest ones)
I'm facing occasional hanging T-SQL requests on my database since this morning (01/15/2025), even those that usually complete in less than a second. The solution I found is to kill the running requests in T-SQL via their session ID and wait one or two…
Best Practices for Ensuring Data Security with Microsoft Synapse SAP Connector and Azure Linked Services
My client asked for research on how SAP teams can protect data when using Azure Linked Services. There's particular concern regarding the lack of understanding of how the technology works. The setup involves using the Microsoft Synapse SAP connector to…
How to read parquet file to update target table?
Hi, I have a question regarding Synapse. I want to create a pipeline that can delta update the target table “quotes_target” from source table “quotes_source” with filtered quotes data. I have a list of quotes id of the updated quotes after I extracted it…
ADF/ Synapse Activity times out after 12 hours
Hello, I have a Notebook activity in one of my pipelines that is expected to run longer than 12 hours. Below we configured the timeout to the max value, 7 days. But the activity would still timeout after 12 hours. Is there a way for the activity…
Azure SQL alternative
Hi, Currently, we are using Azure SQL Single database for a multi-tenant app. We are facing performance issue and keep scaling up to cope with the query volumes (and trying to optimize query with Query Performance Insight). If we don't want to…
In Synapse Notebook using pyspark inbuilt functions how to convert country code to country name as description
Hi Team, Could you please help us that how to change country code to country name (as description) for a column in synapse notebook using pyspark (modules or libraries) could you please share the code. we have used below but it is not working. …
How does Azure Data Explorer in Synapse Analytics work as regards Cosmos db
Hi, I am trying different ways to get Cosmos db data into Synapse Analytics in real-time (near real-time is good enough). I have seen mentioned that it is possible to use Azure Data Explorer. Does Data Explorer use the Analytical Store to get the latest…
Synapse Spark shared Metadata not syncing to serverless sql
I have created a spark delta table using create table if not exists db_name.table_name( id int, propertyA string ) using delta and then later on changed a column name: alter table db_name.table_name RENAME COLUMN propertyA TO…
Error while loading data
Getting below error message while loading data using Pipeline. Failure happened on 'Source' side. ErrorCode=DB2DriverRunFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Error thrown from driver. Sql code:…
Azure Synapse Apache Spark Pools seemingly broken
Apache Spark and Delta Lake are 2 well-established technologies that you can either implement in a private cluster or use out of the box as part of a larger platform such as Databricks or Synapse. Typically, you want to use a managed platform to keep…
How to monitor Synapse Spark real time monitor
Hi Team, We are wondering how to monitor Spark Pool performance metrics in real time (ex CPU, Memory, Errors). In Apache Spark Pool section, I found chart can monitor how many Spark application, Vcore allocation or memory allocation But seems this one…
Connect to eventhub using Managed service identity (MSI) from Azure synapse analytics using pyspark spark streaming
I am trying to write data to Azure Event Hubs from an Azure Synapse Analytics notebook in PySpark using Managed Service Identity (MSI). Could you please provide step-by-step guidance on how to achieve this, along with the necessary code? Please note, I…
how to access a spark config from synapse pipeline
how to access and use an environment variable from spark config in synapse pipeline activity
The new ADF/Synapse connector to Salesforce no longer supports QueryAll to query deleted records in Salesforce. How can this be done?
Hi, The previous version of the ADF Salesforce connector (https://zcusa.951200.xyz/en-us/azure/data-factory/connector-salesforce-legacy?tabs=data-factory) supported querying deleted records in Salesforce: But the new version…
In Azure Synapse dedicated sql pool we have table with Identity column. But we are getting duplicate values in the Identity columns.
We have a table with Identity column in Azure Synapse Sql. Recently we found out that there are duplicates values are generated. We are loading this tables using SP's but there is no SET IDENTITY_INSERT ON command applied. We are not able to understand…
Files vanished from storage account
Morning! I was wondering if anyone knows what might be the case. We have list of jsons files which we store in a blob storage. We need those files in the Azure synapse therefore I created a notebook to ingest those files. After I ran an ingestion couple…