5,134 questions with Azure Synapse Analytics tags

Sort by: Updated
1 answer

Azure Synapse pipeline

Hi, I have a question regarding Synapse. I want to create a pipeline that can delta update the target table “quotes_target” from source table “quotes_source” with filtered quotes data. I have a list of quotes id of the updated quotes after I extracted it…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,134 questions
asked 17 Jan 2025, 9:16 am
Lotus88 46 Reputation points
commented 18 Jan 2025, 1:13 am
Lotus88 46 Reputation points
1 answer

Updating an existing ADF pipeline - ( Importing new code )

We use different adf instances for dev and production and I was wondering how to update an existing pipeline in prod instance. The updated pipeline sitting in the dev instance and if we try to import again it creates a duplicate pipelines and datasets.…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,134 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,119 questions
asked 16 Jan 2025, 6:20 pm
Jiby Jacob 70 Reputation points
commented 17 Jan 2025, 8:14 pm
Chandra Boorla 6,695 Reputation points Microsoft Vendor
0 answers

I can't get diagnostics logging working on a dedicated sql pool

I want to see the exec requests and waits on a dedicated pool. So I created a log analytics workspace in the same workspace as the sql pool and I went to the dedicated sql pool and in the Diagnostics Settings option I created a new one for the logs exec…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,134 questions
asked 15 Jan 2025, 8:25 am
Ellen Heijmans 1 Reputation point
commented 17 Jan 2025, 7:42 pm
Chandra Boorla 6,695 Reputation points Microsoft Vendor
1 answer

Spark Diagnostic Emitter Not Working with Event Hub

I am following this documentation to send logs and metrics for Spark applications running on Synapse Spark pools to Event Hub using Azure Key Vault (AKV). I am using the configuration below and have added the Service Principal secret to AKV. I also tried…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,134 questions
asked 15 Jan 2025, 5:22 am
Raut, Saurabh J 0 Reputation points
commented 17 Jan 2025, 7:33 pm
Chandra Boorla 6,695 Reputation points Microsoft Vendor
5 answers

Sometimes, none of my sql query terminate (even the smallest ones)

I'm facing occasional hanging T-SQL requests on my database since this morning (01/15/2025), even those that usually complete in less than a second. The solution I found is to kill the running requests in T-SQL via their session ID and wait one or two…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,134 questions
asked 15 Jan 2025, 2:38 pm
Thibaut ROPERCH 0 Reputation points
edited an answer 17 Jan 2025, 6:02 pm
Ketsha 250 Reputation points Microsoft Employee
1 answer One of the answers was accepted by the question author.

Best Practices for Ensuring Data Security with Microsoft Synapse SAP Connector and Azure Linked Services

My client asked for research on how SAP teams can protect data when using Azure Linked Services. There's particular concern regarding the lack of understanding of how the technology works. The setup involves using the Microsoft Synapse SAP connector to…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,134 questions
asked 17 Jan 2025, 4:51 pm
Ammar Asim 0 Reputation points
commented 17 Jan 2025, 5:11 pm
Ammar Asim 0 Reputation points
0 answers

How to read parquet file to update target table?

Hi, I have a question regarding Synapse. I want to create a pipeline that can delta update the target table “quotes_target” from source table “quotes_source” with filtered quotes data. I have a list of quotes id of the updated quotes after I extracted it…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,134 questions
asked 17 Jan 2025, 9:18 am
Lotus88 46 Reputation points
commented 17 Jan 2025, 4:02 pm
Ganesh Gurram 3,110 Reputation points Microsoft Vendor
2 answers

ADF/ Synapse Activity times out after 12 hours

Hello, I have a Notebook activity in one of my pipelines that is expected to run longer than 12 hours. Below we configured the timeout to the max value, 7 days. But the activity would still timeout after 12 hours. Is there a way for the activity…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,527 questions
Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
3,326 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,134 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,119 questions
asked 13 Jan 2025, 11:02 pm
Marc 0 Reputation points
commented 17 Jan 2025, 3:23 pm
Marc 0 Reputation points
2 answers One of the answers was accepted by the question author.

Azure SQL alternative

Hi, Currently, we are using Azure SQL Single database for a multi-tenant app. We are facing performance issue and keep scaling up to cope with the query volumes (and trying to optimize query with Query Performance Insight). If we don't want to…

Azure SQL Database
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,134 questions
Azure Database for PostgreSQL
asked 26 Sept 2022, 2:58 am
Testa 551 Reputation points
edited an answer 17 Jan 2025, 1:59 pm
Bobur Umurzokov 0 Reputation points MVP
0 answers

In Synapse Notebook using pyspark inbuilt functions how to convert country code to country name as description

Hi Team, Could you please help us that how to change country code to country name (as description) for a column in synapse notebook using pyspark (modules or libraries) could you please share the code. we have used below but it is not working. …

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,134 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,305 questions
asked 17 Jan 2025, 9:13 am
commented 17 Jan 2025, 1:32 pm
Smaran Thoomu 19,135 Reputation points Microsoft Vendor
1 answer

How does Azure Data Explorer in Synapse Analytics work as regards Cosmos db

Hi, I am trying different ways to get Cosmos db data into Synapse Analytics in real-time (near real-time is good enough). I have seen mentioned that it is possible to use Azure Data Explorer. Does Data Explorer use the Analytical Store to get the latest…

Azure Data Explorer
Azure Data Explorer
An Azure data analytics service for real-time analysis on large volumes of data streaming from sources including applications, websites, and internet of things devices.
541 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,134 questions
asked 17 Jan 2025, 9:15 am
Christopher Wilkinson 0 Reputation points
answered 17 Jan 2025, 9:52 am
Smaran Thoomu 19,135 Reputation points Microsoft Vendor
1 answer One of the answers was accepted by the question author.

Synapse Spark shared Metadata not syncing to serverless sql

I have created a spark delta table using create table if not exists db_name.table_name( id int, propertyA string ) using delta and then later on changed a column name: alter table db_name.table_name RENAME COLUMN propertyA TO…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,134 questions
asked 2 Aug 2024, 8:20 am
Finn Schmidt 86 Reputation points
commented 17 Jan 2025, 9:49 am
Finn Schmidt 86 Reputation points
1 answer

Error while loading data

Getting below error message while loading data using Pipeline. Failure happened on 'Source' side. ErrorCode=DB2DriverRunFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Error thrown from driver. Sql code:…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,134 questions
asked 7 Jan 2025, 5:24 pm
Gunti, Srikanth 0 Reputation points
commented 17 Jan 2025, 8:50 am
Smaran Thoomu 19,135 Reputation points Microsoft Vendor
0 answers

Azure Synapse Apache Spark Pools seemingly broken

Apache Spark and Delta Lake are 2 well-established technologies that you can either implement in a private cluster or use out of the box as part of a larger platform such as Databricks or Synapse. Typically, you want to use a managed platform to keep…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,134 questions
asked 15 Jan 2025, 9:30 pm
Homero Rivera 0 Reputation points
commented 17 Jan 2025, 8:22 am
phemanth 13,050 Reputation points Microsoft Vendor
0 answers

How to monitor Synapse Spark real time monitor

Hi Team, We are wondering how to monitor Spark Pool performance metrics in real time (ex CPU, Memory, Errors). In Apache Spark Pool section, I found chart can monitor how many Spark application, Vcore allocation or memory allocation But seems this one…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,134 questions
asked 17 Jan 2025, 5:26 am
Jiacheng Zhang 20 Reputation points
commented 17 Jan 2025, 6:30 am
Jiacheng Zhang 20 Reputation points
2 answers

Connect to eventhub using Managed service identity (MSI) from Azure synapse analytics using pyspark spark streaming

I am trying to write data to Azure Event Hubs from an Azure Synapse Analytics notebook in PySpark using Managed Service Identity (MSI). Could you please provide step-by-step guidance on how to achieve this, along with the necessary code? Please note, I…

Azure Event Hubs
Azure Event Hubs
An Azure real-time data ingestion service.
673 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,134 questions
asked 26 Sept 2024, 5:45 am
Sunil Yadav 0 Reputation points
edited a comment 17 Jan 2025, 4:10 am
Deshpande, Chetan 0 Reputation points Microsoft Employee
2 answers

how to access a spark config from synapse pipeline

how to access and use an environment variable from spark config in synapse pipeline activity

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,134 questions
asked 13 Jan 2025, 7:42 am
Brunda Yelakala (INFOSYS LIMITED) 0 Reputation points Microsoft Vendor
commented 17 Jan 2025, 3:44 am
Ganesh Gurram 3,110 Reputation points Microsoft Vendor
2 answers

The new ADF/Synapse connector to Salesforce no longer supports QueryAll to query deleted records in Salesforce. How can this be done?

Hi, The previous version of the ADF Salesforce connector (https://zcusa.951200.xyz/en-us/azure/data-factory/connector-salesforce-legacy?tabs=data-factory) supported querying deleted records in Salesforce: But the new version…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,134 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,119 questions
asked 28 Feb 2024, 9:15 pm
Kevin Chadney 10 Reputation points
answered 16 Jan 2025, 4:37 pm
Manish Lagisetty 0 Reputation points
1 answer

In Azure Synapse dedicated sql pool we have table with Identity column. But we are getting duplicate values in the Identity columns.

We have a table with Identity column in Azure Synapse Sql. Recently we found out that there are duplicates values are generated. We are loading this tables using SP's but there is no SET IDENTITY_INSERT ON command applied. We are not able to understand…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,134 questions
asked 13 Apr 2023, 10:24 am
Nikhil Gumulwar 45 Reputation points
commented 16 Jan 2025, 5:42 am
P 0 Reputation points
1 answer

Files vanished from storage account

Morning! I was wondering if anyone knows what might be the case. We have list of jsons files which we store in a blob storage. We need those files in the Azure synapse therefore I created a notebook to ingest those files. After I ran an ingestion couple…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,134 questions
asked 13 Jan 2025, 6:37 am
Aleksandra Stan 0 Reputation points
commented 16 Jan 2025, 4:03 am
Ganesh Gurram 3,110 Reputation points Microsoft Vendor