Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,047 questions
1 answer
How data factory connects to azure batch service Where batch service is in different subscription
Hi All, i have a subscription A where i have Data Factory I have a subscription B where i have azure batch service now i want to create a link service from data factory to azure batch it is not connected , i am trying using access key of batch and system…
asked 2024-12-24T11:23:08.41+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(262.40000000000003, 20%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMV%3C/text%3E%3C/svg%3E)
manish verma
481
Reputation points
commented 2024-12-25T12:27:09.5833333+00:00
manish verma
481
Reputation points
1 answer
Issues with creating Airflow environment in Azure data factory
Unable to create airflow environment.While creating with GUI we are getting some reference ee availability with rest API we are getting below error.How we can resolve it. { "error": { "code": "InternalError", …
Azure
Azure
A cloud computing platform and infrastructure for building, deploying and managing applications and services through a worldwide network of Microsoft-managed datacenters.
1,014 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,047 questions
asked 2024-12-24T19:01:48.83+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 85%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERS%3C/text%3E%3C/svg%3E)
Rashmi Sunkeneni
0
Reputation points
answered 2024-12-24T19:37:15.6066667+00:00
hossein jalilian
9,310
Reputation points
0 answers
ADF api delay
how to fix the ADF api latency where parallel apis sending to microsoft crm .. how to fix this latency api is taking time to send the data
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 13%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVS%3C/text%3E%3C/svg%3E)
2 answers
Unable to connect scope linked service from ADF
Hi Team, I am attempting to use the scope task in ADF to run my scope scripts, but the connection is failing despite having all the necessary permissions for ADLA. I am using a System Assigned Managed Identity (SAMI) to connect to my ADLA with the…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(9.6, 80%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EB%3C/text%3E%3C/svg%3E)
2 answers
Cannot reference to an invalid or deleted Credential object for Fabric Linked Service
Getting this error after refreshing ADF page , credentials is getting removed from Fabric Warehouse Linked Service . Seems a similar problem like…
asked 2024-11-08T05:50:40.7466667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(144, 68%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETB%3C/text%3E%3C/svg%3E)
Tirthankar Biswas
0
Reputation points
commented 2024-12-24T16:11:02.3766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(201.6, 18%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGG%3C/text%3E%3C/svg%3E)
Ganesh Gurram
2,305
Reputation points Microsoft Vendor
2 answers
Salesforce V2 Link Service doens't work. error message: a task was canceled Source=mscorlib
I am migrating the ADF salesforce connector to the new V2 connector and now testing it. When testing the connection for the link service, it is good. However, it failed in the copy activity. with message: Failure happened on 'Source' side.…
asked 2024-05-16T13:35:58.54+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(83.2, 81%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EC5%3C/text%3E%3C/svg%3E)
ChoiLiamhehimhis-5365
35
Reputation points
commented 2024-12-24T14:10:19.3866667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(281.6, 5%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EWS%3C/text%3E%3C/svg%3E)
William Switzer
0
Reputation points
1 answer
Not Able to create connection for Intranet URL to Fabric Data factory
I am working on a project where the requirement is to get content from an Intranet URL through Fabric Data Factory. I have used web Activity and tried to create the connection but getting below error- " request didn't reach the server from the…
asked 2024-12-20T12:59:01.1933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(185.6, 86%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPA%3C/text%3E%3C/svg%3E)
Pragya Awasthi
0
Reputation points
edited a comment 2024-12-24T12:17:53.8766667+00:00
Ganesh Gurram
2,305
Reputation points Microsoft Vendor
0 answers
Select transformation output schema is empty after renaming columns
I've followed the example from the link: https://zcusa.951200.xyz/en-us/azure/data-factory/tutorial-data-flow-dynamic-columns It's exactly what I need for my project, but unfortunately, no matter what I do, the output schema of the select…
asked 2024-11-20T16:26:53.2266667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(195.2, 75%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
AdnanT
0
Reputation points
commented 2024-12-24T06:40:50.8266667+00:00
AnnuKumari-MSFT
33,976
Reputation points Microsoft Employee
2 answers
Delay in Azure data factory pipeline. Why it is taking so long like 15 minutes to complete execution?
Azure data factory pipeline is calling several sub pipelines depending on the use case of the request. It is observed that when using Script activity it is taking minutes to execute and slowing down the pipeline performance. For other activities it is…
asked 2024-12-15T11:06:03.1033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(252.8, 28.000000000000004%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECN%3C/text%3E%3C/svg%3E)
Chatti, Niharika [E]
0
Reputation points
commented 2024-12-24T06:36:08.3266667+00:00
AnnuKumari-MSFT
33,976
Reputation points Microsoft Employee
1 answer
User could debug the copy data activity in pipeline successfully but sometimes fail
Problem Description: Recently, when users run the Copy Data Activity in Azure Data Factory, the debug operations are successful, but sometimes they fail. There have been no changes to the pipeline configuration. Both the source and sink used by the users…
asked 2024-12-20T01:56:41.6+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(211.20000000000002, 59%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EYZ%3C/text%3E%3C/svg%3E)
Yujin Zhu (Shanghai Wicresoft Co Ltd)
0
Reputation points Microsoft Vendor
commented 2024-12-24T02:47:08.37+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 65%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECB%3C/text%3E%3C/svg%3E)
Chandra Boorla
5,755
Reputation points Microsoft Vendor
1 answer
How do I split a large file into multiple files with specific number of rows in each file?
I have a large source file that I want to split, with each file having 10K rows. Data flow allows me to split into set number of partitions. But there is a problem. I don't want to split source file if it has only 10K or less rows. Anything above 10K…
asked 2021-03-22T17:45:22.733+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(64, 20%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBK%3C/text%3E%3C/svg%3E)
Balachandran Kannan
181
Reputation points
commented 2024-12-23T04:46:49.5166667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(236.8, 64%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJW%3C/text%3E%3C/svg%3E)
John Welch
0
Reputation points
0 answers
User configuration issue: Failure happened on 'Source' side.
Hi I am trying to copy Azure SQL database Prod Environment to Azure SQL database Sandbox Environment through ADF pipeline and getting below error during execution of copy data activity inside Foreach loop. ADF Pipeline is failing during Copy data…
asked 2024-12-17T17:28:35.7733333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(316.8, 27%, 40%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKA%3C/text%3E%3C/svg%3E)
Kandan-Azure
0
Reputation points
commented 2024-12-23T03:14:07.7266667+00:00
Ganesh Gurram
2,305
Reputation points Microsoft Vendor
1 answer
ADF MDX query pipeline failure at source side
I'm trying to load one month data of a report from SAP BW using SAP with MDX linked service on ADF. When I load only two or three dimensions as testing, it works, but when I add more dimension in the MDX to replicate the report, the debugging failed…
asked 2024-10-25T02:42:31.24+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(38.4, 93%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENL%3C/text%3E%3C/svg%3E)
Nyan Lynn Tun
0
Reputation points
accepted 2024-12-23T01:31:51.08+00:00
Nyan Lynn Tun
0
Reputation points
0 answers
How to get data for all the fields listed there in CRM using Azure data factory
For getting full load data from the CRM database to the Azure SQL database, I am using the table option in the copy source and getting the data for all the columns. But when I am trying to get incremental data using Query option for which I am using…
asked 2023-02-02T19:45:47.8566667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(304, 31%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJG%3C/text%3E%3C/svg%3E)
Jayasree Gade
5
Reputation points
commented 2024-12-22T21:24:14.4433333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(105.60000000000001, 28.999999999999996%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPM%3C/text%3E%3C/svg%3E)
pankaj mahra
0
Reputation points
2 answers
One of the answers was accepted by the question author.
ADLS Gen2 operation failed for: Operation returned an invalid status code 'Conflict'. Account: 'abc'. ErrorCode: 'EndpointUnsupportedAccountFeatures'. Message: 'This endpoint does not support BlobStorageEvents or SoftDelete.
I am new to ADF. while creating linked service for data lake storage Gen2 I am getting a error ADLS Gen2 operation failed. ADLS Gen2 operation failed for: Operation returned an invalid status code 'Conflict'. Account: 'abc'. ErrorCode:…
asked 2024-12-21T17:12:26.5366667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(179.20000000000002, 6%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMR%3C/text%3E%3C/svg%3E)
manoj reddy
20
Reputation points
commented 2024-12-22T14:50:01.7933333+00:00
manoj reddy
20
Reputation points
1 answer
Servicenow linked service used connectivity vendor
I am interested in using Servicenow Linked Service to extract data from Servicenow. There are two linked services available; the legacy version and version 2 that uses table API. The documentation mentions a built-in driver for enabling connectivity. Can…
asked 2024-06-27T10:50:37.5233333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(179.20000000000002, 99%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAB%3C/text%3E%3C/svg%3E)
Amir Bagheripour
0
Reputation points
edited the question 2024-12-22T13:29:20.1733333+00:00
TP
100.9K
Reputation points
1 answer
How to create a link service in Azure Data Factory for Rest API which has OAuth grant type as password type of authentication
We need to call Rest API which has OAuth grant type as password type of authentication in a loop to store response in CSV. I know we could use Copy Activity but we are unable to create link service with mentioned authentication. We tried to do simple…
asked 2024-12-20T12:27:29.0933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(233.6, 21%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKA%3C/text%3E%3C/svg%3E)
Kukade, Atharva
0
Reputation points
answered 2024-12-20T17:26:41.1433333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 61%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
phemanth
12,575
Reputation points Microsoft Vendor
1 answer
One of the answers was accepted by the question author.
not being able to create azure free account or even to upgrade to pay as u go
how am i suppose to learn the certification if am not being able to create account. i am really frustrated that am not being able to create account. provide me an efficient way to create account to practice for my certification
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(281.6, 81%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(67.2, 19%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESK%3C/text%3E%3C/svg%3E)
1 answer
One of the answers was accepted by the question author.
Azure Data Factory – ForEach overwriting records instead of inserting
I have followed this guide to create a for each loop retrieving the data I need and it is working, however I would like all my JSON records in a single file, but the copy task is overwriting the previously imported record and I cannot see how to make…
asked 2022-12-05T07:55:37.23+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(262.40000000000003, 66%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EML%3C/text%3E%3C/svg%3E)
Mia Lancaster
21
Reputation points
commented 2024-12-20T12:21:36.5066667+00:00
Kukade, Atharva
0
Reputation points
1 answer
One of the answers was accepted by the question author.
Product tag for the cost with wrong location
Hi support, I have been looking into our cost for the different resources and I see something strange for our Data factory as seen below. Our subscription is located in EU west but the Product tag mention US East. Is this an error or are we paying to…
asked 2024-12-17T07:57:03.5133333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(272, 94%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAD%3C/text%3E%3C/svg%3E)
Andre Drejer
20
Reputation points
accepted 2024-12-20T08:35:27.8666667+00:00
Andre Drejer
20
Reputation points