Share via

How to insert huge records like 1000 and 10000 records into sql db using logic app without having timeout or server error using batching

Pragiri 0 Reputation points
2024-10-16T09:10:15.56+00:00

I have a csv file where I am able to read the data andnwhile passing rhe data into my sql getting timeout or server errors of it is a huge data how to do batching effectivily

Azure Functions
Azure Functions
An Azure service that provides an event-driven serverless compute platform.
5,029 questions
Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
3,183 questions
Azure Logic Apps
Azure Logic Apps
An Azure service that automates the access and use of data across clouds without writing code.
3,162 questions
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. LeelaRajeshSayana-MSFT 15,556 Reputation points Microsoft Employee
    2024-10-16T21:35:54.42+00:00

    Hi @Pragiri Greetings! Welcome to Microsoft Q&A forum. Thank you for posting this question here.Using Azure Logic App to process a CSV file, especially if it is large, isn't a recommended approach. Azure Logic app has limited data manipulation capabilities which are supported natively for CSV format.

    A better approach would be to use an Azure Blob storage trigger function which gets triggered when we upload a CSV file into the Azure blob storage end point. If you use Python, you can leverage the Pandas library and parse the CSV file in chunks and load it to the SQL server. You can find a sample code on how to chunk and read a CSV file using Python from the following link

    If you prefer to use a different language, please let us know and we can share a sample with you.

    I am assuming you do not have access to SQL server instance. If you can access it through SSMS, an easy way to do the data import is by Import Flat File to SQL Wizard

    Let us know if you have any additional questions or need further assistance.

    If the response helped, please do click Accept Answer and Yes for the answer provided. Doing so would help other community members with similar issue identify the solution. I highly appreciate your contribution to the community.

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.