Append vs update in Azure Data Lake Storage Gen2 for a csv file.

Hi Ravikiran Srinivasulu ,
Welcome to Microsoft Q&A Forum. Thanks for posting you query here!
Appending data typically means adding new data to the end of an existing file without altering the existing content. For a CSV file, this would mean adding new rows at the end of the file, in ADLS Gen2, appending data requires specific permissions:

• Write (W) permission on the file.
• Execute (X) permission on the parent directory.

Updating data involves modifying the existing content of the file. This could mean changing existing rows, adding new rows, or deleting rows. For updating a file, the required permissions are:

• Write (W) and Execute (X) permissions on the parent directory.
• Write (W) permission on the file.

Based on Your Scenario

In your case, when you try to append new rows to the CSV file, it seems the system treats this as an update operation because the file needs to be saved after appending. This is why you need the update permissions (-WX for the parent directory and none for the file) to successfully append data.
To reasonably test the append operation:

1. Ensure you have the correct permissions:
   • For appending: RW- on the file and --X on the parent directory.
   • For updating: -WX on the parent directory and none on the file.
2. Perform the operation:
   • Try adding new rows to the CSV file and saving it. If the operation fails with append permissions, it indicates that the system is treating it as an update.
3. Adjust permissions if needed:
   • If appending fails, try setting the update permissions and perform the operation again.

These distinctions and ensuring the correct permissions, you should be able to manage your CSV file operations effectively
For more information:
https://zcusa.951200.xyz/en-us/azure/storage/blobs/data-lake-storage-access-control
https://zcusa.951200.xyz/en-us/azure/storage/blobs/data-lake-storage-access-control-model

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members

If you have any other questions or are still running into more issues, let me know in the "comments" and I would be happy to help you

HI Ravikiran Srinivasulu - Thanks for reaching out over Q&A forum.

When writing data to Storage via ADLS Gen2 API. there are 3 operations called in sequence i.e. Create File -> Append File -> Flush File.

So, that's what refers to the Append for ADLS Gen2 and different than what we have in conventional terms or what we have in Blob Storage (Append Blobs)

Whereas, when it comes to update, it means updating the existing file.

Hope that helps!

If there are any queries. let me know and will be glad to assist.

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members