adf: How can I load data from Mangodb?

Hello @Stephen27

Thanks for the question and using MS Q&A platform.

You can load data from MongoDB incrementally using ADF, but you'll need to manage the incremental load process based on certain conditions, such as a timestamp or an incrementing field in your MongoDB collection.

• Identify Incremental Field - MongoDB doesn't inherently track changes, so you need a field in the documents that indicates when a document was created or updated. This is usually a timestamp field (e.g., createdAt or updatedAt) or a numeric field (e.g., an auto-incrementing ID).
• Create a Source Dataset - Create a dataset in ADF for MongoDB that points to your collection.
• Define Query for Incremental Data - In the Source of your ADF pipeline, you can define a filter query to load only the new or modified data since the last load. For example:

{ "updatedAt": { "$gt": "last_loaded_timestamp" } }

• Replace "last_loaded_timestamp" with the value of the last loaded record's timestamp, which you'll need to store in a metadata store (like an Azure SQL Database, blob, or another storage).
• Set Up Control Flow (Lookup/Stored Last Processed Timestamp) - Use the Lookup activity in ADF to retrieve the last loaded timestamp from your metadata store (e.g., Azure SQL or blob storage). This timestamp will be used in the MongoDB filter query to fetch only new or updated records.
• Copy Activity - In the Copy Activity, use the query with the updatedAt condition to only load the incremental data from MongoDB to Azure storage. This could be Azure Blob Storage, Azure Data Lake, etc.
• Update the Last Loaded Timestamp - After the load, update the stored timestamp to the maximum updatedAt value from the latest batch of data. This can be done with another Lookup or Stored Procedure Activity to update the timestamp in the metadata store for future incremental loads.

Here is a link with details that may help you:

https://stackoverflow.com/questions/76654844/load-mongodb-data-incrementally-through-azure-data-factory

Similar thread for reference: https://zcusa.951200.xyz/en-us/answers/questions/2107335/can-i-load-data-from-mongodb-incrementally-with-ad

Hope this helps. Do let us know if you have any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.