다음을 통해 공유

Query about Checkpoint Strategy in HDInsight Spark and Event Hubs Environment

wonho.kim 0 평판 포인트
2024-09-30T08:48:14.2033333+00:00

Dear Azure Support Team,

Hello, I am Wonho Kim from Samsung Electornics.

I have a question regarding the checkpoint strategy in the HDInsight Spark and Event Hubs environment.

As far as I know, using the Checkpoint feature of Spark allows us to perform actions related to failure recovery.

However, when there are some failure in Spark applications resulting in different offsets between Event Hubs and Spark Checkpoint,

is there any recommended method for managing checkpoints by Azure to ensure more accurate exactly-once guarantee?

In our previous service implementation using Flink and Kafka,

we performed a task where we fetched the offset from Kafka and aligned it with Flink's offset.

I would like to know if fetching the offset from Event Hubs and aligning it with Spark is a more precise or preferred approach.

Looking forward to your response.

Microsoft Q&A
Microsoft Q&A
이 태그를 사용하여 제안, 기능 요청 및 버그를 Microsoft Q&A 팀과 공유합니다. Microsoft Q&A 팀은 정기적으로 피드백을 평가하고 그 과정에서 업데이트를 제공합니다.
질문 236개
댓글 0개 설명 없음
투표 {count}개

답변

질문 작성자가 수락한 답변이라고 답변에 표시할 수 있으며, 이를 통해 작성자의 문제를 해결한 답변을 사용자가 알 수 있도록 도와줍니다.