Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-28367

Kafka connector infinite wait because metadata never updated

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete CommentsDelete
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 2.1.3, 2.2.3, 2.3.3, 2.4.3, 3.0.0, 3.1.0
    • 3.1.0
    • Structured Streaming
    • None

    Description

      Spark uses an old and deprecated API named poll(long) which never returns and stays in live lock if metadata is not updated (for instance when broker disappears at consumer creation).

      Attachments

        Issue Links

        Is contained by

        Sub-task - The sub-task of the issue SPARK-30169 Eliminate warnings in Kafka connector

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Open
        is duplicated by

        Improvement - An improvement or enhancement to an existing feature or task. SPARK-30311 Use KafkaConsumer.poll(Duration) instead of deprecated one

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Resolved
        links to

        Web Link GitHub Pull Request #25135

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            gsomogyi Gabor Somogyi Assign to me
            gsomogyi Gabor Somogyi
            Votes:
            2 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment