Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-20639

Add ability to Write Data from Hive Table/Query to Kafka Topic

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 4.0.0-alpha-1
    • kafka integration
    • None

    Description

      This patch adds multiple record writers to allow Hive user writing data directly to a Kafka Topic.
      The end goal is have a writer that provides multiple write semantics modes.

        • A BEST_EFFORT where all the records will be delivered with no guarantee or reties eg best effort mode. (Done)-
      • B At_least_once, each record will be delivered with retries from the Kafka Producer and Hive Write Task. (Done)
      • C Exactly_once , Writer will be using Kafka Transaction API to ensure that each record is delivered once. (not so, still need more discussions)

      In addition to the new feature i have refactored the existing code to make it more readable.

      Attachments

        1. HIVE-20639.patch
          263 kB
          Slim Bouguerra
        2. HIVE-20639.6.patch
          283 kB
          Slim Bouguerra
        3. HIVE-20639.5.patch
          282 kB
          Slim Bouguerra
        4. HIVE-20639.4.patch
          279 kB
          Slim Bouguerra
        5. HIVE-20639.4.patch
          279 kB
          Slim Bouguerra
        6. HIVE-20639.3.patch
          278 kB
          Slim Bouguerra
        7. HIVE-20639.2.patch
          274 kB
          Slim Bouguerra

        Issue Links

          Activity

            People

              bslim Slim Bouguerra
              bslim Slim Bouguerra
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: