Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-27475

Downgrade Kafka version to 2.5.0 to stabilise Kafka handler

    XMLWordPrintableJSON

Details

    Description

      It was needed to shade a version of zstd-jni that is compatible with parquet in hive-ql after upgrading parquet version dowstream, otherwise Tez couldn't work with parquet and zstd encryption. Parquet and kafka-client both use zstd-jni. In hive-ql, zstd-jni is coming as a transitive dependency from kafka-client. The zstd-jni version in kafka-client 2.5.0 was not compatible with parquet, so in PR-4082 kafka-client was upgraded to latest version which used zstd-jni version close to the version in parquet.

      While the upgrade fixed the zstd-ini issue, It introduced multiple compatibility issue in the Kafka-handler and due to lack to test cases and disabled tests these issues went unidentified.

      With the recent refactor in HIVE-27402, Kafka-client dependency was moved out of hive-ql, Which fixed the original zstd-ini issue.(confirmed by difin)

      Hence , It makes more sense to downgrade kafka versions back to 2.5.0, stabalize kafka-handler, enable test and then upgrade.

      Ref to discussion can be found here: #4436

      Attachments

        Issue Links

          Activity

            People

              akshatm Akshat Mathur
              akshatm Akshat Mathur
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: