Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-7388

Apache Drill Kafka Storage module fails to return results for partitions containing single offset record

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.16.0
    • 1.17.0
    • None

    Description

      If a partition only contains one record - e.g.

      [topicName=myTopic, partitionId=117, startOffset=0, endOffset=1]

      no data is returned.

      I fixed this locally with the following code change in contrib/storage-kafka :-

      git diff src/main/java/org/apache/drill/exec/store/kafka/KafkaRecordReader.java

@@ -109,7 +109,7 @@ public class KafkaRecordReader extends AbstractRecordReader {
     currentMessageCount = 0;
 
     try {
-      while (currentOffset < subScanSpec.getEndOffset() - 1 && msgItr.hasNext()) {
+      while (currentOffset < subScanSpec.getEndOffset() && msgItr.hasNext()) {
         ConsumerRecord<byte[], byte[]> consumerRecord = msgItr.next();

      Attachments

        Issue Links

          incorporates

          Bug - A problem which impairs or prevents the functions of the product. DRILL-6723 Kafka reader fails on malformed JSON

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. DRILL-7290 “Failed to construct kafka consumer” using Apache Drill

          • Major - Major loss of function.
          • Resolved

          Task - A task that needs to be done. DRILL-6739 Update Kafka libs to 2.0.0+ version

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link GitHub Pull Request #1901

          Activity

            People

              arina Arina Ielchiieva
              atlassdan daniel kelly
              Vova Vysotskyi Vova Vysotskyi
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: