Uploaded image for project: 'Kylin'
  1. Kylin
  2. KYLIN-3369

Reduce the data size sink from Kafka topic to HDFS

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • v2.4.0
    • NRT Streaming
    • None

    Description

      When building a cube from Kafka topic, the first step is to sink the Kafka data to HDFS. In today's implementation, it will persist all the fields of a message to disk. While in many cases, only a couple of fields will be needed for cubing; Today's behavior wastes network bandwidth and disk space.

      Attachments

        Activity

          People

            shaofengshi Shao Feng Shi
            shaofengshi Shao Feng Shi
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: