Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-9840

Support for Parameterized Types when converting from HCatRecords to Rows in HCatalogIO

Details

    • Improvement
    • Status: Resolved
    • P2
    • Resolution: Resolved
    • None
    • 2.22.0
    • io-java-hcatalog
    • None

    Description

      In Hive, to use CHAR and VARCHAR as the data type for a column, the types have to be parameterized with the length of the character sequence. Refer https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/serde/src/test/org/apache/hadoop/hive/serde2/typeinfo/TestTypeInfoUtils.java#L68

      In addition, for the DECIMAL data type, custom precision and scale can be provided as parameters.

      A user has faced an Exception while reading data from a table created in Hive with a parameterized DECIMAL data type. Refer https://lists.apache.org/thread.html/r159012fbefce24d734096e3ec24ecd112de5f89b8029e57147d233b0%40%3Cuser.beam.apache.org%3E.

      This ticket is created to support converting from HcatRecords to Rows when the HCatRecords have parameterized types.

      To Support Parameterized data types, we can make use of HcatFieldSchema. Refer https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/hcatalog/core/src/main/java/org/apache/hive/hcatalog/data/schema/HCatFieldSchema.java#L34

      Attachments

        Issue Links

          Activity

            People

              rahul8383 Rahul Patwari
              rahul8383 Rahul Patwari
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1.5h
                  1.5h