Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-20225

SerDe to support Teradata Binary Format

    XMLWordPrintableJSON

Details

    • Reviewed

    Description

      When using TPT/BTEQ to export/import Data from Teradata, Teradata will generate/require binary files based on the schema.

      A Customized SerDe is needed in order to directly read these files from Hive or write these files in order to load back to TD.

      CREATE EXTERNAL TABLE `TABLE1`(
      ...)
      PARTITIONED BY (
      ...)
      ROW FORMAT SERDE
        'org.apache.hadoop.hive.contrib.serde2.TeradataBinarySerde'
      STORED AS INPUTFORMAT
       'org.apache.hadoop.hive.contrib.fileformat.teradata.TeradataBinaryFileInputFormat'
      OUTPUTFORMAT
       'org.apache.hadoop.hive.contrib.fileformat.teradata.TeradataBinaryFileOutputFormat'
      LOCATION ...;
      
      SELECT * FROM `TABLE1`;

      Problem Statement:

      Right now the fast way to export/import data from Teradata is using TPT. However, the Hive could not directly utilize/generate these binary format because it doesn't have a SerDe for these files.

      Result:

      Provided with the SerDe, Hive can operate upon/generate the exported Teradata Binary Format file transparently

      Attachments

        1. HIVE-20225.1.patch
          72 kB
          Lu Li
        2. HIVE-20225.2.patch
          83 kB
          Lu Li
        3. HIVE-20225.3.patch
          83 kB
          Lu Li
        4. HIVE-20225.4.patch
          83 kB
          Lu Li
        5. HIVE-20225.5-branch-2.patch
          78 kB
          Lu Li
        6. HIVE-20225.6.patch
          83 kB
          Lu Li
        7. HIVE-20225.7.patch
          83 kB
          Lu Li
        8. HIVE-20225.8.patch
          79 kB
          Lu Li
        9. HIVE-20225.9.patch
          94 kB
          Lu Li
        10. HIVE-20225.10.patch
          94 kB
          Lu Li
        11. HIVE-20225.11.patch
          96 kB
          Lu Li
        12. HIVE-20225.12.patch
          98 kB
          Lu Li
        13. HIVE-20225.13.patch
          115 kB
          Lu Li
        14. HIVE-20225.14-branch-2.patch
          117 kB
          Lu Li
        15. HIVE-20225.15.patch
          115 kB
          Lu Li
        16. HIVE-20225.16-branch-2.patch
          113 kB
          Lu Li
        17. HIVE-20225.17-branch-3.patch
          115 kB
          Lu Li
        18. HIVE-20225.18-branch-3.patch
          115 kB
          Lu Li
        19. HIVE-20225.19-branch-3.patch
          115 kB
          Lu Li

        Activity

          People

            luli Lu Li
            luli Lu Li
            Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: