Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-22771

Partition location incorrectly formed in FileOutputCommitterContainer

    XMLWordPrintableJSON

Details

    Description

      Class HCatOutputFormat in package org.apache.hive.hcatalog.mapreduce uses function setOutput to generate idHash using below statement:

      In file org/apache/hive/hcatalog/mapreduce/HCatOutputFormat.java

       line 116: idHash = String.valueOf(Math.random());

      The output of idHash can be similar to values like this : 7.145347157239135E-4

       

      And, in class FileOutputCommitterContainer in package org.apache.hive.hcatalog.mapreduce;

      Uses below statement to compute final partition path:

      In org/apache/hive/hcatalog/mapreduce/FileOutputCommitterContainer.java

      line 366: String finalLocn = jobLocation.replaceAll(Path.SEPARATOR + SCRATCH_DIR_NAME + "
      d
      .?
      d+"
      ,"");

      line 367: partPath = new Path(finalLocn);

       

      Regex used here is incorrect, since it will only remove integers after the SCRATCH_DIR_NAME, and hence will append  'E-4' (for the above example) in the final partition location. 

      Attachments

        1. HIVE-22771.patch
          1 kB
          Shivam
        2. HIVE-22771.4.patch
          6 kB
          Shivam
        3. HIVE-22771.3.patch
          6 kB
          Shivam
        4. HIVE-22771.2.patch
          6 kB
          Shivam

        Issue Links

          Activity

            People

              shivam-mohan Shivam
              shivam-mohan Shivam
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h
                  1h