Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-5442

Add only credentials from setStoreLocation to the Job Conf

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 0.18.0
    • None
    • None

    Description

      While testing HCatStorer with Iceberg realized Pig calls setStoreLocation on all Stores with the same Job object - https://github.com/apache/pig/blob/b050a33c66fc22d648370b5c6bda04e0e51d3aa3/src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java#L1081

      Setting populated by one store is affecting the other stores. In my case the 
      "mapred.output.committer.class" is set as HiveIcebergCommitter by PigStore that is used by the Iceberg table and the other stores which inserts data to a non-iceberg tables also use that setting and trying to use HiveIcebergCommitter.
       
      On checking with rohini , it is called to get the credentials from all stores since addCredentials API was added later and not all stores have implemented it and some still set configuration in setLocation method (i.e, HCatStorer). 
       
      Fixed it by passing a separate copy of Job object to each store's setLocation method and adding only the credential object from the call.

      Attachments

        1. PIG-5442-1.patch
          1 kB
          Rohini Palaniswamy

        Issue Links

          Activity

            People

              maswin Alagappan Maruthappan
              maswin Alagappan Maruthappan
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: