Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-44845

spark job copies jars repeatedly if fs.defaultFS and application jar are same url

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.4.1
    • 4.0.0
    • YARN
    • None

    Description

      In the org.apache.spark.deploy.yarn.Client#compareUri method, hdfs://hadoop81:8020 and hdfs://192.168.0.81:8020 are regarded as different file systems (hadoop81 corresponds to 192.168.0.81). The specific reason is that in the last pr, different URIs of user information are also regarded as different file systems. Uri.getauthority is used to determine the user information, but authority contains the host so the URI above must be different from authority. To determine whether the user authentication information is different, you only need to determine URI.getUserInfo.

       

      the last pr and issue link:
      https://issues.apache.org/jira/browse/SPARK-22587

      https://github.com/apache/spark/pull/19885

      Attachments

        Activity

          People

            zheju_he zheju_he
            zheju_he zheju_he
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: