Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-7227

Fix job staging directory residual problem in a big yarn cluster composed of multiple independent hdfs clusters

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.6.0, 2.7.0, 3.1.2
    • Fix Version/s: None
    • Component/s: applicationmaster, mrv2
    • Labels:
      None

      Description

      Our yarn cluster is made up of some independent hdfs cluster, the 'default.FS' in every hdfs cluster is different.when user submit job to yarn cluster, if the 'default.FS'  between client and nodemanager  is inconsistent, then the job staging dir can't be cleanup by AppMaster. Because it will produce two job staging dirs in our conditions by client and appmaster. So we can modify AppMaster through client's ‘default.FS’ to create job staging dir.

        Attachments

        1. 1.png
          25 kB
          Yuan LUO
        2. 2.png
          35 kB
          Yuan LUO
        3. HADOOP-MAPREDUCE-7227.001.patch
          3 kB
          Yuan LUO
        4. HADOOP-MAPREDUCE-7227.002.patch
          3 kB
          Yuan LUO
        5. HADOOP-MAPREDUCE-7227.003.patch
          3 kB
          Yuan LUO
        6. HADOOP-MAPREDUCE-7227.004.patch
          3 kB
          Yuan LUO
        7. HADOOP-MAPREDUCE-7227.005.patch
          3 kB
          Yuan LUO
        8. Process_Analysis.png
          64 kB
          Yuan LUO

          Activity

            People

            • Assignee:
              luoyuan Yuan LUO
              Reporter:
              luoyuan Yuan LUO
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: