Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6473

Job submission can take a long time during Cluster initialization

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.8.0, 3.0.0-alpha1
    • performance
    • None
    • Reviewed

    Description

      During initialization in Cluster.java, the framework provider classes are loaded inside a sync block which can considerably increase job submission time when the number of submissions are high. The motive is to reduce time spent in this sync block safely to improve performance.

      synchronized (frameworkLoader) {
            for (ClientProtocolProvider provider : frameworkLoader) {
              LOG.debug("Trying ClientProtocolProvider : "
                  + provider.getClass().getName());
      

      Attachments

        1. 99%ile.png
          9 kB
          Kuhu Shukla
        2. avgtime.png
          9 kB
          Kuhu Shukla
        3. MAPREDUCE-6473-PerfTest.txt
          4 kB
          Kuhu Shukla
        4. MAPREDUCE-6473-v1.patch
          2 kB
          Kuhu Shukla
        5. MAPREDUCE-6473-v2.patch
          3 kB
          Kuhu Shukla
        6. MAPREDUCE-6473-v3.patch
          3 kB
          Kuhu Shukla
        7. MAPREDUCE-6473-v4.patch
          3 kB
          Kuhu Shukla
        8. MAPREDUCE-6473-v5.patch
          4 kB
          Kuhu Shukla
        9. MAPREDUCE-6473-v6.patch
          4 kB
          Kuhu Shukla
        10. MAPREDUCE-6473-v7.patch
          4 kB
          Kuhu Shukla

        Issue Links

          Activity

            People

              kshukla Kuhu Shukla
              kshukla Kuhu Shukla
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: