Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-7370

Parallelize MultipleOutputs#close call

    XMLWordPrintableJSON

Details

    • Reviewed

    Description

      This call takes more time when there are lot of files to close and there is a high latency to close. Parallelize MultipleOutputs#close call to improve the speed.

        public void close() throws IOException {
          for (RecordWriter writer : recordWriters.values()) {
            writer.close(null);
          }
        }
      

      Idea is from stevel@apache.org

      Attachments

        Issue Links

          Activity

            People

              groot Ashutosh Gupta
              prabhujoseph Prabhu Joseph
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 2h 20m
                  2h 20m