Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-11054

Alleviate LocalJobRunnerMetricName Conflicts

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Minor
    • Resolution: Unresolved
    • 3.3.1
    • 3.3.1
    • None

    Description

      In some scenarios, Sqoop will use LocalJobRuner (YarnLocal mode) to run a lot of jobs,  assuming  2 million jobs have been run, LocalJobRunner MetricName generated by nextInt function is in the range of (0, 2147483647),
      Then the probability of conflict is about 2000/2147483647 = 1/1000, which means that an average of 1 task will fail for every 1000 jobs run.
      If LocalJobRunner MetricName is generated by nextLong() whose range is (0, 9223372036854775807), considering that Long's range is 1 billion times that of Int, the probability of a new MetricName conflicts is also reduced by one trillionth times.
      the probability of conflict also goes to 1/500000000 from 1/1000. ( under the situation that there are about 200million jobs have been run)

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              XingjunHao Xingjun Hao
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m