Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-24567

nodeBlacklist does not get updated if a spark executor fails to launch on a mesos node

    XMLWordPrintableJSON

Details

    Description

      As fix of SPARK-19755 we removed custom blacklisting mechanism in spark-mesos integration which has hardcoded constant of 2 failures max before node is marked as blacklisted.

      From now on the usual blacklisting mechanism is in use(when enabled), however it has downside of not counting failures of launching mesos-tasks(spark executors), i.e. only failures in spark-tasks will be counted.

      squito felixcheung  susanxhuynh skonto please add details as you see it

      Attachments

        Issue Links

          is caused by

          Bug - A problem which impairs or prevents the functions of the product. SPARK-19755 Blacklist is always active for MesosCoarseGrainedSchedulerBackend. As result - scheduler cannot create an executor after some time.

          • Major - Major loss of function.
          • Resolved
          is related to

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-16630 Blacklist a node if executors won't launch on it.

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              igor.berman Igor Berman
              Votes:
              1 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: