Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-46238

spark-submit does not provide a method to setApplicationTimeouts

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.5.0
    • None
    • YARN
    • None

    Description

       
      ApplicationSubmissionContext#setApplicationTimeouts(Map<ApplicationTimeoutType, Long> applicationTimeouts)

      This can be used to enforce application maximum lifetime. {}

      {{}}

      Javadoc at

      https://hadoop.apache.org/docs/r3.0.2/api/org/apache/hadoop/yarn/api/records/ApplicationSubmissionContext.html#setApplicationTimeouts-java.util.Map-{}

      https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html

      It would be preferable to be able to set this to enforce timeout of an application.

      {{}}

      It is possible to set these cluster wide from yarn scheduler queues:

      https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html

      {{}}

      {{}}

      yarn.scheduler.capacity.<queue-path>.maximum-application-lifetime

      yarn.scheduler.capacity.root.<queue-path>.default-application-lifetime 

      {{}}

      however this is suboptimal in many cases as a submitted might have different requirements for application lifetime than the queue defaults.

      {{}}

      Attachments

        Activity

          People

            Unassigned Unassigned
            kortemik Mikko Kortelainen
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: