Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33005 Kubernetes GA Preparation
  3. SPARK-25355

Support --proxy-user for Spark on K8s

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.1.0
    • 3.1.0
    • Kubernetes, Spark Core
    • None

    Description

      SPARK-23257 adds kerberized hdfs support for Spark on K8s. A major addition needed is the support for proxy user. A proxy user is impersonated by a superuser who executes operations on behalf of the proxy user. More on this: 

      https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Superusers.html

      https://github.com/spark-notebook/spark-notebook/blob/master/docs/proxyuser_impersonation.md

      This has been implemented for Yarn upstream and Spark on Mesos here:

      https://github.com/mesosphere/spark/pull/26

      ifilonenko creating this issue according to our discussion.

      Attachments

        1. screenshot-1.png
          34 kB
          Gabor Somogyi
        2. with_proxy_extradebugLogs.log
          175 kB
          Shrikant Prasad
        3. client.log
          221 kB
          Shrikant Prasad
        4. driver.log
          156 kB
          Shrikant Prasad

        Issue Links

          Activity

            People

              pedro.rossi Pedro Rossi
              skonto Stavros Kontopoulos
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: