Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-31302

Shared Volume Spark Kubernetes

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Incomplete
    • 2.4.5
    • None
    • Spark Core

    Description

      Please change the deployment script for the workers to support one Shared Volume with the driver. E.g. a possible implementation:
      https://kubernetes.io/docs/tasks/access-application-cluster/communicate-containers-same-pod-shared-volume/

      Atm. If a driver runs in Kubernetes it is not possible to share a volume between the driver and the workers.
      Enabling a shared volume between Driver and Workers would allow the use of sc.textFile(file:///path/file.txt) to be used without any need of HDFS or S3 Storage (Which in fact is a huge benefit)

      Attachments

        Activity

          People

            Unassigned Unassigned
            renedlog Stefan Goldener
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: