Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-37114

Support Submitting Jobs to Cloud Providers Managed Spark Clusters

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Open
    • Minor
    • Resolution: Unresolved
    • 3.2.0
    • None
    • Deploy
    • None

    Description

      To be able to submit jobs to prominent cloud providers managed spark clusters, "spark-submit" can be enhanced. For example, to submit job to "google cloud dataproc", the "spark-submit" can be enhanced to issue "gcloud dataproc jobs submit spark ..." when "–master gcd://cluster-name" arg is used. Once this feature is accepted and prioritized, then it can be rolled out in current and future versions of spark and also back ported to a few previous versions. I can raise the pull request.

      Attachments

        Activity

          People

            Unassigned Unassigned
            NagaVijayapuram Naga Vijayapuram
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: