Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-28652

spark.kubernetes.pyspark.pythonVersion is never passed to executors

    XMLWordPrintableJSON

Details

    • Test
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 2.4.3
    • 3.0.0
    • Kubernetes, Spark Core
    • None
    • Important

    Description

      I suppose this may not be a priority with Python2 on its way out, but given that this setting is only ever sent to the driver and not the executors, no actual work can be performed when the versions don't match, which will tend to be always with the default setting for the driver being changed from 2 to 3, and the executors using `python`, which defaults to v2, by default.

      Attachments

        Issue Links

          Activity

            People

              viirya L. C. Hsieh
              nanav.yorbiz nanav yorbiz
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: