Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-44111 Prepare Apache Spark 4.0.0
  3. SPARK-48068

`mypy` should have `--python-executable` parameter

    XMLWordPrintableJSON

Details

    Description

      We assumed that `PYTHON_EXECUTABLE` is used for `dev/lint-python` like the following. That's not true. We need to use `mypy`'s parameter to make it sure.

      https://github.com/apache/spark/blob/ff401dde50343c9bbc1c49a0294272f2da7d01e2/.github/workflows/build_and_test.yml#L705

      root@18c8eae5791e:/spark# PYTHON_EXECUTABLE=python3.9 mypy --python-executable=python3.11 --namespace-packages --config-file python/mypy.ini python/pyspark | wc -l
3428
root@18c8eae5791e:/spark# PYTHON_EXECUTABLE=python3.9 mypy --namespace-packages --config-file python/mypy.ini python/pyspark | wc -l
1
root@18c8eae5791e:/spark# PYTHON_EXECUTABLE=python3.11 mypy --namespace-packages --config-file python/mypy.ini python/pyspark | wc -l
1

      Attachments

        Issue Links

          is caused by

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-36989 Migrate type hint data tests

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link GitHub Pull Request #46314

          Activity

            People

              dongjoon Dongjoon Hyun
              dongjoon Dongjoon Hyun
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: