Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-45981 Improve Python language test coverage
  3. SPARK-45988

Fix `pyspark.pandas.tests.computation.test_apply_func` in Python 3.11

    XMLWordPrintableJSON

Details

    Description

      https://github.com/apache/spark/actions/runs/6914662405/job/18812759697

      ======================================================================
      ERROR [0.686s]: test_apply_batch_with_type (pyspark.pandas.tests.computation.test_apply_func.FrameApplyFunctionTests.test_apply_batch_with_type)
      ----------------------------------------------------------------------
      Traceback (most recent call last):
        File "/__w/spark/spark/python/pyspark/pandas/tests/computation/test_apply_func.py", line 248, in test_apply_batch_with_type
          def identify3(x) -> ps.DataFrame[float, [int, List[int]]]:
                              ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/__w/spark/spark/python/pyspark/pandas/frame.py", line 13540, in __class_getitem__
          return create_tuple_for_frame_type(params)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/__w/spark/spark/python/pyspark/pandas/typedef/typehints.py", line 721, in create_tuple_for_frame_type
          return Tuple[_to_type_holders(params)]
                       ^^^^^^^^^^^^^^^^^^^^^^^^
        File "/__w/spark/spark/python/pyspark/pandas/typedef/typehints.py", line 766, in _to_type_holders
          data_types = _new_type_holders(data_types, NameTypeHolder)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/__w/spark/spark/python/pyspark/pandas/typedef/typehints.py", line 832, in _new_type_holders
          raise TypeError(
      TypeError: Type hints should be specified as one of:
        - DataFrame[type, type, ...]
        - DataFrame[name: type, name: type, ...]
        - DataFrame[dtypes instance]
        - DataFrame[zip(names, types)]
        - DataFrame[index_type, [type, ...]]
        - DataFrame[(index_name, index_type), [(name, type), ...]]
        - DataFrame[dtype instance, dtypes instance]
        - DataFrame[(index_name, index_type), zip(names, types)]
        - DataFrame[[index_type, ...], [type, ...]]
        - DataFrame[[(index_name, index_type), ...], [(name, type), ...]]
        - DataFrame[dtypes instance, dtypes instance]
        - DataFrame[zip(index_names, index_types), zip(names, types)]
      However, got (<class 'int'>, typing.List[int]).
      
      ----------------------------------------------------------------------
      Ran 10 tests in 34.327s
      
      FAILED (errors=1)
      

      Attachments

        Issue Links

          Activity

            People

              dongjoon Dongjoon Hyun
              dongjoon Dongjoon Hyun
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: