Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-36035

Adjust `test_astype`, `test_neg` for old pandas versions

    XMLWordPrintableJSON

Details

    • Test
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.2.0
    • 3.2.0
    • PySpark
    • None

    Description

      • test_astype

      For pandas < 1.1.0, declaring or converting to StringDtype was in general only possible if the data was already only str or nan-like (GH31204).

      In pandas 1.1.0, the problem is adjusted by https://pandas.pydata.org/pandas-docs/stable/whatsnew/v1.1.0.html#all-dtypes-can-now-be-converted-to-stringdtype.

      That should be considered in `test_astype`, otherwise, current tests will fail with pandas < 1.1.0.

      • test_neg
      dtypes = [
        "Int8",
        "Int16",
        "Int32",
        "Int64",
      ]
      psers = []
      for dtype in dtypes:
        psers.append(pd.Series([1, 2, 3, None], dtype=dtype))
        
      for pser in psers:
        print((-pser).dtype)

      ~ 1.0.5, object dtype
      1.1.0~1.1.2, TypeError: bad operand type for unary -: 'IntegerArray'
      1.1.3, correct respective dtype

      Attachments

        Issue Links

          Activity

            People

              XinrongM Xinrong Meng
              XinrongM Xinrong Meng
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: