Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-36722

Problems with update function in koalas - pyspark pandas.

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.2.0, 3.3.0
    • 3.2.0
    • PySpark
    • None

    Description

      Hi I am using "from pyspark import pandas as ps" in a master build yesterday.
      I do have some columns that I need to join to one.
      In pandas I use update.

      54 FD_OBJECT_SUPPLIES_SERVICES_OBJECT_SUPPLY_SERVICE_ADDITIONAL_INFORMATION 23 non-null object
      55 FD_OBJECT_SUPPLIES_SERVICES_OBJECT_SUPPLY_SERVICE_ADDITIONAL_INFORMATION.P 24348 non-null object

      pd1['FD_OBJECT_SUPPLIES_SERVICES_OBJECT_SUPPLY_SERVICE_ADDITIONAL_INFORMATION'].update(pd1['FD_OBJECT_SUPPLIES_SERVICES_OBJECT_SUPPLY_SERVICE_ADDITIONAL_INFORMATION.P'])

      ---------------------------------------------------------------------------
      AssertionError Traceback (most recent call last)
      /tmp/ipykernel_73/391781247.py in <module>
      ----> 1 pd1['FD_OBJECT_SUPPLIES_SERVICES_OBJECT_SUPPLY_SERVICE_ADDITIONAL_INFORMATION'].update(pd1['FD_OBJECT_SUPPLIES_SERVICES_OBJECT_SUPPLY_SERVICE_ADDITIONAL_INFORMATION.P'])

      /opt/spark/python/pyspark/pandas/series.py in update(self, other)
      4549 raise TypeError("'other' must be a Series")
      4550
      -> 4551 combined = combine_frames(self._psdf, other._psdf, how="leftouter")
      4552
      4553 this_scol = combined["this"]._internal.spark_column_for(self._column_label)

      /opt/spark/python/pyspark/pandas/utils.py in combine_frames(this, how, preserve_order_column, *args)
      139 elif len(args) == 1 and isinstance(args[0], DataFrame):
      140 assert isinstance(args[0], DataFrame)
      --> 141 assert not same_anchor(
      142 this, args[0]
      143 ), "We don't need to combine. `this` and `that` are same."

      AssertionError: We don't need to combine. `this` and `that` are same.

      pd1.info()

      54 FD_OBJECT_SUPPLIES_SERVICES_OBJECT_SUPPLY_SERVICE_ADDITIONAL_INFORMATION 23 non-null object
      55 FD_OBJECT_SUPPLIES_SERVICES_OBJECT_SUPPLY_SERVICE_ADDITIONAL_INFORMATION.P 24348 non-null object

      Attachments

        Activity

          People

            dc-heros dgd_contributor
            bjornjorgensen Bjørn Jørgensen
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: