Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-42471 Distributed ML <> spark connect
  3. SPARK-45130

Avoid Spark connect ML model to change input pandas dataframe

    XMLWordPrintableJSON

Details

    Description

      Currently, to avoid data copy, Spark connect ML model directly changes input pandas dataframe for appending prediction columns. But we can use `pandas_df.copy(deep=False)` to shallow copy it and then append prediction columns in copied dataframe. This is easier for user to use it.

      Attachments

        Issue Links

          Activity

            People

              weichenxu123 Weichen Xu
              weichenxu123 Weichen Xu
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: