Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-31339

Changed PipelineModel(...) to self.cls(...) in pyspark.ml.pipeline.PipelineModelReader.load()

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Not A Problem
    • 2.4.5
    • None
    • ML, PySpark
    • Patch

    Description

      PR: https://github.com/apache/spark/pull/28110

      • What changes were proposed in this pull request?
        pypsark.ml.pipeline.py line 245: Change PipelineModel(...) to self.cls(...)
      • Why are the changes needed?
        This change fixes the loading of class (which inherits from PipelineModel class) from file.
        E.g. Current issue:
      CustomPipelineModel(PipelineModel):
          def _transform(self, df):
              ...
       CustomPipelineModel.save('path/to/file') # works
       CustomPipelineModel.load('path/to/file') # wrong: results in PipelineModel() instead of CustomPipelineModel()
       CustomPipelineModel.transform() # wrong: results in calling PipelineModel.transform() instead of CustomPipelineModel.transform()
      • Does this introduce any user-facing change?
        No.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              surajiyer Suraj
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: