Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-39708

ALS Model Loading

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Question
    • Status: Resolved
    • Critical
    • Resolution: Not A Problem
    • 3.2.0
    • None
    • PySpark, Spark Submit

    Description

      I have an ALS model and saved it with these codes: 

                      als_path = "saved_models/best"
                      best_model.save(sc, path= als_path)

      However, when I try to load this model, it gives this error message:

       

          ---> 10 model2 = ALS.load(als_path)
          
          File /usr/local/spark/python/pyspark/ml/util.py:332, in MLReadable.load(cls, path)
              329 @classmethod
              330 def load(cls, path):
              331     """Reads an ML instance from the input path, a shortcut of `read().load(path)`."""
          --> 332     return cls.read().load(path)
          
          File /usr/local/spark/python/pyspark/ml/util.py:282, in JavaMLReader.load(self, path)
              280 if not isinstance(path, str):
              281     raise TypeError("path should be a string, got type %s" % type(path))
          --> 282 java_obj = self._jread.load(path)
              283 if not hasattr(self._clazz, "_from_java"):
              284     raise NotImplementedError("This Java ML type cannot be loaded into Python currently: %r"
              285                               % self._clazz)
          
          File /usr/local/spark/python/lib/py4j-0.10.9.3-src.zip/py4j/java_gateway.py:1321, in JavaMember.__call__(self, *args)
             1315 command = proto.CALL_COMMAND_NAME +\
             1316     self.command_header +\
             1317     args_command +\
             1318     proto.END_COMMAND_PART
             1320 answer = self.gateway_client.send_command(command)
          -> 1321 return_value = get_return_value(
             1322     answer, self.gateway_client, self.target_id, self.name)
             1324 for temp_arg in temp_args:
             1325     temp_arg._detach()
          
          File /usr/local/spark/python/pyspark/sql/utils.py:111, in capture_sql_exception.<locals>.deco(*a, **kw)
              109 def deco(*a, **kw):
              110     try:
          --> 111         return f(*a, **kw)
              112     except py4j.protocol.Py4JJavaError as e:
              113         converted = convert_exception(e.java_exception)
          
          File /usr/local/spark/python/lib/py4j-0.10.9.3-src.zip/py4j/protocol.py:326, in get_return_value(answer, gateway_client, target_id, name)
              324 value = OUTPUT_CONVERTER[type](answer[2:], gateway_client)
              325 if answer[1] == REFERENCE_TYPE:
          --> 326     raise Py4JJavaError(
              327         "An error occurred while calling {0}{1}{2}.\n".
              328         format(target_id, ".", name), value)
              329 else:
              330     raise Py4JError(
              331         "An error occurred while calling {0}{1}{2}. Trace:\n{3}\n".
              332         format(target_id, ".", name, value))
          
          Py4JJavaError: An error occurred while calling o372.load.
          : org.json4s.MappingException: Did not find value which can be converted into java.lang.String
              at org.json4s.reflect.package$.fail(package.scala:53)
              at org.json4s.Extraction$.$anonfun$convert$2(Extraction.scala:881)
              at scala.Option.getOrElse(Option.scala:189)
              at org.json4s.Extraction$.convert(Extraction.scala:881)
              at org.json4s.Extraction$.$anonfun$extract$10(Extraction.scala:456)
              at org.json4s.Extraction$.$anonfun$customOrElse$1(Extraction.scala:780)
       
      

       

      I both tried to use `ALS.load` or `ALSModel.load` as shown in the Apache spark documentation:

      https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.ml.recommendation.ALS.html#:~:text=als_path%20%3D%20temp_path%20%2B%20%22/als%22%0A%3E%3E%3E[1]

       

        [1]: https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.ml.recommendation.ALS.html#:~:text=als_path%20%3D%20temp_path%20%2B%20%22/als%22%0A%3E%3E%3E

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            zehranrgi zehra
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment