Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-35217

com.google.protobuf.Parser.parseFrom() method Can't use in spark

    XMLWordPrintableJSON

Details

    • Question
    • Status: Resolved
    • Trivial
    • Resolution: Incomplete
    • 2.4.7
    • None
    • Deploy, ML
    • None

    Description

      platform:
      Linux ip-172-17-1-1 4.14.219-164.354.amzn2.x86_64 #1 SMP Mon Feb 22 21:18:39 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
      Hive 2.3.7, ZooKeeper 3.4.14, Spark 2.4.7, TensorFlow 2.4.1
      (Windows10 do not have this problem.)

      problem:
      Can not load TensorFlow model(Java or scala):

      [hadoop@ip-172-17-1-1 ~]$ spark-shell  --master local[2] --jars s3://jars/jasypt-1.9.2.jar,s3://jars/commons-pool2-2.0.jar,s3://jars/tensorflow-core-api-0.3.1.jar,s3://jars/tensorflow-core-api-0.3.1-linux-x86_64-mkl.jar,s3://jars/ndarray-0.3.1.jar,s3://jars/javacpp-1.5.4.jar,s3://jars/tensorflow-core-platform-0.3.1.jar,s3://jars/protobuf-java-3.8.0.jar
      Spark session available as 'spark'.
      Welcome to
            ____              __
           / __/__  ___ _____/ /__
          _\ \/ _ \/ _ `/ __/  '_/
         /___/ .__/\_,_/_/ /_/\_\   version 2.4.7-amzn-1
            /_/
               
      Using Scala version 2.11.12 (OpenJDK 64-Bit Server VM, Java 1.8.0_282)
      Type in expressions to have them evaluated.
      Type :help for more information.
      
      scala> import org.{tensorflow => tf}
      import org.{tensorflow=>tf}
      
      scala> val bundle = tf.SavedModelBundle.load("/home/hadoop/xDeepFM","serve")
      2021-04-23 07:32:56.223881: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:32] Reading SavedModel from: /home/hadoop/xDeepFM
      2021-04-23 07:32:56.266424: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:55] Reading meta graph with tags { serve }
      2021-04-23 07:32:56.266468: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:93] Reading SavedModel debug info (if present) from: /home/hadoop/xDeepFM
      2021-04-23 07:32:56.346757: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:206] Restoring SavedModel bundle.
      2021-04-23 07:32:56.873838: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:190] Running initialization op on SavedModel bundle at path: /home/hadoop/xDeepFM
      2021-04-23 07:32:56.928656: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:277] SavedModel load for tags { serve }; Status: success: OK. Took 704788 microseconds.
      java.lang.NoSuchMethodError: com.google.protobuf.Parser.parseFrom(Ljava/nio/ByteBuffer;)Ljava/lang/Object;
        at org.tensorflow.proto.framework.MetaGraphDef.parseFrom(MetaGraphDef.java:3067)
        at org.tensorflow.SavedModelBundle.load(SavedModelBundle.java:422)
        at org.tensorflow.SavedModelBundle.access$000(SavedModelBundle.java:59)
        at org.tensorflow.SavedModelBundle$Loader.load(SavedModelBundle.java:68)
        at org.tensorflow.SavedModelBundle.load(SavedModelBundle.java:242)
        ... 49 elided
      
      scala>
      

      But I can load the model in a single 'Testtf.java' like that:

      [hadoop@ip-172-17-1-1 ~]$ vi Testtf.java 
      import org.tensorflow.SavedModelBundle;
      public class Testtf {
          public static void main(String[] args) {
              System.out.println("test load...");
              SavedModelBundle bundle = org.tensorflow.SavedModelBundle.load("/home/hadoop/xDeepFM","serve");
              System.out.println("loaded bundle...");
              System.out.println(bundle);
          }
      }
      
      [hadoop@ip-172-17-1-1 ~]$ javac -cp javacpp-1.5.4-linux-x86_64.jar:tensorflow-core-api-0.3.1-linux-x86_64.jar:tensorflow-core-api-0.3.1.jar:ndarray-0.3.1.jar:javacpp-1.5.4.jar:tensorflow-core-platform-0.3.1.jar:tensorflow-core-platform-mkl-0.3.1.jar:protobuf-java-3.8.0.jar Testtf.java
      [hadoop@ip-172-17-1-1 ~]$ java -cp javacpp-1.5.4-linux-x86_64.jar:tensorflow-core-api-0.3.1-linux-x86_64.jar:tensorflow-core-api-0.3.1.jar:ndarray-0.3.1.jar:javacpp-1.5.4.jar:tensorflow-core-platform-0.3.1.jar:tensorflow-core-platform-mkl-0.3.1.jar:protobuf-java-3.8.0.jar:. Testtf
      test load...
      2021-04-25 02:33:47.247120: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:32] Reading SavedModel from: /home/hadoop/xDeepFM
      2021-04-25 02:33:47.275349: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:55] Reading meta graph with tags { serve }
      2021-04-25 02:33:47.275405: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:93] Reading SavedModel debug info (if present) from: /home/hadoop/xDeepFM
      2021-04-25 02:33:47.275480: I external/org_tensorflow/tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
      To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
      2021-04-25 02:33:47.370208: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:206] Restoring SavedModel bundle.
      2021-04-25 02:33:47.381656: I external/org_tensorflow/tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2300065000 Hz
      2021-04-25 02:33:47.667941: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:190] Running initialization op on SavedModel bundle at path: /home/hadoop/xDeepFM
      2021-04-25 02:33:47.725458: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:277] SavedModel load for tags { serve }; Status: success: OK. Took 478354 microseconds.
      loaded bundle...
      org.tensorflow.SavedModelBundle@80503
      [hadoop@ip-172-17-1-1 ~]$ 

      Thanks!

      refer: tensorflow/java#298

      Attachments

        Activity

          People

            Unassigned Unassigned
            ShuDaoNan ShuDaoNan
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: