Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-32418

ClassNotFoundException when using flink-protobuf with sql-client

    XMLWordPrintableJSON

Details

    Description

      When the protobuf format in the kafka connector is used via the sql-client it is not able to load the generated protobuf classes which are either passed via `-j /protobuf-classes.jar` or added in the script via ADD JAR '/protobuf-classes.jar'. The SHOW JARS command prints that the jar is loaded but when the protobuf classes are loaded a ClassNotFoundException occurs.

      executed command:

      sql-client.sh -f protobuf-table.sql -j /protobuf-classes.jar
      

      protobuf-table.sql

      ADD JAR '/opt/sql-client/lib/flink-sql-connector-kafka-1.16.2.jar';
      ADD JAR '/opt/sql-client/lib/flink-protobuf-1.16.2.jar';
      
      SHOW JARS;
      
      CREATE TABLE POSITIONS(id BIGINT) WITH (
            'connector' = 'kafka',
            'format' = 'protobuf',
            'topic' = 'protbuf-topic',
            'properties.bootstrap.servers' = 'kafka:9092',
            'properties.group.id' = 'flink-protobuf',
            'properties.security.protocol' = 'SASL_PLAINTEXT',
            'properties.sasl.mechanism' = 'SCRAM-SHA-512',
            'properties.sasl.jaas.config' = 'org.apache.kafka.common.security.scram.ScramLoginModule required username="user" password="****";',
            'scan.startup.mode' = 'earliest-offset',
            'protobuf.message-class-name' = 'com.example.protobuf.ProtoMessage',
            'protobuf.ignore-parse-errors' = 'true'
            );
      
      SELECT * FROM POSITIONS;
      

      exception in the log:

      Caused by: java.lang.ClassNotFoundException: com.example.protobuf.ProtoMessage
              at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source)
              at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown Source)
              at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
              at java.base/java.lang.Class.forName0(Native Method)
              at java.base/java.lang.Class.forName(Unknown Source)
              at org.apache.flink.formats.protobuf.util.PbFormatUtils.getDescriptor(PbFormatUtils.java:89)
              ... 36 more
      

      This also seems somehow related to FLINK-30318

      Attachments

        1. full-stacktrace.log
          14 kB
          Michael Kreis

        Issue Links

          Activity

            People

              m-kay Michael Kreis
              m-kay Michael Kreis
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated: