Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-31875

OSS throwns NoClassDefFoundError due to old hadoop-common version

    XMLWordPrintableJSON

Details

    Description

      Problem

      When using OSS in 1.17, an exception will be thrown:

      java.lang.NoClassDefFoundError: org/apache/hadoop/thirdparty/com/google/common/base/Preconditions

at org.apache.hadoop.fs.aliyun.oss.AliyunOSSUtils.longOption(AliyunOSSUtils.java:221)
at org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem.initialize(AliyunOSSFileSystem.java:343)
at org.apache.flink.fs.osshadoop.OSSFileSystemFactory.create(OSSFileSystemFactory.java:147)
at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:508)
at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:409)
at org.apache.flink.core.fs.Path.getFileSystem(Path.java:274)

       It could be reproduced in ITCASE of OSS if some envs has been configured.

      Why

      After https://issues.apache.org/jira/browse/FLINK-27308 and  https://issues.apache.org/jira/browse/FLINK-29502 ，hadoop-aliyun has also be upgraded to 3.3.4 which relys on the newest version of hadoop-common.

      OSS still uses the old version (2.10.2) extended from flink-parent so that some classes cannot be found.

      Attachments

        Issue Links

          links to

          Web Link GitHub Pull Request #22439

          Activity

            People

              masteryhx Hangxiang Yu
              masteryhx Hangxiang Yu
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated: