Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-29797

can't run a job on yarn, if set fs.default-scheme

    XMLWordPrintableJSON

Details

    Description

      If I set the value of fs.default-scheme to other scheme and run a job on yarn.

      I got this exception:

      Caused by: java.io.FileNotFoundException: File does not exist: /tmp/application_1667097340114_0001-flink-conf.yaml9110057305807570477.tmp
        at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1128)
        at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1120)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1120)
        at org.apache.flink.yarn.YarnApplicationFileUploader.registerSingleLocalResource(YarnApplicationFileUploader.java:168)
        at org.apache.flink.yarn.YarnClusterDescriptor.startAppMaster(YarnClusterDescriptor.java:1047)
        at org.apache.flink.yarn.YarnClusterDescriptor.deployInternal(YarnClusterDescriptor.java:623)
        at org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:490)
        ... 24 more

      I think the cause of this problem may be 

      tmpConfigurationFile.getAbsolutePath()

       

      Attachments

        1. issue.png
          37 kB
          wangmenglong

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. FLINK-33424 Resolved an issue in YarnClusterDescriptor where temporary files created locally by flink-conf.yaml are treated as remote files

          • Major - Major loss of function.
          • Closed
          fixes

          Bug - A problem which impairs or prevents the functions of the product. FLINK-33424 Resolved an issue in YarnClusterDescriptor where temporary files created locally by flink-conf.yaml are treated as remote files

          • Major - Major loss of function.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. FLINK-33472 Solve the problem that the temporary file of flink-conf.yaml in S3AFileSystem cannot be uploaded

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open
          links to

          Web Link GitHub Pull Request #24733

          Activity

            People

              Unassigned Unassigned
              Melon Wang wangmenglong
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated: