Uploaded image for project: 'Zeppelin'
  1. Zeppelin
  2. ZEPPELIN-5594

HDFS file id should be read as "long", not "int".

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 0.10.0
    • 0.11.0
    • Interpreters
    • None

    Description

       

      ERROR [2021-11-29 14:46:51,519] ({FIFOScheduler-org.apache.zeppelin.file.FileInterpreter1871691756-Worker-1} HDFSFileInterpreter.java[listAll]:249) - listall: listDir /user/*******
      com.google.gson.JsonSyntaxException: java.lang.NumberFormatException: Expected an int but was 4947954640 at line 1 column 585 path $.FileStatuses.FileStatus[2].fileId 

      I found this error when using file(hdfs) interpreter.

       

      Some paths are okay.

      I met this exception when trying to access paths with high file id (above integer).

      I think that "blockSize" & "fileId" should be declared as "long"

       

      Refer to HdfsFileStatus.java,

       

        class Builder {
          // Changing default values will affect cases where values are not
          // specified. Be careful!
          private long length                    = 0L;
          private boolean isdir                  = false;
          private int replication                = 0;
          private long blocksize                 = 0L;
          private long mtime                     = 0L;
          private long atime                     = 0L;
          private FsPermission permission        = null;
          private EnumSet<Flags> flags           = EnumSet.noneOf(Flags.class);
          private String owner                   = null;
          private String group                   = null;
          private byte[] symlink                 = null;
          private byte[] path                    = EMPTY_NAME;
          private long fileId                    = -1L;
          private int childrenNum                = 0;
          private FileEncryptionInfo feInfo      = null;
          private byte storagePolicy             =
              HdfsConstants.BLOCK_STORAGE_POLICY_ID_UNSPECIFIED;
          private ErasureCodingPolicy ecPolicy   = null;
          private LocatedBlocks locations        = null;
      ...
      
      

       

       

      Attachments

        Issue Links

          Activity

            People

              eub YUBI LEE
              eub YUBI LEE
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h
                  1h