Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-17379

setQuota/setSpaceQuota command doesn't fail if the directory would immediately exceed the new quota

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Minor
    • Resolution: Unresolved
    • 3.1.3
    • None
    • hdfs
    • None

    Description

      With regards to documentation - https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsQuotaAdminGuide.html#Space_Quotas

      • hdfs dfsadmin -setSpaceQuota <N> <directory>...<directory>

      Set the space quota to be N bytes for each directory. This is a hard limit on total size of all the files under the directory tree. The space quota takes replication also into account, i.e. one GB of data with replication of 3 consumes 3GB of quota. N can also be specified with a binary prefix for convenience, for e.g. 50g for 50 gigabytes and 2t for 2 terabytes etc. Best effort for each directory, with faults reported if N is neither zero nor a positive integer, the directory does not exist or it is a file, or the directory would immediately exceed the new quota.

      So, command to set Quota have to fail if quota already exceeded

       

      Actual result – command finished with exit code 0

      Steps to reproduce:

      -          put file 1GB into hdfs folder

      -          try to set quota of 512MB

      hdfs dfs -ls  /hdfs_dir
      Exit code: 0
      Found 1 items

      •   3 user hadoop 1073741824 2024-01-17 18:57 /hdfs_dir/File

       

      hdfs dfsadmin -setSpaceQuota 512m /hdfs_dir

      result:

      Exit code: 0

      Quota report shows negative value for SPACE_REM_QUOTA

       

      REM_SPACE_QUOTA   -2.5GB
      QUOTA       REM_QUOTA     SPACE_QUOTA REM_SPACE_QUOTA    DIR_COUNT   FILE_COUNT       CONTENT_SIZE PATHNAME
              none             inf           512 M          -2.5 G            1            1                1 G /hdfs_dir
       
      

       

      The same situation with setQuota method, if we try to set name quota, for example, 3 for folder already contained 5 files – it doesn’t fail and shows negative value for REM_QUOTA

      The reason for this behavior is absent quota check during SET operations

      Attachments

        Activity

          People

            Unassigned Unassigned
            smatytsin Stanislav Matytsin
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: