Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
2.10.2, 3.3.6
-
None
-
None
Description
When upgrading hdfs cluster from 2.10.2 to 3.3.6, the results returned from dfs count command is inconsistent.
Reproduce
Start up 2.10.2 hdfs cluster (1 NN, 2 DN, 1 SNN), execute the following commands
dfs -mkdir /GscWZRxS
dfs -put -f -d /tmp/hpLjvJVW/cl /GscWZRxS/
dfs -put -f -d /tmp/hpLjvJVW/Zjpk /GscWZRxS/cl/lBsmFBlyBd/pozIeNFjzd/PsLbgpR
dfsadmin -clrQuota /GscWZRxS/cl
dfsadmin -refreshSuperUserGroupsConfiguration
dfs -mkdir /GscWZRxS/cl/lBsmFBlyBd/pozIeNFjzd/PsLbgpR/Zjpk/Cf/mGpVA
dfsadmin -refreshCallQueue
dfsadmin -clrQuota /GscWZRxS/cl/lBsmFBlyBd/pozIeNFjzd
dfsadmin -setSpaceQuota 2 -storageType DISK /GscWZRxS/cl/lBsmFBlyBd/pozIeNFjzd/PsLbgpR/Zjpk/Cf
dfsadmin -refreshNodes
dfsadmin -setSpaceQuota 2 -storageType DISK /GscWZRxS/cl/lBsmFBlyBd/pozIeNFjzd
dfsadmin -clrSpaceQuota -storageType ARCHIVE /GscWZRxS/cl
dfsadmin -restoreFailedStorage true
before upgrade, check the quota results
bin/hdfs dfs -count -q -h -u /GscWZRxS/cl/lBsmFBlyBd/pozIeNFjzd/PsLbgpR/Zjpk/Cf none inf none inf /GscWZRxS/cl/lBsmFBlyBd/pozIeNFjzd/PsLbgpR/Zjpk/Cf
Then prepare the upgrade. Enter safemode, create image, shutdown the cluster and start up the new cluster
bin/hdfs dfs -count -q -h -u /GscWZRxS/cl/lBsmFBlyBd/pozIeNFjzd/PsLbgpR/Zjpk/Cf 8.0 E 8.0 E none inf /GscWZRxS/cl/lBsmFBlyBd/pozIeNFjzd/PsLbgpR/Zjpk/Cf
The values of the first two columns are inconsistent with the quota I set before.
I have attached the file used by the command. I am digging out the root cause, I'll try to submit a patch once I can fix it. Any help is appreciated!