Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
3.1.1
-
None
-
None
Description
We have seen quite a good number of customer cases w.r.t fsimage size increased drastically while storing snapshot meta for fileDiff entries. Here is an example fsimage meta storing entire inode info after deleting a file. I'm not sure about any restrictions on why the entire inode meta needs to be stored in fileDiff entry when there is no change w.r.t actual inode meta and it's just a delete file operation.
The fileDiffEntry for the inode 1860467 seems redundant for a simple file delete operation.
<snapshot><id>43</id><root><id>1860465</id><type>DIRECTORY</type><name>s3</name><mtime>1704197935903</mtime><permission>hdfs:supergroup:0755</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></root></snapshot> <snapshot><id>44</id><root><id>1860465</id><type>DIRECTORY</type><name>s4</name><mtime>1704197951829</mtime><permission>hdfs:supergroup:0755</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></root></snapshot> <inode><id>1860467</id><type>FILE</type><name>file1</name><replication>3</replication><mtime>1704197917315</mtime><atime>1704197917031</atime><preferredBlockSize>134217728</preferredBlockSize><permission>hdfs:supergroup:0644</permission><blocks><block><id>1074008442</id><genstamp>267653</genstamp><numBytes>418</numBytes></block> <ref><referredId>1860467</referredId><name>file1</name><dstSnapshotId>0</dstSnapshotId><lastSnapshotId>43</lastSnapshotId></ref> <ref><referredId>1860467</referredId><name></name><dstSnapshotId>2147483646</dstSnapshotId><lastSnapshotId>0</lastSnapshotId></ref> </dirDiffEntry><fileDiffEntry><inodeId>1860467</inodeId><count>1</count><fileDiff><snapshotId>43</snapshotId><size>418</size><name>file1</name><snapshotCopy><replication>3</replication><mtime>1704197917315</mtime><atime>1704197917031</atime><preferredBlockSize>134217728</preferredBlockSize><permission>hdfs:supergroup:0644</permission><storagePolicyId>0</storagePolicyId></snapshotCopy> </fileDiff> </fileDiffEntry>