[HDFS-15052] WebHDFS getTrashRoot leads to OOM due to FileSystem object creation - ASF JIRA

Log work

Agile Board

Rank to Top

Rank to Bottom

Attach files

Attach Screenshot

Bulk Copy Attachments

Bulk Move Attachments

Voters

Watch issue

Watchers

Create sub-task

Convert to sub-task

Move

Link

Clone

Labels

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: 2.9.0, 3.0.0-alpha2
Fix Version/s: 3.3.0, 3.1.4, 3.2.2, 2.10.1
Component/s: webhdfs
Labels:
None

Hadoop Flags:

Reviewed

Description

Quoting Daryn Sharp in ~~HDFS-10756~~ :

Surprised nobody has discovered this will lead to an inevitable OOM in the NN. The NN should not be creating filesystems to itself, and must never create filesystems in a remote user's context or the cache will explode.

I guess the problem lies in side NamenodeWebHdfsMethods#getTrashRoot

	  private static String getTrashRoot(String fullPath,
	      Configuration conf) throws IOException {
	    FileSystem fs = FileSystem.get(conf != null ? conf : new Configuration());
	    return fs.getTrashRoot(
	        new org.apache.hadoop.fs.Path(fullPath)).toUri().getPath();
	  }