Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Incomplete
-
1.6.3, 2.1.0, 2.2.0
-
None
-
- Hadoop distributions: HDP 2.5 - 2.6.3.0
- OS: Seen on SLES12, RHEL 7.3 & RHEL 7.4
Description
Spark Thrift is not cleaning up /tmp for files & directories named like:
/tmp/hive/*.pipeout
/tmp/*_resources
There are such a large number that /tmp quickly runs out of inodes causing the partition to be unusable and many services to crash. This is even true when the only jobs submitted are routine service checks.
Used `strace` to show that Spark Thrift is responsible:
strace.out.118864:04:53:49 open("/tmp/hive/55ad7fc1-f79a-4ad8-8e02-26bbeaa86bbc7288010135864174970.pipeout", O_RDWR|O_CREAT|O_EXCL, 0666) = 134 strace.out.118864:04:53:49 mkdir("/tmp/b6dfbf9e-2f7c-4c25-95a1-73c44318ecf4_resources", 0777) = 0
Those files were left behind, even days later.
Example files:
# stat /tmp/hive/55ad7fc1-f79a-4ad8-8e02-26bbeaa86bbc7288010135864174970.pipeout File: ‘/tmp/hive/55ad7fc1-f79a-4ad8-8e02-26bbeaa86bbc7288010135864174970.pipeout’ Size: 0 Blocks: 0 IO Block: 4096 regular empty file Device: fe09h/65033d Inode: 678 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 1000/ hive) Gid: ( 1002/ hadoop) Access: 2017-12-19 04:53:49.126777260 -0600 Modify: 2017-12-19 04:53:49.126777260 -0600 Change: 2017-12-19 04:53:49.126777260 -0600 Birth: - # stat /tmp/b6dfbf9e-2f7c-4c25-95a1-73c44318ecf4_resources File: ‘/tmp/b6dfbf9e-2f7c-4c25-95a1-73c44318ecf4_resources’ Size: 4096 Blocks: 8 IO Block: 4096 directory Device: fe09h/65033d Inode: 668 Links: 2 Access: (0700/drwx------) Uid: ( 1000/ hive) Gid: ( 1002/ hadoop) Access: 2017-12-19 04:57:38.458937635 -0600 Modify: 2017-12-19 04:53:49.062777216 -0600 Change: 2017-12-19 04:53:49.066777218 -0600 Birth: -
Showing the large number:
# find /tmp/ -name '*_resources' | wc -l 68340 # find /tmp/hive -name "*.pipeout" | wc -l 51837
Attachments
Issue Links
- is related to
-
SPARK-15401 Spark Thrift server creates empty directories in tmp directory on the driver
- Resolved
-
HIVE-6091 Empty pipeout files are created for connection create/close
- Closed