[HIVE-27317] Temporary (local) session files cleanup improvements - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Improvement
Status: Closed
Priority: Major
Resolution: Fixed
Affects Version/s: 3.1.3
Fix Version/s: 4.0.0-beta-1
Component/s: None
Labels:
- pull-request-available

Description

When Hive session is killed, no chance for shutdown hook to clean-up tmp files.

There is a Hive service to clean residual files https://issues.apache.org/jira/browse/HIVE-13429, and later on its execution is scheduled inside HS2 https://issues.apache.org/jira/browse/HIVE-15068 to make sure not to leave any temp file behind. But this service cleans up only HDFS temp files, there are still residual files/dirs in HiveConf.ConfVars.LOCALSCRATCHDIR location as follows;

> ll /tmp/user/97c4ef50-5e80-480e-a6f0-4f779050852b*
drwx------ 2 user user 4096 Oct 29 10:09 97c4ef50-5e80-480e-a6f0-4f779050852b
-rw------- 1 user user    0 Oct 29 10:09 97c4ef50-5e80-480e-a6f0-4f779050852b10571819313894728966.pipeout
-rw------- 1 user user    0 Oct 29 10:09 97c4ef50-5e80-480e-a6f0-4f779050852b16013956055489853961.pipeout
-rw------- 1 user user    0 Oct 29 10:09 97c4ef50-5e80-480e-a6f0-4f779050852b4383913570068173450.pipeout
-rw------- 1 user user    0 Oct 29 10:09 97c4ef50-5e80-480e-a6f0-4f779050852b889740171428672108.pipeout

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending

HIVE-27317.patch
04/May/23 13:34
6 kB
Sercan Tekin

Issue Links

relates to

HIVE-13429 Tool to remove dangling scratch dir

Closed

HIVE-15068 Run ClearDanglingScratchDir periodically inside HS2

Closed

links to

GitHub Pull Request #4293

GitHub Pull Request #4399

GitHub Pull Request #4403

Activity

People

Assignee:: Sercan Tekin

Reporter:: Sercan Tekin

Votes:: 0 Vote for this issue

Watchers:: 4 Start watching this issue

Dates

Created:: 04/May/23 13:08

Updated:: 15/Aug/23 06:33

Resolved:: 22/Jun/23 08:40

Time Tracking

Estimated:

Not Specified

Remaining:

Logged:

1h 10m