Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
-
ghx-label-4
Description
The current implementation in AllocateRemoteSpace() (https://github.com/apache/impala/blob/4be5fd8896dcd445a6379bdcda4bdcf318f24511/be/src/runtime/tmp-file-mgr.cc#L1391) has a bug when both local and remote scratch directories are configured.
Currently, the function utilizes current_bytes_allocated_ to determine whether to allocate space based on the remote directory limit (capacity). However, this approach is flawed when the system allows spilling to other local directories, as current_bytes_allocated_ represents the total used scratch space rather than the usage of the remote directory alone, potentially causing allocation failures when the combined usage (local + remote) exceeds the remote limit.
This bug happens when both remote and local scratch space are configured, and a limit is set to the remote directory (normally remote capacity is set to unlimited)