Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-12971

Incorrect capacity of remote scratch dir when both local and remote scratch dir are configured

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • Backend
    • None

    Description

      The current implementation in AllocateRemoteSpace() (https://github.com/apache/impala/blob/4be5fd8896dcd445a6379bdcda4bdcf318f24511/be/src/runtime/tmp-file-mgr.cc#L1391) has a bug when both local and remote scratch directories are configured.

      Currently, the function utilizes current_bytes_allocated_ to determine whether to allocate space based on the remote directory limit (capacity). However, this approach is flawed when the system allows spilling to other local directories, as current_bytes_allocated_ represents the total used scratch space rather than the usage of the remote directory alone, potentially causing allocation failures when the combined usage (local + remote) exceeds the remote limit.

      This bug happens when both remote and local scratch space are configured, and a limit is set to the remote directory (normally remote capacity is set to unlimited)

      Attachments

        Activity

          People

            baggio000 Yida Wu
            baggio000 Yida Wu
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: