Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-8374 Add high water mark metrics and limits for spill disk usage
  3. IMPALA-8376

Add per-directory limits for scratch disk usage

Attach filesAttach ScreenshotVotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • ghx-label-4

    Description

      The current syntax is:

      --scratch_dirs=/data/1/impala/impalad,/data/10/impala/impalad,/data/11/impala/impalad,/data/2/impala/impalad,/data/3/impala/impalad,/data/4/impala/impalad,/data/5/impala/impalad,/data/6/impala/impalad,/data/7/impala/impalad,/data/8/impala/impalad,/data/9/impala/impalad,/data/12/impala/impalad

      The current syntax for the data cache is

      --data_cache_dir=/tmp --data_cache_size=500MB

      One idea is to allow optionally specifying the limit after each directory:

      --scratch_dirs=/data/1/impala/impalad:500MB,/data/10/impala/impalad:2GB,/data/11/impala/impalad

      Attachments

        Issue Links

        breaks

        Bug - A problem which impairs or prevents the functions of the product. IMPALA-8850 TmpFileMgrTest.TestDirectoryLimitParsing failed in asf-master build with error "Value of: dirs3.size() Actual: 3 Expected: 4"

        • Blocker - Blocks development and/or testing work, production could not run
        • Resolved

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            tarmstrong Tim Armstrong
            tarmstrong Tim Armstrong
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment