Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-24111

Enable CompactionTool executions on non-HDFS filesystems

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      CompactionTool on non-HDFS filesystems does not work because the used filesystems are mixed up. To collect the StoreFiles the Filesystem.get(conf) method is used instead of relying on the root dir filesystem.

      With YARN the delegation tokens are not handled correctly with a different filesystem and the mappers only get delegation token for the staging directory's filesystem.

      Another issue is in MapReduce mode when YARN is used. In this case the TMP directory (hbase.tmp.dir) is created under yarn.nodemanager.local-dirs but this directory is created (even on HDFS a regular user can't create this directory) on the Storefile's filesystem where a local filesystem temp directory shouldn't be used.

      In this ticket I plan to clean up the used filesystems, remove the custom temp directory (use the regions' .tmp directory) and collect delegation tokens for staging dir + store dir paths' FileSystems.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            psomogyi Peter Somogyi
            psomogyi Peter Somogyi
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment