Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-22041

HiveServer2 creates Delegation Token crc files and never deletes them

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 1.1.0
    • Fix Version/s: None
    • Component/s: HiveServer2
    • Labels:
      None

      Description

      It seems that in secure clusters, HiveServer2 creates an unbounded amount of crc files related to the Delegation Token, e.g.:

      # ls -latr .hive_hadoop_delegation_token*.tmp.crc | more
      rw-rr- 1 hive hive 12 Jul 22 18:41 .hive_hadoop_delegation_token7336761305415229297.tmp.crc
      rw-rr- 1 hive hive 12 Jul 22 18:41 .hive_hadoop_delegation_token2098027444797756507.tmp.crc
      rw-rr- 1 hive hive 12 Jul 22 18:41 .hive_hadoop_delegation_token570615377517838929.tmp.crc
      rw-rr- 1 hive hive 12 Jul 22 18:41 .hive_hadoop_delegation_token2806157469169711507.tmp.crc
      rw-rr- 1 hive hive 12 Jul 22 18:41 .hive_hadoop_delegation_token4743780849236152782.tmp.crc
      rw-rr- 1 hive hive 12 Jul 22 18:41 .hive_hadoop_delegation_token4779276962989484605.tmp.crc
      rw-rr- 1 hive hive 12 Jul 22 18:41 .hive_hadoop_delegation_token614278562135964419.tmp.crc
      rw-rr- 1 hive hive 12 Jul 22 18:41 .hive_hadoop_delegation_token5163699580858054526.tmp.crc
      rw-rr- 1 hive hive 12 Jul 22 18:41 .hive_hadoop_delegation_token6764054090932425303.tmp.crc
      rw-rr- 1 hive hive 12 Jul 22 18:42 .hive_hadoop_delegation_token6389909203128084522.tmp.crc
      rw-rr- 1 hive hive 12 Jul 22 18:42 .hive_hadoop_delegation_token7810958545651446366.tmp.crc
      [...]

      this could quickly fill the inode table of the filesystem (usually the one mounted on /tmp).

      I've experienced it in Hive 1.1.0 (CDH 5.16.2), but I think the same issue is present in the current master branch, since here:

      https://github.com/apache/hive/blob/d1343a69e6be7a312b7c0bb2aeebefaa40535b65/ql/src/java/org/apache/hadoop/hive/ql/exec/SecureCmdDoAs.java#L71

      the crc file gets created, but only the token file gets deleted:

      https://github.com/apache/hive/blob/d1343a69e6be7a312b7c0bb2aeebefaa40535b65/ql/src/java/org/apache/hadoop/hive/ql/exec/SecureCmdDoAs.java#L80

      HIVE-13883 greatly relates to the issue; I think that the same logic could be used here to fix this, although I wonder if the Credentials class should also expose some kind of delete method that cleans every file it has produced...

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              alessandrodd Alessandro Di Diego
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: