Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25681

Delegation Tokens fetched twice upon HadoopFSDelegationTokenProvider creation

    XMLWordPrintableJSON

Details

    Description

      Looking for a refactor to HadoopFSDelegationTokenProvider. Within the function obtainDelegationTokens():

      This code-block:

      val fetchCreds = fetchDelegationTokens(getTokenRenewer(hadoopConf),...)
          // Get the token renewal interval if it is not set. It will only be called once.
          if (tokenRenewalInterval == null) {
            tokenRenewalInterval = getTokenRenewalInterval(...)
      }

       calls fetchDelegationTokens() twice since the tokenRenewalInterval will always be null upon creation of the TokenManager which I think is unnecessary in the case of Kubernetes (as you are creating 2 DTs when only one is needed.) Could this possibly be refactored to only call fetchDelegationTokens() once upon startup or to have a param to specify tokenRenewalInterval

      Attachments

        Activity

          People

            Unassigned Unassigned
            ifilonenko Ilan Filonenko
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: