Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-2790

NM can't aggregate logs past HDFS delegation token expiry.

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 2.6.0
    • 2.6.0
    • nodemanager
    • None
    • Reviewed

    Description

      We shorten hdfs delegation token lifetime, set RM to act as as a proxy user in order to be able to renew the token on behalf of a user submitting the application. Still we see NM log aggregation fail due to token expiry error.

      2014-10-31 00:11:56,579 INFO  logaggregation.AppLogAggregatorImpl (AppLogAggregatorImpl.java:finishLogAggregation(481)) - Application just finished : application_1414705229376_0004
2014-10-31 00:11:56,588 WARN  ipc.Client (Client.java:run(675)) - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (HDFS_DELEGATION_TOKEN token 75 for user1) is expired
2014-10-31 00:11:56,589 ERROR logaggregation.AppLogAggregatorImpl (AppLogAggregatorImpl.java:uploadLogsForContainers(233)) - Cannot create writer for app application_1414705229376_0004. Skip log upload this time.

      Attachments

        1. YARN-2790.1.patch
          16 kB
          Jian He

        Issue Links

          is broken by

          Sub-task - The sub-task of the issue YARN-2704 Localization and log-aggregation will fail if hdfs delegation token expired after token-max-life-time

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Activity

            People

              jianhe Jian He
              tassapola Tassapol Athiapinya
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: