[HADOOP-18222] Prevent DelegationTokenSecretManagerMetrics from registering multiple times - ASF JIRA

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: 3.4.0, 3.3.5
Fix Version/s: 3.4.0, 3.3.5
Component/s: common
Labels:
- pull-request-available

Target Version/s:

3.4.0, 3.3.5
Hadoop Flags:

Reviewed

Description

After committing ~~HADOOP-18167~~, we received reports of the following error when ResourceManager is initialized:

Caused by: java.io.IOException: Problem starting http server
        at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:1389)
        at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:475)
        ... 4 more
Caused by: org.apache.hadoop.metrics2.MetricsException: Metrics source DelegationTokenSecretManagerMetrics already exists!
        at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:152)
        at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:125)
        at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:229)
        at org.apache.hadoop.metrics2.MetricsSystem.register(MetricsSystem.java:71)
        at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager$DelegationTokenSecretManagerMetrics.create(AbstractDelegationTokenSecretManager.java:878)
        at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.<init>(AbstractDelegationTokenSecretManager.java:152)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenManager$DelegationTokenSecretManager.<init>(DelegationTokenManager.java:72)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenManager.<init>(DelegationTokenManager.java:122)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.initTokenManager(DelegationTokenAuthenticationHandler.java:161)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.init(DelegationTokenAuthenticationHandler.java:130)
        at org.apache.hadoop.security.authentication.server.AuthenticationFilter.initializeAuthHandler(AuthenticationFilter.java:194)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.initializeAuthHandler(DelegationTokenAuthenticationFilter.java:214)
        at org.apache.hadoop.security.authentication.server.AuthenticationFilter.init(AuthenticationFilter.java:180)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.init(DelegationTokenAuthenticationFilter.java:180)
        at org.apache.hadoop.yarn.server.security.http.RMAuthenticationFilter.init(RMAuthenticationFilter.java:53)

This can happen if MetricsSystemImpl#init is called and multiple metrics are registered with the same name. A proposed solution is to declare the metrics in AbstractDelegationTokenSecretManager as singleton, which would prevent multiple instances DelegationTokenSecretManagerMetrics from being registered.

Attachments

Issue Links

Add Link

Dependent

YARN-11122 Support getClusterNodes API in FederationClientInterceptor

Resolved

Delete this link

YARN-11130 RouterClientRMService Has Unused import

Resolved

Delete this link

YARN-11123 ResourceManager webapps test failures due to org.apache.hadoop.metrics2.MetricsException and subsequent java.net.BindException: Address already in use