Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-15383

RBF: Disable watch in ZKDelegationSecretManager for performance

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersStop watchingWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.4.0, 3.3.6
    • 3.4.0, 3.3.6
    • rbf
    • Reviewed

    Description

      Based on the current design for delegation token in secure Router, the total number of watches for tokens is the product of number of routers and number of tokens, this is due to ZKDelegationTokenManager is using PathChildrenCache from curator, which automatically sets the watch and ZK will push the sync information to each router. There are some evaluations about the number of watches in Zookeeper has negative performance impact to Zookeeper server.

      In our practice when the number of watches exceeds 1.2 Million in a single ZK server there will be significant ZK performance degradation. Thus this ticket is to rewrite ZKDelegationTokenManagerImpl.java to explicitly disable the PathChildrenCache and have Routers sync periodically from Zookeeper. This has been working fine at the scale of 10 Routers with 2 million tokens. 

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            fengnanli Fengnan Li Assign to me
            fengnanli Fengnan Li
            Votes:
            0 Vote for this issue
            Watchers:
            13 Stop watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment