Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-6426

Compress ZK YARN keys to scale up (especially AppStateData

Add voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Patch Available
    • Major
    • Resolution: Unresolved
    • 3.0.0-alpha2
    • None
    • yarn

    Description

      ZK today stores the protobuf files uncompressed. This is not an issue except that if a customer job has thousands of files, AppStateData will store the user context as a string with multiple URLs and it is easy to get to 1MB or more.

      This can put unnecessary strain on ZK and make the process slow.

      The proposal is to simply compress protobufs before sending them to ZK

      Attachments

        1. zkcompression.patch
          8 kB
          Roni Burd

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            roniburd Roni Burd
            roniburd Roni Burd
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:

              Slack

                Issue deployment