Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-12357

Let NameNode to bypass external attribute provider for special user

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.9.0, 3.0.0-beta1
    • None
    • None
    • Reviewed

    Description

      This is a third proposal to solve the problem described in HDFS-12202.

      The problem is, when we do distcp from one cluster to another (or within the same cluster), in addition to copying file data, we copy the metadata from source to target. If external attribute provider is enabled, the metadata may be read from the provider, thus provider data read from source may be saved to target HDFS.

      We want to avoid saving metadata from external provider to HDFS, so we want to bypass external provider when doing the distcp (or hadoop fs -cp) operation.

      Two alternative approaches were proposed earlier, one in HDFS-12202, the other in HDFS-12294. The proposal here is the third one.

      The idea is, we introduce a new config, that specifies a special user (or a list of users), and let NN bypass external provider when the current user is a special user.

      If we run applications as the special user that need data from external attribute provider, then it won't work. So the constraint on this approach is, the special users here should not run applications that need data from external provider.

      Thanks Arun Suresh for proposing this idea and [~chris.douglas], Daryn Sharp, Manoj Govindassamy for the discussions in the other jiras.

      I'm creating this one to discuss further.

      Attachments

        1. HDFS-12357.001.patch
          9 kB
          Yongjun Zhang
        2. HDFS-12357.002.patch
          13 kB
          Christopher Douglas
        3. HDFS-12357.003.patch
          12 kB
          Christopher Douglas
        4. HDFS-12357.004.patch
          12 kB
          Christopher Douglas
        5. HDFS-12357.005.patch
          13 kB
          Yongjun Zhang
        6. HDFS-12357.001a.patch
          10 kB
          Yongjun Zhang
        7. HDFS-12357.001b.patch
          10 kB
          Yongjun Zhang
        8. HDFS-12357.006.patch
          10 kB
          Yongjun Zhang
        9. HDFS-12357.007.patch
          14 kB
          Yongjun Zhang

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            yzhangal Yongjun Zhang
            yzhangal Yongjun Zhang
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment