Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-2246

Shortcut a local client reads to a Datanodes files directly

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.23.1, 1.0.0, 0.22.1
    • None
    • None
    • Reviewed
    • Hide
      1. New configurations
      a. dfs.block.local-path-access.user is the key in datanode configuration to specify the user allowed to do short circuit read.
      b. dfs.client.read.shortcircuit is the key to enable short circuit read at the client side configuration.
      c. dfs.client.read.shortcircuit.skip.checksum is the key to bypass checksum check at the client side.
      2. By default none of the above are enabled and short circuit read will not kick in.
      3. If security is on, the feature can be used only for user that has kerberos credentials at the client, therefore map reduce tasks cannot benefit from it in general.
      Show
      1. New configurations a. dfs.block.local-path-access.user is the key in datanode configuration to specify the user allowed to do short circuit read. b. dfs.client.read.shortcircuit is the key to enable short circuit read at the client side configuration. c. dfs.client.read.shortcircuit.skip.checksum is the key to bypass checksum check at the client side. 2. By default none of the above are enabled and short circuit read will not kick in. 3. If security is on, the feature can be used only for user that has kerberos credentials at the client, therefore map reduce tasks cannot benefit from it in general.

    Attachments

      1. HDFS-2246-22.patch
        118 kB
        Benoy Antony
      2. HDFS-2246-branch-0.20-security.patch
        65 kB
        Jitendra Nath Pandey
      3. HDFS-2246-trunk.patch
        69 kB
        Jitendra Nath Pandey
      4. HDFS-2246-trunk.patch
        69 kB
        Jitendra Nath Pandey
      5. HDFS-2246-trunk.patch
        68 kB
        Jitendra Nath Pandey
      6. HDFS-2246-trunk.patch
        68 kB
        Jitendra Nath Pandey
      7. HDFS-2246-branch-0.20-security.patch
        63 kB
        Jitendra Nath Pandey
      8. TestShortCircuitLocalRead.java
        10 kB
        Todd Lipcon
      9. HDFS-2246-branch-0.20-security-205.2.patch
        61 kB
        Jitendra Nath Pandey
      10. HDFS-2246-branch-0.20-security.patch
        61 kB
        Jitendra Nath Pandey
      11. HDFS-2246-trunk.patch
        60 kB
        Jitendra Nath Pandey
      12. HDFS-2246-trunk.patch
        60 kB
        Jitendra Nath Pandey
      13. HDFS-2246-branch-0.20-security-205.1.patch
        60 kB
        Jitendra Nath Pandey
      14. HDFS-2246-branch-0.20-security-205.patch
        62 kB
        Jitendra Nath Pandey
      15. HDFS-2246-branch-0.20-security.no-softref.patch
        62 kB
        Jitendra Nath Pandey
      16. HDFS-2246-branch-0.20-security.3.patch
        65 kB
        Jitendra Nath Pandey
      17. HDFS-2246-branch-0.20-security-205.patch
        59 kB
        Jitendra Nath Pandey
      18. HDFS-2246-branch-0.20-security-205.patch
        208 kB
        Jitendra Nath Pandey
      19. HDFS-2246-branch-0.20-security.patch
        59 kB
        Jitendra Nath Pandey
      20. HDFS-2246.20s.4.txt
        55 kB
        Jitendra Nath Pandey
      21. HDFS-2246.20s.3.txt
        53 kB
        Suresh Srinivas
      22. HDFS-2246.20s.2.txt
        53 kB
        Suresh Srinivas
      23. HDFS-2246.20s.1.patch
        48 kB
        Suresh Srinivas
      24. HDFS-2246.20s.patch
        46 kB
        Suresh Srinivas
      25. localReadShortcut20-security.2patch
        47 kB
        Sanjay Radia
      26. 0001-HDFS-347.-Local-reads.patch
        48 kB
        Andrew Kyle Purtell

      Issue Links

        duplicates

        Improvement - An improvement or enhancement to an existing feature or task. HDFS-281 Explore usage of the sendfile api via java.nio.channels.FileChannel.transfer{To|From} for i/o in datanodes

        • Major - Major loss of function.
        • Resolved
        is related to

        Improvement - An improvement or enhancement to an existing feature or task. HADOOP-7804 enable hadoop config generator to set dfs.block.local-path-access.user to enable short circuit read

        • Major - Major loss of function.
        • Closed

        Improvement - An improvement or enhancement to an existing feature or task. HDFS-3907 Allow multiple users for local block readers

        • Major - Major loss of function.
        • Closed
        relates to

        New Feature - A new feature of the product, which has yet to be developed. HADOOP-6311 Add support for unix domain sockets to JNI libs

        • Major - Major loss of function.
        • Resolved

        New Feature - A new feature of the product, which has yet to be developed. HDFS-245 Create symbolic links in HDFS

        • Major - Major loss of function.
        • Closed

        Improvement - An improvement or enhancement to an existing feature or task. HDFS-347 DFS read performance suboptimal when client co-located on nodes with data

        • Major - Major loss of function.
        • Closed

        Improvement - An improvement or enhancement to an existing feature or task. HDFS-2654 Make BlockReaderLocal not extend RemoteBlockReader2

        • Major - Major loss of function.
        • Closed

        Activity

          People

            jnp Jitendra Nath Pandey
            sanjay.radia Sanjay Radia
            Votes:
            0 Vote for this issue
            Watchers:
            44 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: