Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-12943 Consistent Reads from Standby Node
  3. HDFS-13923

Add a configuration to turn on/off observer reads

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • None

    Description

      In some situations having a config to turn on/off observer reads dynamically may be useful. For instance, some applications may be sensitive for data freshness and want to always reach directly to active NN. In a more complex scenario, services such as Presto may want to apply observer reads for different types of queries. In this case, simply change dfs.client.failover.proxy.provider.<nameservice> may not be enough, since with FileSystem cache (which is usually turned on) will ignore the change and still use the same FileSystem object.

      Here I'm proposing to add a flag in HdfsClientConfigKeys, such as dfs.client.observer.reads.enabled, that can be used to dynamically turn on/off observer reads. The FileSystem cache key should also take account of this flag in its hashCode and equals impl, so that different FileSystem objects will be used depending on the flag.

       

      cc shv, xkrogenvagarychen, zero45 for discussion.

      Attachments

        Issue Links

          Activity

            People

              shwetayakkali Shweta
              csun Chao Sun
              Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

                Created:
                Updated:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 40m
                  40m