Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-3211

Add a cluster supported features request

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 1.13.0
    • None
    • master, supportability
    • None

    Description

      Recently we have come across a few scenarios where it would be useful to make decisions in client integrations (Backup/Restore, Spark, NiFi, Impala) based on the supported features of the target Kudu cluster. This can especially helpful when we want to use new features by default if available but using the new feature requires client/integration logic changes.

      Some recent examples:

      • Push bloomfilter predicates only if supported
      • Use insert ignore operations (vs session based ignore) only if supported

      It is technically possible to be optimistic about the support of a feature and try to handle errors in a clever way using the required feature capabilities of the RPCs. However, that can be difficult to express and near impossible if you want to make a decision for multiple requests or based on what all tablet servers support instead of based on a single request to a single tablet server.

      Additionally now that we support rolling restart, we can't assume that because a single master or tablet server supports a feature that all servers in the cluster support the feature.

      Some thoughts on the feature/implementation:

      • This should be a master request in order to prevent needing to talk to all the tablet servers.
      • We could leverage server registration requests or heartbeats to aggregate the current state on the leader master.
      • We could represent these features as "cluster" level features and indicate that some (union) or all (intersect) of the servers support a given feature.
      • If this request/response is not available in a cluster the response would indicate that feature support is unknown and the user can decide how to proceed.
      • If we want to support disabling features via runtime flags we will need to ensure we update the master, maybe via heartbeat, with changed support for a running server.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              granthenke Grant Henke
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated: