Uploaded image for project: 'Ranger'
  1. Ranger
  2. RANGER-3923

Dataset policies

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: In Progress
    • Major
    • Resolution: Unresolved
    • None
    • None
    • Ranger
    • None

    Description

      Given the primary business value of Apache Ranger is to enable sharing of resources, it will help if Apache Ranger provides an abstraction that enables a set of resources/data across services, a dataset, to be the unit of sharing instead of one or more resources in each service. This has several benefits, like:

      1. A single policy to manage access to data in multiple services - like HBase, Hive, Snowflake, Kafka, Google BigQuery, AWS S3, AWS Redshift, ADLS-Gen2. This enables authorization to be centered around a purpose, like:
        • Marketing Campaign 2022 dataset
        • Sales 2021 dataset
        • CA Claims 2021 dataset
      2. Enables different set of users to manage sharing data into a dataset and manage access to the data in a dataset:
        • Data owners share data into a dataset, with necessary masking,  row-filters and schedules; they can update the share details, including stop sharing into a dataset.
        • Dataset admins manage who has access to the data in the dataset. This relieves data owners from having to micromanage access to the shared data, for example when a user needs access to the data in multiple services to participate in a project.

      Attached document has more details on this new abstraction, including a number of questions & answers that to help understand various aspects of this feature. Please read and add your comments/suggestions.

      Attachments

        1. Apache Ranger - Dataset-1.pdf
          329 kB
          Madhan Neethiraj
        1.
        		 update ServicePolicies to include GDS policies relevant to a service Sub-task Resolved Madhan Neethiraj
        2.
        		 REST APIs for plugins to receive GDS data relevant to a service Sub-task Resolved Madhan Neethiraj
        3.
        		 Link resource-based services with GDS service Sub-task Resolved Madhan Neethiraj
        4.
        		 Increment policy-version of linked services on change to GDS policy Sub-task Resolved Madhan Neethiraj
        5.
        		 GDS context-enricher to find datasets relevant to accessed resource Sub-task Resolved Madhan Neethiraj
        6.
        		 Policy engine enhancements to evaluate GDS policies Sub-task Resolved Madhan Neethiraj
        7.
        		 Audit log to capture datasets and projects Sub-task Resolved Madhan Neethiraj
        8.
        		 UI for GDS: dashboard, datasets, projects, data-shares Sub-task In Progress Anand Nadar
        9.
        		 GDS Download API with Cache Sub-task Resolved Subhrat Chaudhary
        10.
        		 Java patches to support GDS changes Sub-task Resolved Monika kachhadiya
        11.
        		 Dataset lookup is failing in GDS policy Sub-task Resolved Unassigned
        12.
        		 API to get DataShare id,name,description List Sub-task Resolved Prashant Satam
        13.
        		 Need API to get list of Datashare/Dataset request for current logged in user Sub-task Open Subhrat Chaudhary
        14.
        		 Support for delta download of GDS data to plugin Sub-task Open Unassigned
        15.
        		 Need a new API to get DataShareInDataset request details to be displayed in UI Sub-task Open Subhrat Chaudhary
        16.
        		 Need additional filters in GET DataShare,Dataset API Sub-task Resolved Prashant Satam
        17.
        		 Fix pagination for GDS APIs Sub-task Resolved Subhrat Chaudhary
        18.
        		 Need a new API to create multiple datashare requests for a dataset Sub-task Resolved Subhrat Chaudhary
        19.
        		 Need a new API to get datashares to be dispalyed on pop up for Add Request Sub-task Resolved Unassigned
        20.
        		 Add default policy for new dataset Sub-task Resolved Prashant Satam
        21.
        		 Add support for filter by service and zone name in get datashare API Sub-task Closed Subhrat Chaudhary
        22.
        		 Need a new API to get history of operations on dataset/datashare requests Sub-task Open Abhishek
        23.
        		 Add field approver in dataset request entities Sub-task Closed Subhrat Chaudhary
        24.
        		 Need a new API to get list of datasets for a datashare with request status Sub-task Patch Available Prashant Satam
        25.
        		 Need a new API to get list of zones mapped to a service Sub-task Resolved Prashant Satam
        26.
        		 Create datshare and sharedResource in cascade Sub-task Resolved Unassigned
        27.
        		 Need additional filters for GET sharedResources API Sub-task Resolved Unassigned
        28.
        		 Need a new API for service lookup Sub-task Resolved Subhrat Chaudhary
        29.
        		 Implement ACL for get API for datashare by id Sub-task Open Unassigned
        30.
        		 Need a new API to get list of requests for the current user Sub-task Resolved Unassigned
        31.
        		 Support cascading delete for datashare Sub-task Closed Subhrat Chaudhary
        32.
        		 Support cascading delete for dataset Sub-task Resolved Prashant Satam
        33.
        		 Add loggedIn user as the admin in dataset/datshare ACL Sub-task Closed Prashant Satam
        34.
        		 Implement request state transition evaluation for DataShareinDataset and DatasetInProject Sub-task Open Unassigned
        35.
        		 When security-zone is deleted with force, trigger cascade delete of datashare Sub-task Open Madhan Neethiraj
        36.
        		 GDS APIs to manage policies Sub-task Resolved Madhan Neethiraj
        37.
        		 Need an API to return dataset summary Sub-task Resolved Subhrat Chaudhary
        38.
        		 Need a new API to get security-zone summary Sub-task Closed Subhrat Chaudhary
        39.
        		 GDS CRUD APIs should return proper response and message in case of failure Sub-task Resolved Prashant Satam
        40.
        		 Dataset with ACL for "public" group List permission are not listed if the user is not added to public Sub-task Closed Prashant Satam
        41.
        		 Need a new API to get datasets for a resource name like db or column name Sub-task Open Abhishek
        42.
        		 Need additional filter on GET sharedResource API Sub-task Resolved Subhrat Chaudhary
        43.
        		 GDS validation - service/zone admins not allowed to update/delete dataShares Sub-task Resolved Madhan Neethiraj
        44.
        		 PrincipalCount not getting updated in DatasetSummary Sub-task Resolved Subhrat Chaudhary
        45.
        		 Need a new API to get dataShare summary Sub-task Resolved Subhrat Chaudhary
        46.
        		 Update approver when request status is updated Sub-task Resolved Subhrat Chaudhary
        47.
        		 Dataset summary API throws 403 Sub-task Resolved Subhrat Chaudhary
        48.
        		 GDS data structure updates to capture masking order and sub-resource type Sub-task Resolved Madhan Neethiraj
        49.
        		 Sort param sortType not considered if sortBy not passed Sub-task Resolved Subhrat Chaudhary
        50.
        		 Generate trx-logs for dataset/project/data-share/shared-resource CRUD Sub-task Resolved Madhan Neethiraj
        51.
        		 support for policies to refer access-types using category, like Create/Read/Update/Delete/Manage Sub-task Resolved Madhan Neethiraj
        52.
        		 Dataset/Datashare/Shared-resource/project name should not be created with trailing spaces Sub-task Resolved Prashant Satam
        53.
        		 Need new API for DataShare in Dataset Summary Sub-task Resolved Prashant Satam
        54.
        		 Need new API for DataShare in Dataset Summary Sub-task Resolved Unassigned
        55.
        		 update service-defs to assign category for access-types Sub-task Resolved Madhan Neethiraj
        56.
        		 GDS --> newly created user with admin role doesn't get permission for GDS module automatically Sub-task Patch Available Abhishek
        57.
        		 NPE in RangerGdsInfoRefresher.run() Sub-task Resolved Madhan Neethiraj
        58.
        		 GDS update APIs fail when guid is not included in payload Sub-task Resolved Madhan Neethiraj
        59.
        		 update GdsPolicyEngine with methods to find datasets/projects for principals Sub-task Resolved Madhan Neethiraj
        60.
        		 GET dataset API should return public:LIST in ACL if available Sub-task Resolved Subhrat Chaudhary
        61.
        		 Audit UI updates to support datasets and projects Sub-task Resolved Abhishek
        62.
        		 upgrade audit schema ,in Solr and Elasticsearch, to support datasets and projects Sub-task Open Unassigned
        63.
        		 Plugin status should track GDS info download details as well Sub-task Resolved Madhan Neethiraj
        64.
        		 Plugin status UI should render GDS download details Sub-task Resolved Brijesh Bhalala
        65.
        		 Need api to get count of all request list according to their status and the total count of requests. Sub-task Resolved Unassigned
        66.
        		 Java patch to assign category to existing accessTypeDefs in Ranger database Sub-task Resolved Madhan Neethiraj
        67.
        		 API to support retrieval of datasets/projects shared with the caller Sub-task Resolved Madhan Neethiraj
        68.
        		 Need to add query param createdBy for dataset GET API Sub-task Resolved Prashant Satam
        69.
        		 Add java patch to update changes in db Sub-task Resolved Monika kachhadiya
        70.
        		 Delete GDS objects on security-zone update/delete Sub-task Resolved Madhan Neethiraj
        71.
        		 [Ranger UI] Clicking on the policy Id in the access audits (audits related to gds) leads to an error Sub-task Resolved Abhishek
        72.
        		 Add Unit tests for GDS related implementation files Sub-task Open Abhishek
        73.
        		 [Ranger UI] Text overflowing in the datashares table listing in the dataset -> datashares page Sub-task Resolved Abhishek
        74.
        		 Return proper error message if GDS entities name is longer than 512 characters and assertions in the UI form as well Sub-task Resolved Abhishek
        75.
        		 Value for datasets and projects field in the access audits must be JSON serializable Sub-task Resolved Abhishek
        76.
        		 [Ranger React UI] Handle Dataset and Datashare creation errors gracefully Sub-task Resolved Abhishek
        77.
        		 Hive: USE fails even though user has access in the database via GDS policies Sub-task Resolved Madhan Neethiraj
        78.
        		 Restrict the use of special characters (other than _ and - ) in GDS entity names Sub-task Open Abhishek

        Activity

          People

            madhan Madhan Neethiraj
            madhan Madhan Neethiraj
            Votes:
            1 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated: