Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-21769 Support Partition level filtering for hive replication command
  3. HIVE-21772

Support dynamic addition and deletion of partitions in the policy

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 4.0.0
    • None
    • None
    • None

    Description

      If the user modify the filter condition in the policy, then the participating partitions of the policy can change. During such scenarios, user needs to provide the old filter condition along with the REPL DUMP command.

      • The old filter will be passed as a string along with ‘with’ clause of the REPL dump command. Need to create the AST from the string to be used for filtering.
      • Convert the string to list of AST, one for each table and make a list of the partitions satisfying the old filter condition.
      • List of partition satisfying the new filter condition will be compared with the old list.
      • If the partition is not present in old but is present in new, then the partition will be added to the list of partitions to be bootstrapped.
      • If the partition is present in old, but not present in new then the partition will be added to the list of partitions to be deleted.
      • During load operation, after all the events are replayed, the list of bootstrap and list of deleted will be read and corresponding action will be executed at target.
      • There is a possibility that the partitioned to be deleted is already deleted using some event replayed, in that case delete will be ignored.
      • Similarly if some partition from the list of bootstrap is already present, then bootstrap will be ignored.
      • As the partition can not be present in both bootstrap and delete list, the list can be executed in parallel.

       

      Attachments

        Activity

          People

            maheshk114 mahesh kumar behera
            maheshk114 mahesh kumar behera
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: