Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-21769 Support Partition level filtering for hive replication command
  3. HIVE-21773

Supporting external table replication with partition filter.

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 4.0.0
    • Fix Version/s: 4.0.0
    • Component/s: HiveServer2, repl
    • Labels:
      None
    • Target Version/s:

      Description

      Hive external table replication is done differently than managed table replication. In case of external table, list is created for the locations of the table and partitions to be replicated. If the partition location is within the table location, then partition location is not added to the list. For partitions with location outside table, partition location is added to the list. In case of incremental dump, the data related events are ignored and just the metadata related events are dumped. The list of location is prepared and that is used for replication. During load, the events are replayed and then the distcp tasks are created, one for each location present in the list.

      For partition level replication, not all partition will be present in the dump. So even if the partition locations are within the table location, each partition location will be added to the list.

      • If where condition is present in the REPL DUMP command then add location for each satisfying partition even though the partition location is within table location.
      • If table is not mentioned in the where clause then follow the older behavior.
      • If table is mentioned with a key but the key does not match any of the partitioned column then fail repl dump.
      • If the table is mentioned with the key and even if all the partitions are satisfying the filter condition, add location for each partition. This is to avoid copying partitions which are added using alter after the dump.

        Attachments

          Activity

            People

            • Assignee:
              maheshk114 mahesh kumar behera
              Reporter:
              maheshk114 mahesh kumar behera
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: