Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-27494

Deduplicate the task result that generated by more branches in union all

Log workAgile BoardRank to TopRank to BottomBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      HIVE-23891 adds the ability to deduplicate the task result that under the directory,

      <table-dir>/<staging-dir>/_tmp.-ext-10000/<dynamic-partition-dir>/HIVE_UNION_SUBDIR_1,

      but turns out to ignore taking the same action to the output directory for the same query:

      <table-dir>/<staging-dir>/_tmp.-ext-10000/<dynamic-partition-dir>/HIVE_UNION_SUBDIR_2.

      So user may still have the same data duplication problem upon multiple tez task attempts.

      Attachments

        1. ddl.q
          0.5 kB
          Zhihua Deng
        2. explain.output
          17 kB
          Zhihua Deng

        Issue Links

        relates to

        Bug - A problem which impairs or prevents the functions of the product. HIVE-23891 UNION ALL and multiple task attempts can cause file duplication

        • Major - Major loss of function.
        • Resolved
        links to

        Web Link GitHub Pull Request #4479

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            dengzh Zhihua Deng Assign to me
            dengzh Zhihua Deng
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment