Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-19924

Tag distcp jobs run by Repl Load

Log workAgile BoardRank to TopRank to BottomBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      Add tags in jobconf for distcp related jobs started by replication. This will allow hive to kill these jobs in case beacon retries, or hs2 dies and beacon issues a kill command.

      • one of the tags should definitely be the query_id that starts the job : With this flow beacon before retrying the bootstrap load, will issue a kill command to hs2 with the query id of the previous issued command. hs2 will then kill an running jobs on yarn tagged with the Query_id.
      • To get around the additional failure point as mentioned above. The jobs can be tagged with an additional unique tag_id provided by Beacon in the WITH clause in repl load command to be used to tag distcp jobs ). Enhance the kill api to take the tag as input and kill jobs associated with that tag. Problem here is how do we validate the association of the tag with a hive query id to make sure this api is not used to kill jobs run by other components, however we can provide this capability to only admins and should be ok in that case.

      Attachments

        1. HIVE-19924.14-branch-3.patch
          34 kB
          mahesh kumar behera
        2. HIVE-19924.14.patch
          34 kB
          mahesh kumar behera
        3. HIVE-19924.13.patch
          33 kB
          mahesh kumar behera
        4. HIVE-19924.12.patch
          33 kB
          mahesh kumar behera
        5. HIVE-19924.11.patch
          23 kB
          mahesh kumar behera
        6. HIVE-19924.10.patch
          20 kB
          mahesh kumar behera
        7. HIVE-19924.09.patch
          21 kB
          mahesh kumar behera
        8. HIVE-19924.08.patch
          19 kB
          mahesh kumar behera
        9. HIVE-19924.07.patch
          19 kB
          mahesh kumar behera
        10. HIVE-19924.06.patch
          19 kB
          mahesh kumar behera
        11. HIVE-19924.05.patch
          19 kB
          mahesh kumar behera
        12. HIVE-19924.04.patch
          19 kB
          mahesh kumar behera
        13. HIVE-19924.03.patch
          19 kB
          mahesh kumar behera
        14. HIVE-19924.02.patch
          19 kB
          mahesh kumar behera
        15. HIVE-19924.01.patch
          13 kB
          mahesh kumar behera

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            maheshk114 mahesh kumar behera Assign to me
            maheshk114 mahesh kumar behera
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

              Estimated:
              Original Estimate - Not Specified
              Not Specified
              Remaining:
              Remaining Estimate - 0h
              0h
              Logged:
              Time Spent - 20m
              20m

              Slack

                Issue deployment