Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-8548

Integrate with remote Spark context after HIVE-8528 [Spark Branch]

Log workAgile BoardRank to TopRank to BottomBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • None
    • Spark
    • None

    Description

      With HIVE-8528, HiverSever2 should use remote Spark context to submit job and monitor progress, etc. This is necessary if Hive runs on standalone cluster, Yarn, or Mesos. If Hive runs with spark.master=local, we should continue using SparkContext in current way.
      We take this as root JIRA to track all Remote Spark Context integration related subtasks.

      Attachments

        Issue Links

        depends upon

        Sub-task - The sub-task of the issue HIVE-8528 Add remote Spark client to Hive [Spark Branch]

        • Major - Major loss of function.
        • Resolved
        is part of

        Improvement - An improvement or enhancement to an existing feature or task. HIVE-7292 Hive on Spark

        • Major - Major loss of function.
        • Resolved
        1.
        		 Unify spark client API and implement remote spark client.[Spark Branch] Sub-task Resolved Chengxiang Li Actions
        2.
        		 enable job progress monitoring of Remote Spark Context [Spark Branch] Sub-task Resolved Rui Li Actions
        3.
        		 Enhance metrics gathering in Spark Client [Spark Branch] Sub-task Resolved Marcelo Masiero Vanzin Actions
        4.
        		 identify dependency scope for Remote Spark Context.[Spark Branch] Sub-task Resolved Chengxiang Li Actions
        5.
        		 Enable automatic tests with remote spark client [Spark Branch] Sub-task Resolved Rui Li Actions
        6.
        		 Guava dependency conflict between hive driver and remote spark context[Spark Branch] Sub-task Resolved Marcelo Masiero Vanzin Actions
        7.
        		 Automatic calculate reduce number for spark job [Spark Branch] Sub-task Resolved Jimmy Xiang Actions
        8.
        		 SparkSession and SparkClient mapping[Spark Branch] Sub-task Resolved Rui Li Actions
        9.
        		 Spark remote context doesn't work with local-cluster [Spark Branch] Sub-task Resolved Xuefu Zhang Actions
        10.
        		 Hive hangs while some error/exception happens beyond job execution [Spark Branch] Sub-task Resolved Rui Li Actions
        11.
        		 Remote spark context needs to clean up itself in case of connection timeout [Spark Branch] Sub-task Open Xuefu Zhang Actions
        12.
        		 SparkSession is not closed until JVM exit.[Spark Branch] Sub-task Resolved Chengxiang Li Actions
        13.
        		 Implement more fine-grained remote client-level events [Spark Branch] Sub-task Resolved Rui Li Actions
        14.
        		 Enable job metrics/statistics gathering for remote spark context [Spark Branch] Sub-task Resolved Rui Li Actions
        15.
        		 Fix custom_input_output_format [Spark Branch] Sub-task Resolved Rui Li Actions

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            chengxiang li Chengxiang Li Assign to me
            xuefuz Xuefu Zhang
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment