Uploaded image for project: 'Livy'
  1. Livy
  2. LIVY-428

Does livy support concurrency for centain spark session?

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Duplicate
    • 0.4.0
    • None
    • API
    • None
    • livy 0.4
      spark2.2.0-cloudera1

    Description

      To use spark sql concurrency for internal fair schedule
      in [post] /sessions/

      {id}

      /statements use param "conf":

      {"spark.scheduler.mode":"FAIR","spark.scheduler.allocation.file":"/home/.../spark-schedule.xml"}

      spark-schedule.xml is as follows:
      <allocations>
      <pool name="FAIR">
      <schedulingMode>FAIR</schedulingMode>
      </pool>
      </allocations>

      Log shows tow jobs surely be added into fair pool:
      INFO scheduler.FairSchedulableBuilder: Added task set TaskSet_10.0 tasks to pool FAIR
      ...
      INFO scheduler.FairSchedulableBuilder: Added task set TaskSet_11.0 tasks to pool FAIR

      But from spark web console the second job starts only when the first finished.
      I also tried with zeppelin, which concurrency works well with spark but not work in livy.
      Did I miss some configuration? Thanks.

      Attachments

        1. spark-fair.png
          19 kB
          suheng.cloud

        Issue Links

          Activity

            People

              Unassigned Unassigned
              suheng.cloud suheng.cloud
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: