Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Duplicate
-
0.4.0
-
None
-
None
-
livy 0.4
spark2.2.0-cloudera1
Description
To use spark sql concurrency for internal fair schedule
in [post] /sessions/
/statements use param "conf":
{"spark.scheduler.mode":"FAIR","spark.scheduler.allocation.file":"/home/.../spark-schedule.xml"}spark-schedule.xml is as follows:
<allocations>
<pool name="FAIR">
<schedulingMode>FAIR</schedulingMode>
</pool>
</allocations>
Log shows tow jobs surely be added into fair pool:
INFO scheduler.FairSchedulableBuilder: Added task set TaskSet_10.0 tasks to pool FAIR
...
INFO scheduler.FairSchedulableBuilder: Added task set TaskSet_11.0 tasks to pool FAIR
But from spark web console the second job starts only when the first finished.
I also tried with zeppelin, which concurrency works well with spark but not work in livy.
Did I miss some configuration? Thanks.
Attachments
Attachments
Issue Links
- duplicates
-
LIVY-421 support multi statements run at the same time
- Open