Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Won't Fix
-
2.4.0, 3.0.0
-
None
-
None
Description
Our user always submit bad SQL in query platform, Such as :
- write wrong join condition but submit that sql
- write wrong where condition
- etc..
This case will make Spark scheduler to submit a lot of task. It will cause spark run very slow and impact other user(spark thrift server) even run out of memory because of too many object generated by a big num of tasks.
So I add a constraint when submit tasks and abort stage early when TaskSet size num is bigger then set limit . I wonder if the community will accept this way.
cc srowen dongjoon yumwang