Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
3.5.0
-
None
Description
Current Spark Remote connects looks to have a single spark remote connect session. Can it be extended to do multiple Spark Sessions. This would help in creating decentralized Kubernetes/Custom Cloud Environment and reduce the compute of the current Spark Session.
Making Spark to work like a full remote API call for multiple datasets where requried.