SPIP: Proposal to Adopt Google's Spark K8s Operator as Official Spark Operator

Description:

This proposal aims to recommend the adoption of Google's Spark K8s Operator as the official Spark Operator for the Apache Spark community. The operator has gained significant traction among many users and organizations and used heavily in production environments, but challenges related to maintenance and governance necessitate this recommendation.

Background:

• Google's Spark K8s Operator is currently in use by hundreds of users and organizations. However, due to maintenance issues, many of these users and organizations have resorted to forking the repository and implementing their own fixes.

• The project boasts an impressive user base with 167 contributors, 2.5k likes, and endorsements from 45 organizations, as documented in the "Who is using" document. Notably, there are many more organizations using it than the initially reported 45.

• The primary issue at hand is that this project resides under the GoogleCloudPlatform GitHub organization and is exclusively moderated by a Google employee. Concerns have been raised by numerous users and customers regarding the maintenance of the repository.

• The existing Google maintainers are constrained by limitations in terms of time and support, which negatively impacts both the project and its user community.

 

Recent Developments:

• During Kubecon Chicago 2023, AWS OSS Architects (Vara Bonthu) and the Apple infrastructure team engaged in discussions with the Google's team, specifically with Marcin Wielgus. They expressed their interest in contributing the project to either the Kubeflow or Apache Spark community.

• Marcin from Google confirmed their willingness to donate the project to either of these communities.

• An adoption process has been initiated by the Kubeflow project under CNCF, as documented in the following thread: Link to the thread.

 

Primary Goal:

• The primary goal is to ensure the collaborative support and adoption of Google's Spark Operator by the Apache Spark , thereby avoiding the development of redundant tools and reducing confusion among users.

Next Steps:

• Meeting with Apache Spark Working Group Maintainers: We propose arranging a meeting with the Apache Spark working group maintainers to delve deeper into this matter, address any questions or concerns they may have, and collectively work towards a decision.

• Establish a New Working Group: Upon reaching an agreement, we intend to create a new working group comprising members from diverse organizations who are willing to contribute and collaborate on this initiative.

• Repository Transfer: Our plan involves transferring the project repository from Google's organization to either the Apache or Kubeflow organization, aligning with the chosen community.

• Roadmap Development: We will formulate a new roadmap that encompasses immediate issue resolution and a long-term design strategy aimed at enhancing performance, scalability, and security for this tool.

 
We believe that working towards one Spark Operator will benefit the Apache Spark community and address the current maintenance challenges. Your feedback and support in this matter are highly valued. Let's collaborate to ensure a robust and well-maintained Spark Operator for the Apache Spark community's benefit.

Community members are encouraged to leave their comments or give a thumbs-up to express their support for adopting Google's Spark Operator as the official Apache Spark operator.

 

Proposed Authors

Vara Bonthu (AWS)

Marcin Wielgus (Google)