Scala Spark Test Utils/Framework

As users of Apache Spark in Scala, I've observed a gap in the availability of a dedicated testing framework similar to the one present for PySpark. While various open-source repositories and resources exist for PySpark testing also has Scala counterpart:

• [spark-testing-base|holdenk/spark-testing-base: Base classes to use when writing tests with Spark (github.com)]
• [spark-fast-test|MrPowers/spark-fast-tests: Apache Spark testing helpers (dependency free & works with Scalatest, uTest, and MUnit) (github.com)]

The same level of support is lacking for Spark Scala. For example, spark-fast-test was last updated 2 years ago and still not supporting newer spark version. This cause some pain when I am trying to do Integration and performance testing on our cluster where we provide Fat Test Jar with provided Spark Jar of newer version.

I propose the development of an official testing utils tailored specifically for Spark Scala with similar if not the same feature providing by PySpark Test utils provided since 3.5.0. These utilities would empower Scala developers with the tools and capabilities necessary to write robust and reliable tests for their Spark applications.

Key Features to Include:

• Utility Functions: Provide a suite of utility functions designed to simplify common testing tasks, such as assert_dataframe_equality and assert_schema_equality, thereby reducing boilerplate code and accelerating test development. In the Future this would also help us implement, consolidate the best practices for comparing two dataframes correctly and efficiently with automated testing and benchmarks. 
• Comprehensive Documentation: Offer comprehensive documentation and examples to guide users in effectively utilizing the testing framework/utils, ensuring ease of adoption and accelerating the learning curve.

I am happy to contribute to this feature and would welcome any help. Also Let me know if This would require a SPIP