Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-45029

Ignore `from_protobuf messageClassName` and `from_protobuf messageClassName options` in `PlanGenerationTestSuite`

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.5.0, 4.0.0
    • 3.5.0, 4.0.0
    • Connect, Tests
    • None

    Description

      The SPARK-43646 makes both Maven and SBT use the shaded `spark-protobuf` module when testing the connect module. This allows `mvn clean install` and `mvn package test` to successfully pass tests. But if `mvn clean test` is executed directly, an error in `package org.sparkproject.spark_protobuf.protobuf does not exist` will occur. This is because `mvn clean test` directly uses the classes file of the `spark-protobuf` module for testing, without the 'package', hence it does not `shade` and `relocate` protobuf. 

      On the other hand, SPARK-43646 breaks the usability of importing Spark as a Maven project into IDEA. 

      Therefore, the changes of SPARK-43646 be reverted in https://github.com/apache/spark/pull/42746

      In summary, when using Maven to test the connect module:

      • sometimes the shaded spark-protobuf jar will be used, such as 'mvn package test',
      • sometimes the unshaded classes directory will be used, such as 'mvn clean test'.

      Therefore, there is no perfect solution to pass both tests, so 'from_protobuf messageClassName' and 'from_protobuf messageClassName' in 'PlanGenerationTestSuite' are ignored for now, and they will be re-enabled when we find a better solution.

      Attachments

        Issue Links

          Activity

            People

              LuciferYang Yang Jie
              LuciferYang Yang Jie
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: