Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-45854

spark.catalog.listTables fails with ParseException after upgrading to Spark 3.4.1 from 3.3.1

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsAdd voteVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete CommentsDelete
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.4.0, 3.4.1
    • None
    • None

    Description

      After upgrading to Spark 3.4.1, the listTables() method in PySpark now throws a ParseException with the message "Syntax error at or near end of input.". This did not occur in previous versions of Spark, such as 3.3.1.

      Install Spark version 3.4.1.
       
      Run pyspark
      ```bash
      pyspark --packages io.delta:delta-core_2.12:2.4.0 --conf "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension" --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog"
      ```
       
      Attempt to list tables using
      ```console
      spark.range(1).createTempView("test_view")
      spark.catalog.listTables()
      ```
      Expected result: The listTables() method should return a list of tables without throwing any exceptions.

      Actual result:
      Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File ".venv/lib/python3.10/site-packages/pyspark/sql/catalog.py", line 302, in listTables
      iter = self._jcatalog.listTables(dbName).toLocalIterator()
      File ".venv/lib/python3.10/site-packages/pyspark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1322, in {}call{}
      File ".venv/lib/python3.10/site-packages/pyspark/errors/exceptions/captured.py", line 175, in deco
      raise converted from None
      pyspark.errors.exceptions.captured.ParseException:
      [PARSE_SYNTAX_ERROR] Syntax error at or near end of input.(line 1, pos 0)

      == SQL ==

      ^^^

      >>>

      The same code worked correctly in Spark version 3.3.1.
      No changes were made to the code aside from upgrading Spark.

      Thank you for considering this issue! Any assistance in resolving it would be greatly appreciated.

      Best regards,
      Andrej

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned Assign to me
            orolesko Andrej Zachar

            Dates

              Created:
              Updated:

              Slack

                Issue deployment