Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-15777 Catalog federation
  3. SPARK-17767

Spark SQL ExternalCatalog API custom implementation support

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Later
    • 2.0.0
    • None
    • SQL
    • None

    Description

      There is no way/easy way to configure Spark to use customized ExternalCatalog. Internal source code is hardcoded to use either hive or in-memory metastore. Spark SQL thriftserver is hardcoded to use HiveExternalCatalog. We should be able to create a custom external catalog and thriftserver should be able to use it. Potentially Spark SQL thriftserver shouldn't depend on Hive thriftserer.

      Attachments

        Issue Links

          is duplicated by

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-19632 Allow configuring non-hive and non-local SessionState and ExternalCatalog

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-30617 Is there any possible that spark no longer restrict enumerate types of spark.sql.catalogImplementation

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed
          is related to

          New Feature - A new feature of the product, which has yet to be developed. SPARK-23443 Spark with Glue as external catalog

          • Major - Major loss of function.
          • Open
          is required by

          New Feature - A new feature of the product, which has yet to be developed. IGNITE-3084 Spark Data Frames Support in Apache Ignite

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          links to

          Pull request #15336 [Github] Pull Request #15336 (dongjoon-hyun)

          Activity

            People

              Unassigned Unassigned
              alexliu68 Alex Liu
              Votes:
              1 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: