Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-17861

Store data source partitions in metastore and push partition pruning into metastore

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • None
    • 2.1.0
    • SQL
    • None

    Description

      Initially, Spark SQL does not store any partition information in the catalog for data source tables, because initially it was designed to work with arbitrary files. This, however, has a few issues for catalog tables:

      1. Listing partitions for a large table (with millions of partitions) can be very slow during cold start.
      2. Does not support heterogeneous partition naming schemes.
      3. Cannot leverage pushing partition pruning into the metastore.

      This ticket tracks the work required to push the tracking of partitions into the metastore. This change should be feature flagged.

      Attachments

        Issue Links

          is depended upon by

          Sub-task - The sub-task of the issue HADOOP-13525 Optimize uses of FS operations in the ASF analysis frameworks and libraries

          • Major - Major loss of function.
          • Resolved
          is related to

          New Feature - A new feature of the product, which has yet to be developed. SPARK-15691 Refactor and improve Hive support

          • Major - Major loss of function.
          • Resolved
          relates to

          Sub-task - The sub-task of the issue SPARK-16980 Load only catalog table partition metadata required to answer a query

          • Major - Major loss of function.
          • Resolved
          There are no Sub-Tasks for this issue.

          Activity

            People

              ekhliang Eric Liang
              rxin Reynold Xin
              Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: