Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-22386 Data Source V2 improvements
  3. SPARK-25127

DataSourceV2: Remove SupportsPushDownCatalystFilters

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.3.0, 2.3.1, 2.4.0
    • 2.4.0
    • SQL
    • None

    Description

      Discussion about adding TableCatalog on the dev list focused around whether Expression should be used in the public DataSourceV2 API, with SupportsPushDownCatalystFilters as an example of where it is already exposed. The early consensus is that Expression should not be exposed in the public API.

      From rxin:

      I completely disagree with using Expression in critical public APIs that we expect a lot of developers to use . . . If we are depending on Expressions on the more common APIs in dsv2 already, we should revisit that.

      The main use of this API is to pass Expression to FileFormat classes that used Expression instead of Filter. External sources also use it for more complex push-down, like to_date(ts) = '2018-05-13', but those uses can be done with Analyzer rules or when translating to Filters.

      Attachments

        Issue Links

          Activity

            People

              rxin Reynold Xin
              rdblue Ryan Blue
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: