Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-23049

`spark.sql.files.ignoreCorruptFiles` should work for ORC files

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.0.2, 2.1.2, 2.2.1
    • 2.3.0
    • SQL
    • None

    Description

      When `spark.sql.files.ignoreCorruptFiles=true`, we should ignore corrupted ORC files.

      Attachments

        Issue Links

          blocks

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-20901 Feature parity for ORC with Parquet

          • Major - Major loss of function.
          • Open
          links to

          Pull request #20240 [Github] Pull Request #20240 (dongjoon-hyun)

          Activity

            People

              dongjoon Dongjoon Hyun
              dongjoon Dongjoon Hyun
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: